[PATCH v2 00/28] x86: Improve operation under QEMU

[PATCH v2 00/28] x86: Improve operation under QEMU

[Simon Glass]

U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
is not perfect.

With both builds, executing the VESA ROM causes an intermittent hang, at
least on some AMD CPUs.

With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
is done in a way that works on real hardware but not with QEMU. This
means that performance is 4-5x slower than it could be, at least on my
CPU.

We can work around the first problem by using Bochs, which is anyway a
better choice than VESA for QEMU. The second can be addressed by using
the same descriptor across the jump to long mode.

With an MTRR fix this allows booting into Ubuntu on qemu-x86_64

Changes in v2:
- Redo commit message
- Add new patch to rename the _D dirty flag
- Add new patch to support CPU functions in long mode
- Add new patch to tidy up address size in MTRR calculations
- Add new patch with a 64-bit version of is_power_of_2()
- Add new patch to set an MTRR for the RAM in QEMU
- Add new patch with a helper to send characters
- Add new patch to allow tests to be filtered by role
- Add more patches to support booting with kvm
- Add new patch with a test for booting Ubuntu 24.04

Simon Glass (28):
  x86: Expand x86_64 early memory
  x86: qemu: Switch to bochs display
  x86: qemu: Enable dhrystone
  x86: qemu: Avoid accessing BSS too early
  x86: Drop mpspec from the SPL build
  x86: Add some log categories
  x86: Drop use of CONFIG_REALMODE_DEBUG
  x86: Avoid clearing the VESA display
  x86: Add 64-bit entries to the GDT
  x86: Use defines for the cache flags
  x86: spl: Drop duplicate CPU init
  x86: Drop the message about features missing in 64-bit
  x86: Include stdbool.h in interrupt header
  x86: Tidy up the GDT size in start/16.S
  x86: Disable paging before changing to long mode
  x86: Use the same GDT when jumping to long mode
  x86: Use a simple jump into long mode
  x86: Rename the _D dirty flag
  x86: Support CPU functions in long mode
  x86: Tidy up address size in MTRR calculations
  Add a 64-bit version of is_power_of_2()
  x86: Support MTRRs of 4GB on 32-bit machines
  x86: emulation: Set an MTRR for the RAM
  scripts: Expand a few options
  test/py: Add a helper to send characters
  test/py: Allow tests to be filtered by role
  RFC: test/py: Deal with timeouts
  test: Add a test for booting Ubuntu 24.04

 .gitlab-ci.yml                   |  5 +++
 arch/x86/cpu/cpu.c               | 24 +++++++++++++++
 arch/x86/cpu/i386/call64.S       | 35 +++++++++------------
 arch/x86/cpu/i386/cpu.c          | 41 +++++++++---------------
 arch/x86/cpu/mtrr.c              | 12 +++++---
 arch/x86/cpu/qemu/dram.c         | 15 +++++++++
 arch/x86/cpu/qemu/qemu.c         | 20 ++++++++----
 arch/x86/cpu/start.S             |  4 ++-
 arch/x86/cpu/start16.S           |  3 +-
 arch/x86/cpu/x86_64/cpu.c        |  5 ---
 arch/x86/include/asm/cpu.h       | 35 ++++++++++++++++++---
 arch/x86/include/asm/interrupt.h |  1 +
 arch/x86/include/asm/processor.h |  5 ++-
 arch/x86/lib/Makefile            |  2 ++
 arch/x86/lib/bios.c              | 27 +++++++++-------
 arch/x86/lib/bios_interrupts.c   |  8 ++---
 arch/x86/lib/i8259.c             |  2 ++
 arch/x86/lib/spl.c               |  4 +--
 configs/qemu-x86_64_defconfig    | 10 +++---
 configs/qemu-x86_defconfig       |  6 ++--
 include/linux/log2.h             |  6 ++++
 scripts/build-qemu.sh            |  4 +--
 test/py/conftest.py              | 22 +++++++++++++
 test/py/console_base.py          | 33 ++++++++++++--------
 test/py/pytest.ini               |  1 +
 test/py/tests/test_distro.py     | 53 ++++++++++++++++++++++++++++++++
 test/py/tests/test_sleep.py      |  1 +
 27 files changed, 273 insertions(+), 111 deletions(-)
 create mode 100644 test/py/tests/test_distro.py

-- 
2.43.0

[PATCH v2 01/28] x86: Expand x86_64 early memory

[Simon Glass]

The SPL and pre-reloc malloc()-space is not large enough to start up
with a display. Expand it.

Switch the order of SPL_SYS_MALLOC_F_LEN and SPL_TEXT_BASE since this
matches what 'savedefconfig' gives us.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 configs/qemu-x86_64_defconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/configs/qemu-x86_64_defconfig b/configs/qemu-x86_64_defconfig
index 792ba06c2b8..af275e8ce8f 100644
--- a/configs/qemu-x86_64_defconfig
+++ b/configs/qemu-x86_64_defconfig
@@ -1,13 +1,13 @@
 CONFIG_X86=y
 CONFIG_TEXT_BASE=0x1110000
-CONFIG_SYS_MALLOC_F_LEN=0x1000
+CONFIG_SYS_MALLOC_F_LEN=0x1800
 CONFIG_BLOBLIST_SIZE_RELOC=0x20000
 CONFIG_NR_DRAM_BANKS=8
 CONFIG_ENV_SIZE=0x40000
 CONFIG_MAX_CPUS=2
 CONFIG_SPL_DM_SPI=y
 CONFIG_DEFAULT_DEVICE_TREE="qemu-x86_i440fx"
-CONFIG_SPL_SYS_MALLOC_F_LEN=0x3000
+CONFIG_SPL_SYS_MALLOC_F_LEN=0x4800
 CONFIG_SPL_TEXT_BASE=0xfffd0000
 CONFIG_DEBUG_UART_BASE=0x3f8
 CONFIG_DEBUG_UART_CLOCK=1843200
-- 
2.43.0

[PATCH v2 02/28] x86: qemu: Switch to bochs display

[Simon Glass]

The vesa display is widely used on hardware, but it is a bit of a pain
with QEMU. It requires executing option ROMs, which either doesn't work
with kvm, or is difficult to do in a kvm/QEMU-friendly way.

THe bochs display is probably better anyway, so switch to that. It works
fine with kvm as it doesn't need an option ROM.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: https://source.denx.de/u-boot/custodians/u-boot-dm/-/issues/31
---

Changes in v2:
- Redo commit message

 configs/qemu-x86_64_defconfig | 5 ++---
 configs/qemu-x86_defconfig    | 5 ++---
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/configs/qemu-x86_64_defconfig b/configs/qemu-x86_64_defconfig
index af275e8ce8f..ff2a192ee7d 100644
--- a/configs/qemu-x86_64_defconfig
+++ b/configs/qemu-x86_64_defconfig
@@ -83,9 +83,8 @@ CONFIG_SYS_NS16550_PORT_MAPPED=y
 CONFIG_SPI=y
 CONFIG_USB_KEYBOARD=y
 CONFIG_CONSOLE_TRUETYPE=y
-CONFIG_FRAMEBUFFER_SET_VESA_MODE=y
-CONFIG_FRAMEBUFFER_VESA_MODE_USER=y
-CONFIG_FRAMEBUFFER_VESA_MODE=0x144
+CONFIG_VIDEO_BOCHS=y
+# CONFIG_VIDEO_VESA is not set
 CONFIG_CONSOLE_SCROLL_LINES=5
 CONFIG_SPL_VIDEO=y
 # CONFIG_SPL_USE_TINY_PRINTF is not set
diff --git a/configs/qemu-x86_defconfig b/configs/qemu-x86_defconfig
index 0b0e10c795f..ee5b150e062 100644
--- a/configs/qemu-x86_defconfig
+++ b/configs/qemu-x86_defconfig
@@ -62,9 +62,8 @@ CONFIG_SYS_NS16550_PORT_MAPPED=y
 CONFIG_SPI=y
 CONFIG_USB_KEYBOARD=y
 CONFIG_CONSOLE_TRUETYPE=y
-CONFIG_FRAMEBUFFER_SET_VESA_MODE=y
-CONFIG_FRAMEBUFFER_VESA_MODE_USER=y
-CONFIG_FRAMEBUFFER_VESA_MODE=0x144
+CONFIG_VIDEO_BOCHS=y
+# CONFIG_VIDEO_VESA is not set
 CONFIG_CONSOLE_SCROLL_LINES=5
 CONFIG_GENERATE_ACPI_TABLE=y
 # CONFIG_GZIP is not set
-- 
2.43.0

[PATCH v2 03/28] x86: qemu: Enable dhrystone

[Simon Glass]

Provide the 'dhry' command, which helps to check that kvm is being used
properly with QEMU.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 configs/qemu-x86_64_defconfig | 1 +
 configs/qemu-x86_defconfig    | 1 +
 2 files changed, 2 insertions(+)

diff --git a/configs/qemu-x86_64_defconfig b/configs/qemu-x86_64_defconfig
index ff2a192ee7d..84b7dc8e99f 100644
--- a/configs/qemu-x86_64_defconfig
+++ b/configs/qemu-x86_64_defconfig
@@ -89,4 +89,5 @@ CONFIG_CONSOLE_SCROLL_LINES=5
 CONFIG_SPL_VIDEO=y
 # CONFIG_SPL_USE_TINY_PRINTF is not set
 CONFIG_GENERATE_ACPI_TABLE=y
+CONFIG_CMD_DHRYSTONE=y
 # CONFIG_GZIP is not set
diff --git a/configs/qemu-x86_defconfig b/configs/qemu-x86_defconfig
index ee5b150e062..d52afa42955 100644
--- a/configs/qemu-x86_defconfig
+++ b/configs/qemu-x86_defconfig
@@ -66,4 +66,5 @@ CONFIG_VIDEO_BOCHS=y
 # CONFIG_VIDEO_VESA is not set
 CONFIG_CONSOLE_SCROLL_LINES=5
 CONFIG_GENERATE_ACPI_TABLE=y
+CONFIG_CMD_DHRYSTONE=y
 # CONFIG_GZIP is not set
-- 
2.43.0

[PATCH v2 04/28] x86: qemu: Avoid accessing BSS too early

[Simon Glass]

BSS is placed in DRAM which is actually available early with QEMU. But
it is cleared by the init sequence, so values stored there are lost.

Move the system-type flag into a function, instead.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/qemu/qemu.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/arch/x86/cpu/qemu/qemu.c b/arch/x86/cpu/qemu/qemu.c
index 563f63e2bc8..e846ccd44aa 100644
--- a/arch/x86/cpu/qemu/qemu.c
+++ b/arch/x86/cpu/qemu/qemu.c
@@ -15,14 +15,21 @@
 #include <asm/arch/qemu.h>
 #include <asm/u-boot-x86.h>
 
-static bool i440fx;
-
 #if CONFIG_IS_ENABLED(QFW_PIO)
 U_BOOT_DRVINFO(x86_qfw_pio) = {
 	.name = "qfw_pio",
 };
 #endif
 
+static bool is_i440fx(void)
+{
+	u16 device;
+
+	pci_read_config16(PCI_BDF(0, 0, 0), PCI_DEVICE_ID, &device);
+
+	return device == PCI_DEVICE_ID_INTEL_82441;
+}
+
 static void enable_pm_piix(void)
 {
 	u8 en;
@@ -50,16 +57,17 @@ static void enable_pm_ich9(void)
 
 void qemu_chipset_init(void)
 {
-	u16 device, xbcs;
+	bool i440fx;
+	u16 xbcs;
 	int pam, i;
 
+	i440fx = is_i440fx();
+
 	/*
 	 * i440FX and Q35 chipset have different PAM register offset, but with
 	 * the same bitfield layout. Here we determine the offset based on its
 	 * PCI device ID.
 	 */
-	pci_read_config16(PCI_BDF(0, 0, 0), PCI_DEVICE_ID, &device);
-	i440fx = (device == PCI_DEVICE_ID_INTEL_82441);
 	pam = i440fx ? I440FX_PAM : Q35_PAM;
 
 	/*
@@ -123,7 +131,7 @@ int mp_determine_pci_dstirq(int bus, int dev, int func, int pirq)
 {
 	u8 irq;
 
-	if (i440fx) {
+	if (is_i440fx()) {
 		/*
 		 * Not like most x86 platforms, the PIRQ[A-D] on PIIX3 are not
 		 * connected to I/O APIC INTPIN#16-19. Instead they are routed
-- 
2.43.0

[PATCH v2 05/28] x86: Drop mpspec from the SPL build

[Simon Glass]

This is not needed in SPL, so drop it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 43e6a1de77d..a908356e8a6 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -26,7 +26,9 @@ obj-y	+= e820.o
 obj-y	+= init_helpers.o
 obj-y	+= interrupts.o
 obj-y	+= lpc-uclass.o
+ifndef CONFIG_XPL_BUILD
 obj-y	+= mpspec.o
+endif
 obj-$(CONFIG_$(PHASE_)ACPIGEN) += acpi_nhlt.o
 obj-y	+= northbridge-uclass.o
 obj-$(CONFIG_I8259_PIC) += i8259.o
-- 
2.43.0

[PATCH v2 06/28] x86: Add some log categories

[Simon Glass]

Add categories for i8259 and bios files, so that log statements have the
right category.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/bios.c            | 3 +++
 arch/x86/lib/bios_interrupts.c | 2 ++
 arch/x86/lib/i8259.c           | 2 ++
 3 files changed, 7 insertions(+)

diff --git a/arch/x86/lib/bios.c b/arch/x86/lib/bios.c
index 03f7360032c..acae635cf19 100644
--- a/arch/x86/lib/bios.c
+++ b/arch/x86/lib/bios.c
@@ -5,6 +5,9 @@
  * Copyright (C) 2007 Advanced Micro Devices, Inc.
  * Copyright (C) 2009-2010 coresystems GmbH
  */
+
+#define LOG_CATEGRORY	LOGC_ARCH
+
 #include <compiler.h>
 #include <bios_emul.h>
 #include <irq_func.h>
diff --git a/arch/x86/lib/bios_interrupts.c b/arch/x86/lib/bios_interrupts.c
index b2cf1527b1c..3ae6c193ec7 100644
--- a/arch/x86/lib/bios_interrupts.c
+++ b/arch/x86/lib/bios_interrupts.c
@@ -7,6 +7,8 @@
  * Copyright (C) 2007-2009 coresystems GmbH
  */
 
+#define LOG_CATEGRORY	LOGC_ARCH
+
 #include <log.h>
 #include <asm/pci.h>
 #include "bios_emul.h"
diff --git a/arch/x86/lib/i8259.c b/arch/x86/lib/i8259.c
index 465ff70146f..088f78f4661 100644
--- a/arch/x86/lib/i8259.c
+++ b/arch/x86/lib/i8259.c
@@ -13,6 +13,8 @@
  * Programmable Interrupt Controllers.
  */
 
+#define LOG_CATEGORY	UCLASS_IRQ
+
 #include <log.h>
 #include <asm/io.h>
 #include <asm/i8259.h>
-- 
2.43.0

[PATCH v2 07/28] x86: Drop use of CONFIG_REALMODE_DEBUG

[Simon Glass]

This option is not actually defined in Kconfig anymore. Use a normal
debug print instead, which has a similar effect.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/bios.c            | 18 ++++++++----------
 arch/x86/lib/bios_interrupts.c |  6 ++----
 2 files changed, 10 insertions(+), 14 deletions(-)

diff --git a/arch/x86/lib/bios.c b/arch/x86/lib/bios.c
index acae635cf19..5dfe5a647eb 100644
--- a/arch/x86/lib/bios.c
+++ b/arch/x86/lib/bios.c
@@ -301,16 +301,14 @@ asmlinkage int interrupt_handler(u32 intnumber, u32 gsfs, u32 dses,
 	cs = cs_ip >> 16;
 	flags = stackflags;
 
-#ifdef CONFIG_REALMODE_DEBUG
-	debug("oprom: INT# 0x%x\n", intnumber);
-	debug("oprom: eax: %08x ebx: %08x ecx: %08x edx: %08x\n",
-	      eax, ebx, ecx, edx);
-	debug("oprom: ebp: %08x esp: %08x edi: %08x esi: %08x\n",
-	      ebp, esp, edi, esi);
-	debug("oprom:  ip: %04x      cs: %04x   flags: %08x\n",
-	      ip, cs, flags);
-	debug("oprom: stackflags = %04x\n", stackflags);
-#endif
+	log_debug("oprom: INT# 0x%x\n", intnumber);
+	log_debug("oprom: eax: %08x ebx: %08x ecx: %08x edx: %08x\n",
+		  eax, ebx, ecx, edx);
+	log_debug("oprom: ebp: %08x esp: %08x edi: %08x esi: %08x\n",
+		  ebp, esp, edi, esi);
+	log_debug("oprom:  ip: %04x      cs: %04x   flags: %08x\n",
+		  ip, cs, flags);
+	log_debug("oprom: stackflags = %04x\n", stackflags);
 
 	/*
 	 * Fetch arguments from the stack and put them to a place
diff --git a/arch/x86/lib/bios_interrupts.c b/arch/x86/lib/bios_interrupts.c
index 3ae6c193ec7..e0c2284a901 100644
--- a/arch/x86/lib/bios_interrupts.c
+++ b/arch/x86/lib/bios_interrupts.c
@@ -200,10 +200,8 @@ int int1a_handler(void)
 			dm_pci_write_config32(dev, reg, dword);
 			break;
 		}
-#ifdef CONFIG_REALMODE_DEBUG
-		debug("0x%x: bus %d devfn 0x%x reg 0x%x val 0x%x\n", func,
-		      bus, devfn, reg, M.x86.R_ECX);
-#endif
+		log_debug("0x%x: bus %d devfn 0x%x reg 0x%x val 0x%x\n", func,
+			  bus, devfn, reg, M.x86.R_ECX);
 		M.x86.R_EAX &= 0xffff00ff; /* Clear AH */
 		M.x86.R_EAX |= PCIBIOS_SUCCESSFUL;
 		retval = 1;
-- 
2.43.0

[PATCH v2 08/28] x86: Avoid clearing the VESA display

[Simon Glass]

U-Boot clears the display when it starts up, so there is no need to ask
the VESA driver to do this. Fix this and add a comment explaining the
flags.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/bios.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/lib/bios.c b/arch/x86/lib/bios.c
index 5dfe5a647eb..de4578666fb 100644
--- a/arch/x86/lib/bios.c
+++ b/arch/x86/lib/bios.c
@@ -231,7 +231,11 @@ static void vbe_set_graphics(int vesa_mode, struct vesa_state *mode_info)
 {
 	unsigned char *framebuffer;
 
-	mode_info->video_mode = (1 << 14) | vesa_mode;
+	/*
+	 * bit 14 is linear-framebuffer mode
+	 * bit 15 means don't clear the display
+	 */
+	mode_info->video_mode = (1 << 14) | (1 << 15) | vesa_mode;
 	vbe_get_mode_info(mode_info);
 
 	framebuffer = (unsigned char *)(ulong)mode_info->vesa.phys_base_ptr;
-- 
2.43.0

[PATCH v2 09/28] x86: Add 64-bit entries to the GDT

[Simon Glass]

At present it is not possible to execution 64-bit code without
installing an entire new Global Descriptor Table. This is inconvenient
since kvm does not seem to like switching into long mode with a new
table.

It isn't actually necessary, since we can just extend the existing
table. Add some new entries to this effect.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/i386/cpu.c          | 3 +++
 arch/x86/include/asm/processor.h | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index a51a24498a7..3bbad3b2eca 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -160,6 +160,9 @@ void arch_setup_gd(gd_t *new_gd)
 
 	gdt_addr[X86_GDT_ENTRY_16BIT_FLAT_CS] = GDT_ENTRY(0x809b, 0, 0xfffff);
 	gdt_addr[X86_GDT_ENTRY_16BIT_FLAT_DS] = GDT_ENTRY(0x8093, 0, 0xfffff);
+	gdt_addr[X86_GDT_ENTRY_64BIT_CS] = GDT_ENTRY(0xaf9b, 0, 0xfffff);
+	gdt_addr[X86_GDT_ENTRY_64BIT_TS1] = GDT_ENTRY(0x8980, 0, 0xfffff);
+	gdt_addr[X86_GDT_ENTRY_64BIT_TS2] = 0;
 
 	load_gdt(gdt_addr, X86_GDT_NUM_ENTRIES);
 	load_ds(X86_GDT_ENTRY_32BIT_DS);
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index d7b68367861..ad8240be387 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -18,7 +18,10 @@
 #define X86_GDT_ENTRY_16BIT_DS		6
 #define X86_GDT_ENTRY_16BIT_FLAT_CS	7
 #define X86_GDT_ENTRY_16BIT_FLAT_DS	8
-#define X86_GDT_NUM_ENTRIES		9
+#define X86_GDT_ENTRY_64BIT_CS		9
+#define X86_GDT_ENTRY_64BIT_TS1		10
+#define X86_GDT_ENTRY_64BIT_TS2		11
+#define X86_GDT_NUM_ENTRIES		12
 
 #define X86_GDT_SIZE		(X86_GDT_NUM_ENTRIES * X86_GDT_ENTRY_SIZE)
 
-- 
2.43.0

[PATCH v2 10/28] x86: Use defines for the cache flags

[Simon Glass]

Use some named flags when setting up the cache, so it is easier to see
what is going on.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/i386/cpu.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index 3bbad3b2eca..845e00ca439 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -592,6 +592,13 @@ int cpu_has_64bit(void)
 #define PAGETABLE_BASE		0x80000
 #define PAGETABLE_SIZE		(6 * 4096)
 
+#define _PRES BIT(0)	/* present */
+#define _RW   BIT(1)	/* write allowed */
+#define _US   BIT(2)	/* user-access allowed */
+#define _A    BIT(5)	/* has been accessed */
+#define _D    BIT(6)	/* has been written to */
+#define _PS   BIT(7)	/* indicates 2MB page size here */
+
 /**
  * build_pagetable() - build a flat 4GiB page table structure for 64-bti mode
  *
@@ -604,15 +611,17 @@ static void build_pagetable(uint32_t *pgtable)
 	memset(pgtable, '\0', PAGETABLE_SIZE);
 
 	/* Level 4 needs a single entry */
-	pgtable[0] = (ulong)&pgtable[1024] + 7;
+	pgtable[0] = (ulong)&pgtable[1024] + _PRES + _RW + _US + _A;
 
 	/* Level 3 has one 64-bit entry for each GiB of memory */
 	for (i = 0; i < 4; i++)
-		pgtable[1024 + i * 2] = (ulong)&pgtable[2048] + 0x1000 * i + 7;
+		pgtable[1024 + i * 2] = (ulong)&pgtable[2048] + 0x1000 * i +
+			_PRES + _RW + _US + _A;
 
 	/* Level 2 has 2048 64-bit entries, each repesenting 2MiB */
 	for (i = 0; i < 2048; i++)
-		pgtable[2048 + i * 2] = 0x183 + (i << 21UL);
+		pgtable[2048 + i * 2] = _PRES + _RW + _US + _PS + _A +  _D +
+					 (i << 21UL);
 }
 
 int cpu_jump_to_64bit(ulong setup_base, ulong target)
-- 
2.43.0

[PATCH v2 11/28] x86: spl: Drop duplicate CPU init

[Simon Glass]

x86_cpu_init_f() is called by arch_cpu_init() a few lines below this
code. Drop the duplicate call.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/spl.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/x86/lib/spl.c b/arch/x86/lib/spl.c
index 7a033505101..47cf9c862b5 100644
--- a/arch/x86/lib/spl.c
+++ b/arch/x86/lib/spl.c
@@ -84,8 +84,6 @@ static int x86_spl_init(void)
 	log_debug("x86 spl starting\n");
 	if (IS_ENABLED(TPL))
 		ret = x86_cpu_reinit_f();
-	else
-		ret = x86_cpu_init_f();
 	ret = spl_init();
 	if (ret) {
 		log_debug("spl_init() failed (err=%d)\n", ret);
-- 
2.43.0

[PATCH v2 12/28] x86: Drop the message about features missing in 64-bit

[Simon Glass]

This functions normally and has done for a while, so drop this scary
message.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/lib/spl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/lib/spl.c b/arch/x86/lib/spl.c
index 47cf9c862b5..0a6a761987e 100644
--- a/arch/x86/lib/spl.c
+++ b/arch/x86/lib/spl.c
@@ -281,7 +281,7 @@ void __noreturn jump_to_image_no_args(struct spl_image_info *spl_image)
 {
 	int ret;
 
-	printf("Jumping to 64-bit U-Boot: Note many features are missing\n");
+	log_debug("Jumping to 64-bit U-Boot\n");
 	ret = cpu_jump_to_64bit_uboot(spl_image->entry_point);
 	debug("ret=%d\n", ret);
 	hang();
-- 
2.43.0

[PATCH v2 13/28] x86: Include stdbool.h in interrupt header

[Simon Glass]

This makes use of a 'bool' type, so include the required header.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/include/asm/interrupt.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/include/asm/interrupt.h b/arch/x86/include/asm/interrupt.h
index e23fb2c8e72..c689fc23d08 100644
--- a/arch/x86/include/asm/interrupt.h
+++ b/arch/x86/include/asm/interrupt.h
@@ -10,6 +10,7 @@
 #ifndef __ASM_INTERRUPT_H_
 #define __ASM_INTERRUPT_H_ 1
 
+#include <stdbool.h>
 #include <asm/types.h>
 
 #define SYS_NUM_IRQS	16
-- 
2.43.0

[PATCH v2 14/28] x86: Tidy up the GDT size in start/16.S

[Simon Glass]

Use a symbol to select the size of the GDT, rather than hard-coding a
value. This matches how it is done in start64

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/start.S   | 4 +++-
 arch/x86/cpu/start16.S | 3 ++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/arch/x86/cpu/start.S b/arch/x86/cpu/start.S
index 0ef27cc5a00..385a691265e 100644
--- a/arch/x86/cpu/start.S
+++ b/arch/x86/cpu/start.S
@@ -254,7 +254,7 @@ multiboot_header:
 	 * GDT is setup in a safe location in RAM
 	 */
 gdt_ptr2:
-	.word	0x1f		/* limit (31 bytes = 4 GDT entries - 1) */
+	.word	gdt2_end - gdt_ptr2 - 1
 	.long	gdt_rom2	/* base */
 
 	/* Some CPUs are picky about GDT alignment... */
@@ -313,4 +313,6 @@ gdt_rom2:
 	.byte	0x93		/* access */
 	.byte	0xcf		/* flags + limit_high */
 	.byte	0x00		/* base_high */
+gdt2_end:
+
 #endif
diff --git a/arch/x86/cpu/start16.S b/arch/x86/cpu/start16.S
index 865a49731e5..8d9acb193e0 100644
--- a/arch/x86/cpu/start16.S
+++ b/arch/x86/cpu/start16.S
@@ -61,7 +61,7 @@ idt_ptr:
 	 * GDT is setup in a safe location in RAM
 	 */
 gdt_ptr:
-	.word	0x1f		/* limit (31 bytes = 4 GDT entries - 1) */
+	.word	gdt_end - gdt_rom - 1
 	.long	BOOT_SEG + gdt_rom	/* base */
 
 	/* Some CPUs are picky about GDT alignment... */
@@ -120,3 +120,4 @@ gdt_rom:
 	.byte	0x93		/* access */
 	.byte	0xcf		/* flags + limit_high */
 	.byte	0x00		/* base_high */
+gdt_end:
-- 
2.43.0

[PATCH v2 15/28] x86: Disable paging before changing to long mode

[Simon Glass]

This is required as part of the procedure. The existing code works
because it changes the GDT at the same time, but this makes kvm
unhappy.

Update the algorithm to disable and then re-enable paging.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/i386/call64.S | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/x86/cpu/i386/call64.S b/arch/x86/cpu/i386/call64.S
index d81bcc6f8f4..e5a0420d1ba 100644
--- a/arch/x86/cpu/i386/call64.S
+++ b/arch/x86/cpu/i386/call64.S
@@ -25,6 +25,11 @@ cpu_call64:
 	push	%edx		/* arg1 = setup_base */
 	mov	%eax, %ebx
 
+	# disable paging
+	movl	%cr0, %eax
+	andl	$~X86_CR0_PG, %eax
+	movl	%eax, %cr0
+
 	/* Load new GDT with the 64bit segments using 32bit descriptor */
 	leal	gdt, %eax
 	movl	%eax, gdt+2
@@ -67,7 +72,8 @@ cpu_call64:
 	pushl	%eax
 
 	/* Enter paged protected Mode, activating Long Mode */
-	movl	$(X86_CR0_PG | X86_CR0_PE), %eax
+	movl	%cr0, %eax
+	orl	$X86_CR0_PG, %eax
 	movl	%eax, %cr0
 
 	/* Jump from 32bit compatibility mode into 64bit mode. */
-- 
2.43.0

[PATCH v2 16/28] x86: Use the same GDT when jumping to long mode

[Simon Glass]

Make use the existing GDT which now includes entries for 64-bit code.
Leave the interrupt descriptors alone. They can be tidied up once U-Boot
starts up.

With this, kvm mode works with QEMU.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: https://source.denx.de/u-boot/custodians/u-boot-dm/-/issues/31
---

(no changes since v1)

 arch/x86/cpu/i386/call64.S | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)

diff --git a/arch/x86/cpu/i386/call64.S b/arch/x86/cpu/i386/call64.S
index e5a0420d1ba..3137ec17d31 100644
--- a/arch/x86/cpu/i386/call64.S
+++ b/arch/x86/cpu/i386/call64.S
@@ -7,6 +7,7 @@
  */
 
 #include <asm/msr-index.h>
+#include <asm/processor.h>
 #include <asm/processor-flags.h>
 
 .code32
@@ -30,11 +31,6 @@ cpu_call64:
 	andl	$~X86_CR0_PG, %eax
 	movl	%eax, %cr0
 
-	/* Load new GDT with the 64bit segments using 32bit descriptor */
-	leal	gdt, %eax
-	movl	%eax, gdt+2
-	lgdt	gdt
-
 	/* Enable PAE mode */
 	movl	$(X86_CR4_PAE), %eax
 	movl	%eax, %cr4
@@ -49,12 +45,6 @@ cpu_call64:
 	btsl	$_EFER_LME, %eax
 	wrmsr
 
-	/* After gdt is loaded */
-	xorl	%eax, %eax
-	lldt	%ax
-	movl    $0x20, %eax
-	ltr	%ax
-
 	/*
 	 * Setup for the jump to 64bit mode
 	 *
@@ -67,7 +57,7 @@ cpu_call64:
 	 */
 	pop	%esi			/* setup_base */
 
-	pushl	$0x10
+	pushl	$(X86_GDT_ENTRY_64BIT_CS * X86_GDT_ENTRY_SIZE)
 	leal	lret_target, %eax
 	pushl	%eax
 
-- 
2.43.0

[PATCH v2 17/28] x86: Use a simple jump into long mode

[Simon Glass]

With the 64-bit descriptor we can use a jump instruction, rather than
pushing things on the stack.

Since the processor is in 64-bit mode by this point, pop a 64-bit value
from the stack, containing the target address.

This simplifies the code slightly, in particular its use of the stack.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/i386/call64.S | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/arch/x86/cpu/i386/call64.S b/arch/x86/cpu/i386/call64.S
index 3137ec17d31..c6308b92e25 100644
--- a/arch/x86/cpu/i386/call64.S
+++ b/arch/x86/cpu/i386/call64.S
@@ -22,6 +22,7 @@ cpu_call64:
 	 * ecx - target
 	 */
 	cli
+	pushl	$0		/* top 64-bits of target */
 	push	%ecx		/* arg2 = target */
 	push	%edx		/* arg1 = setup_base */
 	mov	%eax, %ebx
@@ -32,7 +33,8 @@ cpu_call64:
 	movl	%eax, %cr0
 
 	/* Enable PAE mode */
-	movl	$(X86_CR4_PAE), %eax
+	movl	%cr4, %eax
+	orl	$X86_CR4_PAE, %eax
 	movl	%eax, %cr4
 
 	/* Enable the boot page tables */
@@ -57,23 +59,18 @@ cpu_call64:
 	 */
 	pop	%esi			/* setup_base */
 
-	pushl	$(X86_GDT_ENTRY_64BIT_CS * X86_GDT_ENTRY_SIZE)
-	leal	lret_target, %eax
-	pushl	%eax
-
 	/* Enter paged protected Mode, activating Long Mode */
 	movl	%cr0, %eax
 	orl	$X86_CR0_PG, %eax
 	movl	%eax, %cr0
 
 	/* Jump from 32bit compatibility mode into 64bit mode. */
-	lret
+	ljmp	$(X86_GDT_ENTRY_64BIT_CS * X86_GDT_ENTRY_SIZE), $lret_target
 
-code64:
+.code64
 lret_target:
-	pop	%eax			/* target */
-	mov	%eax, %eax		/* Clear bits 63:32 */
-	jmp	*%eax			/* Jump to the 64-bit target */
+	pop	%rax			/* target */
+	jmp	*%rax			/* Jump to the 64-bit target */
 
 .globl call64_stub_size
 call64_stub_size:
-- 
2.43.0

[PATCH v2 18/28] x86: Rename the _D dirty flag

[Simon Glass]

This value happens to be used by ctype.h so chose a different name.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to rename the _D dirty flag

 arch/x86/cpu/i386/cpu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index 845e00ca439..b108f724c51 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -596,7 +596,7 @@ int cpu_has_64bit(void)
 #define _RW   BIT(1)	/* write allowed */
 #define _US   BIT(2)	/* user-access allowed */
 #define _A    BIT(5)	/* has been accessed */
-#define _D    BIT(6)	/* has been written to */
+#define _DT   BIT(6)	/* has been written to */
 #define _PS   BIT(7)	/* indicates 2MB page size here */
 
 /**
@@ -620,7 +620,7 @@ static void build_pagetable(uint32_t *pgtable)
 
 	/* Level 2 has 2048 64-bit entries, each repesenting 2MiB */
 	for (i = 0; i < 2048; i++)
-		pgtable[2048 + i * 2] = _PRES + _RW + _US + _PS + _A +  _D +
+		pgtable[2048 + i * 2] = _PRES + _RW + _US + _PS + _A +  _DT +
 					 (i << 21UL);
 }
 
-- 
2.43.0

[PATCH v2 19/28] x86: Support CPU functions in long mode

[Simon Glass]

At present it is not possible to find out the physical-address size in
long mode, so a predefined value is used.

Update the macros to support this properly, since it is important when
programming MTRRs.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to support CPU functions in long mode

 arch/x86/cpu/cpu.c         | 24 ++++++++++++++++++++++++
 arch/x86/cpu/i386/cpu.c    | 23 -----------------------
 arch/x86/cpu/x86_64/cpu.c  |  5 -----
 arch/x86/include/asm/cpu.h | 35 ++++++++++++++++++++++++++++++-----
 4 files changed, 54 insertions(+), 33 deletions(-)

diff --git a/arch/x86/cpu/cpu.c b/arch/x86/cpu/cpu.c
index a8b21406ac0..c373b14df30 100644
--- a/arch/x86/cpu/cpu.c
+++ b/arch/x86/cpu/cpu.c
@@ -364,3 +364,27 @@ long locate_coreboot_table(void)
 
 	return addr;
 }
+
+static bool has_cpuid(void)
+{
+	return flag_is_changeable_p(X86_EFLAGS_ID);
+}
+
+static uint cpu_cpuid_extended_level(void)
+{
+	return cpuid_eax(0x80000000);
+}
+
+int cpu_phys_address_size(void)
+{
+	if (!has_cpuid())
+		return 32;
+
+	if (cpu_cpuid_extended_level() >= 0x80000008)
+		return cpuid_eax(0x80000008) & 0xff;
+
+	if (cpuid_edx(1) & (CPUID_FEATURE_PAE | CPUID_FEATURE_PSE36))
+		return 36;
+
+	return 32;
+}
diff --git a/arch/x86/cpu/i386/cpu.c b/arch/x86/cpu/i386/cpu.c
index b108f724c51..ee6dbeb5c48 100644
--- a/arch/x86/cpu/i386/cpu.c
+++ b/arch/x86/cpu/i386/cpu.c
@@ -35,10 +35,6 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
-#define CPUID_FEATURE_PAE	BIT(6)
-#define CPUID_FEATURE_PSE36	BIT(17)
-#define CPUID_FEAURE_HTT	BIT(28)
-
 /*
  * Constructor for a conventional segment GDT (or LDT) entry
  * This is a macro so it can be used in initialisers
@@ -412,25 +408,6 @@ static void setup_identity(void)
 	}
 }
 
-static uint cpu_cpuid_extended_level(void)
-{
-	return cpuid_eax(0x80000000);
-}
-
-int cpu_phys_address_size(void)
-{
-	if (!has_cpuid())
-		return 32;
-
-	if (cpu_cpuid_extended_level() >= 0x80000008)
-		return cpuid_eax(0x80000008) & 0xff;
-
-	if (cpuid_edx(1) & (CPUID_FEATURE_PAE | CPUID_FEATURE_PSE36))
-		return 36;
-
-	return 32;
-}
-
 static void setup_mtrr(void)
 {
 	u64 mtrr_cap;
diff --git a/arch/x86/cpu/x86_64/cpu.c b/arch/x86/cpu/x86_64/cpu.c
index 71bc07f872a..25ae92c702f 100644
--- a/arch/x86/cpu/x86_64/cpu.c
+++ b/arch/x86/cpu/x86_64/cpu.c
@@ -59,11 +59,6 @@ int x86_cpu_reinit_f(void)
 	return 0;
 }
 
-int cpu_phys_address_size(void)
-{
-	return CONFIG_CPU_ADDR_BITS;
-}
-
 int x86_cpu_init_f(void)
 {
 	return 0;
diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h
index fd389d4024c..53f9226112a 100644
--- a/arch/x86/include/asm/cpu.h
+++ b/arch/x86/include/asm/cpu.h
@@ -58,6 +58,10 @@ enum {
 	X86_SYSCON_PUNIT,	/* Power unit */
 };
 
+#define CPUID_FEATURE_PAE	BIT(6)
+#define CPUID_FEATURE_PSE36	BIT(17)
+#define CPUID_FEAURE_HTT	BIT(28)
+
 struct cpuid_result {
 	uint32_t eax;
 	uint32_t ebx;
@@ -161,12 +165,33 @@ static inline unsigned int cpuid_edx(unsigned int op)
 	return edx;
 }
 
-#if !CONFIG_IS_ENABLED(X86_64)
+#if CONFIG_IS_ENABLED(X86_64)
+/* Standard macro to see if a specific flag is changeable */
+static inline int flag_is_changeable_p(u64 flag)
+{
+	u64 f1, f2;
+
+	asm (
+		"pushfq\n\t"
+		"pushfq\n\t"
+		"popq %0\n\t"
+		"movq %0,%1\n\t"
+		"xorq %2,%0\n\t"
+		"pushq %0\n\t"
+		"popfq\n\t"
+		"pushfq\n\t"
+		"popq %0\n\t"
+		"popfq\n\t"
+		: "=&r" (f1), "=&r" (f2)
+		: "ir" (flag));
 
+	return ((f1 ^ f2) & flag) != 0;
+}
+#else
 /* Standard macro to see if a specific flag is changeable */
-static inline int flag_is_changeable_p(uint32_t flag)
+static inline int flag_is_changeable_p(u32 flag)
 {
-	uint32_t f1, f2;
+	u32 f1, f2;
 
 	asm(
 		"pushfl\n\t"
@@ -181,9 +206,9 @@ static inline int flag_is_changeable_p(uint32_t flag)
 		"popfl\n\t"
 		: "=&r" (f1), "=&r" (f2)
 		: "ir" (flag));
-	return ((f1^f2) & flag) != 0;
+	return ((f1 ^ f2) & flag) != 0;
 }
-#endif
+#endif /* X86_64 */
 
 /**
  * cpu_enable_paging_pae() - Enable PAE-paging
-- 
2.43.0

[PATCH v2 20/28] x86: Tidy up address size in MTRR calculations

[Simon Glass]

The CONFIG value should only be used when it cannot be read from the
CPU. This is particularly important when using kvm since the CPU size
depends on the host CPU.

Update set_var_mtrr() to use the correct function for this, which now
supports 64-bit operation. Ensure that the lower bits of the mask are
zeroed as required.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to tidy up address size in MTRR calculations

 arch/x86/cpu/mtrr.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
index 07ea89162de..d05263c8eab 100644
--- a/arch/x86/cpu/mtrr.c
+++ b/arch/x86/cpu/mtrr.c
@@ -16,6 +16,7 @@
  * since the MTRR registers are sometimes in flux.
  */
 
+#include <cpu.h>
 #include <cpu_func.h>
 #include <log.h>
 #include <sort.h>
@@ -70,9 +71,10 @@ static void set_var_mtrr(uint reg, uint type, uint64_t start, uint64_t size)
 {
 	u64 mask;
 
-	wrmsrl(MTRR_PHYS_BASE_MSR(reg), start | type);
 	mask = ~(size - 1);
-	mask &= (1ULL << CONFIG_CPU_ADDR_BITS) - 1;
+	mask &= (1ull << cpu_phys_address_size()) - 1;
+
+	wrmsrl(MTRR_PHYS_BASE_MSR(reg), start | type);
 	wrmsrl(MTRR_PHYS_MASK_MSR(reg), mask | MTRR_PHYS_MASK_VALID);
 }
 
@@ -205,7 +207,7 @@ int mtrr_add_request(int type, uint64_t start, uint64_t size)
 	debug("%d: type=%d, %08llx  %08llx\n", gd->arch.mtrr_req_count - 1,
 	      req->type, req->start, req->size);
 	mask = ~(req->size - 1);
-	mask &= (1ULL << CONFIG_CPU_ADDR_BITS) - 1;
+	mask &= (1ULL << cpu_phys_address_size()) - 1;
 	mask |= MTRR_PHYS_MASK_VALID;
 	debug("   %016llx %016llx\n", req->start | req->type, mask);
 
@@ -360,7 +362,7 @@ int mtrr_list(int reg_count, int cpu_select)
 
 		base = info.mtrr[i].base;
 		mask = info.mtrr[i].mask;
-		size = ~mask & ((1ULL << CONFIG_CPU_ADDR_BITS) - 1);
+		size = ~mask & ((1ULL << cpu_phys_address_size()) - 1);
 		size |= (1 << 12) - 1;
 		size += 1;
 		valid = mask & MTRR_PHYS_MASK_VALID;
-- 
2.43.0

[PATCH v2 21/28] Add a 64-bit version of is_power_of_2()

[Simon Glass]

The existing function only works with ulong which is not enough on
32-bit x86 machines which need to set up MTRRs up to 4GB.

There doesn't seem to be a 64-bit version in Linux, so add one here.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch with a 64-bit version of is_power_of_2()

 include/linux/log2.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/include/linux/log2.h b/include/linux/log2.h
index d4e32ecfc64..89097cdc694 100644
--- a/include/linux/log2.h
+++ b/include/linux/log2.h
@@ -52,6 +52,12 @@ bool is_power_of_2(unsigned long n)
 	return (n != 0 && ((n & (n - 1)) == 0));
 }
 
+static inline __attribute__((const))
+bool is_power_of_2_u64(u64 n)
+{
+	return (n != 0 && ((n & (n - 1)) == 0));
+}
+
 /**
  * __roundup_pow_of_two() - round up to nearest power of two
  * @n: value to round up
-- 
2.43.0

[PATCH v2 22/28] x86: Support MTRRs of 4GB on 32-bit machines

[Simon Glass]

Use the 64-bit function to detect a power-of-two. This allows use of
a full 4GB MTRR on 32-bit machines.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 arch/x86/cpu/mtrr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/cpu/mtrr.c b/arch/x86/cpu/mtrr.c
index d05263c8eab..5eab3cc1953 100644
--- a/arch/x86/cpu/mtrr.c
+++ b/arch/x86/cpu/mtrr.c
@@ -195,7 +195,7 @@ int mtrr_add_request(int type, uint64_t start, uint64_t size)
 	if (!gd->arch.has_mtrr)
 		return -ENOSYS;
 
-	if (!is_power_of_2(size))
+	if (!is_power_of_2_u64(size))
 		return -EINVAL;
 
 	if (gd->arch.mtrr_req_count == MAX_MTRR_REQUESTS)
-- 
2.43.0

[PATCH v2 23/28] x86: emulation: Set an MTRR for the RAM

[Simon Glass]

QEMU likes to have an MTRR set up, just like real machines. Add an MTRR
which covers the total RAM size.

This does nothing on machines without MTRRs.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to set an MTRR for the RAM in QEMU

 arch/x86/cpu/qemu/dram.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/x86/cpu/qemu/dram.c b/arch/x86/cpu/qemu/dram.c
index 62a301c0fd3..efacc7b3457 100644
--- a/arch/x86/cpu/qemu/dram.c
+++ b/arch/x86/cpu/qemu/dram.c
@@ -4,7 +4,9 @@
  */
 
 #include <init.h>
+#include <spl.h>
 #include <asm/global_data.h>
+#include <asm/mtrr.h>
 #include <asm/post.h>
 #include <asm/arch/qemu.h>
 #include <linux/sizes.h>
@@ -40,10 +42,23 @@ u64 qemu_get_high_memory_size(void)
 
 int dram_init(void)
 {
+	int ret;
+
 	gd->ram_size = qemu_get_low_memory_size();
 	gd->ram_size += qemu_get_high_memory_size();
 	post_code(POST_DRAM);
 
+	if (xpl_phase() == PHASE_BOARD_F) {
+		ret = mtrr_add_request(MTRR_TYPE_WRBACK, 0, gd->ram_size);
+		if (ret != -ENOSYS) {
+			if (ret)
+				return log_msg_ret("mta", ret);
+			ret = mtrr_commit(false);
+			if (ret)
+				return log_msg_ret("mtc", ret);
+		}
+	}
+
 	return 0;
 }
 
-- 
2.43.0

[PATCH v2 24/28] scripts: Expand a few options

[Simon Glass]

This gets the script to where it can boot with kvm:

- Use -cpu host with kvm so that the host CPU is used
- Show the full QEMU arguments

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/build-qemu.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/build-qemu.sh b/scripts/build-qemu.sh
index f9ee0072573..03f06105246 100755
--- a/scripts/build-qemu.sh
+++ b/scripts/build-qemu.sh
@@ -93,7 +93,7 @@ while getopts "a:Beko:rR:sS:w" opt; do
 		extra+=" -netdev user,id=net0"
 		;;
 	k)
-		kvm="-enable-kvm"
+		kvm="-enable-kvm -cpu host"
 		;;
 	o)
 		os=$OPTARG
@@ -150,7 +150,7 @@ run_qemu() {
 	else
 		extra+=" -serial mon:stdio"
 	fi
-	echo "Running ${qemu} ${extra}"
+	echo "Running ${qemu} -bios "$DIR/${BIOS}" ${kvm} ${extra}"
 	"${qemu}" -bios "$DIR/${BIOS}" \
 		-m 512 \
 		-nic none \
-- 
2.43.0

[PATCH v2 25/28] test/py: Add a helper to send characters

[Simon Glass]

The existing run_command() method is not great for sending things other
than U-Boot commands. Add a helper for sending arbitrary strings as well
as control characters.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch with a helper to send characters

 test/py/console_base.py | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/test/py/console_base.py b/test/py/console_base.py
index b080c389547..073ab0462dc 100644
--- a/test/py/console_base.py
+++ b/test/py/console_base.py
@@ -369,21 +369,30 @@ class ConsoleBase(object):
             output.append(self.run_command(cmd))
         return output
 
-    def ctrlc(self):
-        """Send a CTRL-C character to U-Boot.
+    def send(self, msg):
+        """Send characters without waiting for echo, etc."""
+        self.run_command(msg, wait_for_prompt=False, wait_for_echo=False,
+                         send_nl=False)
+
+    def ctrl(self, char):
+        """Send a CTRL- character to U-Boot.
 
         This is useful in order to stop execution of long-running synchronous
         commands such as "ums".
 
         Args:
-            None.
-
-        Returns:
-            Nothing.
+            char (str): Character to send, e.g. 'C' to send Ctrl-C
         """
+        self.log.action(f'Sending Ctrl-{char}')
+        self.send(chr(ord(char) - ord('@')))
 
-        self.log.action('Sending Ctrl-C')
-        self.run_command(chr(3), wait_for_echo=False, send_nl=False)
+    def ctrlc(self):
+        """Send a CTRL-C character to U-Boot.
+
+        This is useful in order to stop execution of long-running synchronous
+        commands such as "ums".
+        """
+        self.ctrl('C')
 
     def wait_for(self, text):
         """Wait for a pattern to be emitted by U-Boot.
-- 
2.43.0

[PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Simon Glass]

Some test can only be run by a particular board in a lab, e.g. because
they are loaded with an OS image used by the test. Add a way to specify
this in tests.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to allow tests to be filtered by role

 test/py/conftest.py | 22 ++++++++++++++++++++++
 test/py/pytest.ini  |  1 +
 2 files changed, 23 insertions(+)

diff --git a/test/py/conftest.py b/test/py/conftest.py
index e59897c1f78..5aea85647af 100644
--- a/test/py/conftest.py
+++ b/test/py/conftest.py
@@ -334,6 +334,7 @@ def pytest_configure(config):
     ubconfig.dtb = build_dir + '/arch/sandbox/dts/test.dtb'
     ubconfig.connection_ok = True
     ubconfig.timing = config.getoption('timing')
+    ubconfig.role = config.getoption('role')
 
     env_vars = (
         'board_type',
@@ -760,6 +761,26 @@ def setup_singlethread(item):
         if worker_id and worker_id != 'master':
             pytest.skip('must run single-threaded')
 
+def setup_role(item):
+    """Process any 'role' marker for a test.
+
+    Skip this test if the role does not match.
+
+    Args:
+        item (pytest.Item): The pytest test item
+    """
+    required_roles = []
+    for roles in item.iter_markers('role'):
+        role = roles.args[0]
+        if role.startswith('!'):
+            if ubconfig.role == role[1:]:
+                pytest.skip(f'role "{ubconfig.role}" not supported')
+                return
+        else:
+            required_roles.append(role)
+    if required_roles and ubconfig.role not in required_roles:
+        pytest.skip(f'board "{ubconfig.role}" not supported')
+
 def start_test_section(item):
     anchors[item.name] = log.start_section(item.name)
 
@@ -781,6 +802,7 @@ def pytest_runtest_setup(item):
     setup_buildconfigspec(item)
     setup_requiredtool(item)
     setup_singlethread(item)
+    setup_role(item)
 
 def pytest_runtest_protocol(item, nextitem):
     """pytest hook: Called to execute a test.
diff --git a/test/py/pytest.ini b/test/py/pytest.ini
index 26d83f83e00..361be0178ee 100644
--- a/test/py/pytest.ini
+++ b/test/py/pytest.ini
@@ -12,3 +12,4 @@ markers =
     requiredtool: U-Boot: Required host tools for a test.
     slow: U-Boot: Specific test will run slowly.
     singlethread: Cannot run in parallel
+    role: U-Boot: Indicates the lab 'role' which can execute this test
-- 
2.43.0

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:

> Some test can only be run by a particular board in a lab, e.g. because
> they are loaded with an OS image used by the test. Add a way to specify
> this in tests.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
> Changes in v2:
> - Add new patch to allow tests to be filtered by role
> 
>  test/py/conftest.py | 22 ++++++++++++++++++++++
>  test/py/pytest.ini  |  1 +
>  2 files changed, 23 insertions(+)

This seems duplicative of the existing functionality we have, all
hardware tests can only be run on platforms which configure what the
test requires, in the boardenv file.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Simon Glass]

Hi Tom,

On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
>
> On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
>
> > Some test can only be run by a particular board in a lab, e.g. because
> > they are loaded with an OS image used by the test. Add a way to specify
> > this in tests.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > Changes in v2:
> > - Add new patch to allow tests to be filtered by role
> >
> >  test/py/conftest.py | 22 ++++++++++++++++++++++
> >  test/py/pytest.ini  |  1 +
> >  2 files changed, 23 insertions(+)
>
> This seems duplicative of the existing functionality we have, all
> hardware tests can only be run on platforms which configure what the
> test requires, in the boardenv file.

I was afraid you would say that, and yes you are right, it is duplicative.

With my lab I am trying to use Labgrid's configuration and not need
anything else. For example, the params on how to launch QEMU and where
to find the OS file are there, rather than in u-boot-test-hooks. It
allows me to run a session interactively just as easily (and with the
same flow) as I run the pytests and gitlab.

In my lab, I actually don't have any Python things, apart from what I
added for that tftp test.

I wonder if I could create a way to pass the env__ stuff in from Labgrid?

Regards,
Simon

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> >
> > > Some test can only be run by a particular board in a lab, e.g. because
> > > they are loaded with an OS image used by the test. Add a way to specify
> > > this in tests.
> > >
> > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > ---
> > >
> > > Changes in v2:
> > > - Add new patch to allow tests to be filtered by role
> > >
> > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > >  test/py/pytest.ini  |  1 +
> > >  2 files changed, 23 insertions(+)
> >
> > This seems duplicative of the existing functionality we have, all
> > hardware tests can only be run on platforms which configure what the
> > test requires, in the boardenv file.
> 
> I was afraid you would say that, and yes you are right, it is duplicative.
> 
> With my lab I am trying to use Labgrid's configuration and not need
> anything else. For example, the params on how to launch QEMU and where
> to find the OS file are there, rather than in u-boot-test-hooks. It
> allows me to run a session interactively just as easily (and with the
> same flow) as I run the pytests and gitlab.
> 
> In my lab, I actually don't have any Python things, apart from what I
> added for that tftp test.
> 
> I wonder if I could create a way to pass the env__ stuff in from Labgrid?

Well, I was also afraid that was going to be part of your answer. A
challenge I see with the way you're using labgrid is that details are
kept there, instead. Is there some way to publish your configs so
they're also available for reference? I indeed have been lax in doing so
for the way I use it and so posted:
https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
yesterday.

But then also yes, I think you need to figure out how to have boardenv
files so that you can have pytest tests that read from the config that
they expect.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Simon Glass]

Hi Tom,

On Tue, 18 Feb 2025 at 08:12, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> > >
> > > > Some test can only be run by a particular board in a lab, e.g. because
> > > > they are loaded with an OS image used by the test. Add a way to specify
> > > > this in tests.
> > > >
> > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > ---
> > > >
> > > > Changes in v2:
> > > > - Add new patch to allow tests to be filtered by role
> > > >
> > > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > > >  test/py/pytest.ini  |  1 +
> > > >  2 files changed, 23 insertions(+)
> > >
> > > This seems duplicative of the existing functionality we have, all
> > > hardware tests can only be run on platforms which configure what the
> > > test requires, in the boardenv file.
> >
> > I was afraid you would say that, and yes you are right, it is duplicative.
> >
> > With my lab I am trying to use Labgrid's configuration and not need
> > anything else. For example, the params on how to launch QEMU and where
> > to find the OS file are there, rather than in u-boot-test-hooks. It
> > allows me to run a session interactively just as easily (and with the
> > same flow) as I run the pytests and gitlab.
> >
> > In my lab, I actually don't have any Python things, apart from what I
> > added for that tftp test.
> >
> > I wonder if I could create a way to pass the env__ stuff in from Labgrid?
>
> Well, I was also afraid that was going to be part of your answer. A
> challenge I see with the way you're using labgrid is that details are
> kept there, instead. Is there some way to publish your configs so
> they're also available for reference? I indeed have been lax in doing so
> for the way I use it and so posted:
> https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
> yesterday.

Yes, that's great. It is a big help to have all the lab stuff in one
place and not being able to get things into that repo was one reason I
gave up with my local lab that used the hook scripts. Just too many
local branches and hacks, and too hard.

Anyway, yes I'd like to get it in u-boot-test-hooks. I last pushesd it
to Labgrid 7 months ago [1]. I thought I did it at the end of November
but must have imagined it. Perhaps I could put the config files, etc.
into u-boot-test somewhere, like in a labgrid/sjg directory? Could you
give me access to that repo?

>
> But then also yes, I think you need to figure out how to have boardenv
> files so that you can have pytest tests that read from the config that
> they expect.

I can have pytest read the config from Labgrid, like it does today.
Needs some thought though.

Regards,
Simon

[1] https://github.com/labgrid-project/labgrid/pull/1411

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]

On Tue, Feb 18, 2025 at 05:04:09PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 18 Feb 2025 at 08:12, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> > > >
> > > > > Some test can only be run by a particular board in a lab, e.g. because
> > > > > they are loaded with an OS image used by the test. Add a way to specify
> > > > > this in tests.
> > > > >
> > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > ---
> > > > >
> > > > > Changes in v2:
> > > > > - Add new patch to allow tests to be filtered by role
> > > > >
> > > > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > > > >  test/py/pytest.ini  |  1 +
> > > > >  2 files changed, 23 insertions(+)
> > > >
> > > > This seems duplicative of the existing functionality we have, all
> > > > hardware tests can only be run on platforms which configure what the
> > > > test requires, in the boardenv file.
> > >
> > > I was afraid you would say that, and yes you are right, it is duplicative.
> > >
> > > With my lab I am trying to use Labgrid's configuration and not need
> > > anything else. For example, the params on how to launch QEMU and where
> > > to find the OS file are there, rather than in u-boot-test-hooks. It
> > > allows me to run a session interactively just as easily (and with the
> > > same flow) as I run the pytests and gitlab.
> > >
> > > In my lab, I actually don't have any Python things, apart from what I
> > > added for that tftp test.
> > >
> > > I wonder if I could create a way to pass the env__ stuff in from Labgrid?
> >
> > Well, I was also afraid that was going to be part of your answer. A
> > challenge I see with the way you're using labgrid is that details are
> > kept there, instead. Is there some way to publish your configs so
> > they're also available for reference? I indeed have been lax in doing so
> > for the way I use it and so posted:
> > https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
> > yesterday.
> 
> Yes, that's great. It is a big help to have all the lab stuff in one
> place and not being able to get things into that repo was one reason I
> gave up with my local lab that used the hook scripts. Just too many
> local branches and hacks, and too hard.
> 
> Anyway, yes I'd like to get it in u-boot-test-hooks. I last pushesd it
> to Labgrid 7 months ago [1]. I thought I did it at the end of November
> but must have imagined it. Perhaps I could put the config files, etc.
> into u-boot-test somewhere, like in a labgrid/sjg directory? Could you
> give me access to that repo?

I've updated the permissions and you should update the "bin/ellesmere"
directory and add "py/ellesmere".

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Simon Glass]

Hi Tom,

On Tue, 18 Feb 2025 at 17:25, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 18, 2025 at 05:04:09PM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 18 Feb 2025 at 08:12, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> > > > >
> > > > > > Some test can only be run by a particular board in a lab, e.g. because
> > > > > > they are loaded with an OS image used by the test. Add a way to specify
> > > > > > this in tests.
> > > > > >
> > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > ---
> > > > > >
> > > > > > Changes in v2:
> > > > > > - Add new patch to allow tests to be filtered by role
> > > > > >
> > > > > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > > > > >  test/py/pytest.ini  |  1 +
> > > > > >  2 files changed, 23 insertions(+)
> > > > >
> > > > > This seems duplicative of the existing functionality we have, all
> > > > > hardware tests can only be run on platforms which configure what the
> > > > > test requires, in the boardenv file.
> > > >
> > > > I was afraid you would say that, and yes you are right, it is duplicative.
> > > >
> > > > With my lab I am trying to use Labgrid's configuration and not need
> > > > anything else. For example, the params on how to launch QEMU and where
> > > > to find the OS file are there, rather than in u-boot-test-hooks. It
> > > > allows me to run a session interactively just as easily (and with the
> > > > same flow) as I run the pytests and gitlab.
> > > >
> > > > In my lab, I actually don't have any Python things, apart from what I
> > > > added for that tftp test.
> > > >
> > > > I wonder if I could create a way to pass the env__ stuff in from Labgrid?
> > >
> > > Well, I was also afraid that was going to be part of your answer. A
> > > challenge I see with the way you're using labgrid is that details are
> > > kept there, instead. Is there some way to publish your configs so
> > > they're also available for reference? I indeed have been lax in doing so
> > > for the way I use it and so posted:
> > > https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
> > > yesterday.
> >
> > Yes, that's great. It is a big help to have all the lab stuff in one
> > place and not being able to get things into that repo was one reason I
> > gave up with my local lab that used the hook scripts. Just too many
> > local branches and hacks, and too hard.
> >
> > Anyway, yes I'd like to get it in u-boot-test-hooks. I last pushesd it
> > to Labgrid 7 months ago [1]. I thought I did it at the end of November
> > but must have imagined it. Perhaps I could put the config files, etc.
> > into u-boot-test somewhere, like in a labgrid/sjg directory? Could you
> > give me access to that repo?
>
> I've updated the permissions and you should update the "bin/ellesmere"
> directory and add "py/ellesmere".

OK, thanks, will do. Where should the labgrid config go? It is three
files and perhaps a few systemd scripts. See [2]

Regards,
Simon

[2] https://github.com/labgrid-project/labgrid/pull/1411/commits/6d3b4a0cfc7e35e0547dcd14065328df4b4b5e1f

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 3612 bytes --]

On Tue, Feb 18, 2025 at 05:39:53PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 18 Feb 2025 at 17:25, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Feb 18, 2025 at 05:04:09PM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Tue, 18 Feb 2025 at 08:12, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> > > > > >
> > > > > > > Some test can only be run by a particular board in a lab, e.g. because
> > > > > > > they are loaded with an OS image used by the test. Add a way to specify
> > > > > > > this in tests.
> > > > > > >
> > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > ---
> > > > > > >
> > > > > > > Changes in v2:
> > > > > > > - Add new patch to allow tests to be filtered by role
> > > > > > >
> > > > > > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > > > > > >  test/py/pytest.ini  |  1 +
> > > > > > >  2 files changed, 23 insertions(+)
> > > > > >
> > > > > > This seems duplicative of the existing functionality we have, all
> > > > > > hardware tests can only be run on platforms which configure what the
> > > > > > test requires, in the boardenv file.
> > > > >
> > > > > I was afraid you would say that, and yes you are right, it is duplicative.
> > > > >
> > > > > With my lab I am trying to use Labgrid's configuration and not need
> > > > > anything else. For example, the params on how to launch QEMU and where
> > > > > to find the OS file are there, rather than in u-boot-test-hooks. It
> > > > > allows me to run a session interactively just as easily (and with the
> > > > > same flow) as I run the pytests and gitlab.
> > > > >
> > > > > In my lab, I actually don't have any Python things, apart from what I
> > > > > added for that tftp test.
> > > > >
> > > > > I wonder if I could create a way to pass the env__ stuff in from Labgrid?
> > > >
> > > > Well, I was also afraid that was going to be part of your answer. A
> > > > challenge I see with the way you're using labgrid is that details are
> > > > kept there, instead. Is there some way to publish your configs so
> > > > they're also available for reference? I indeed have been lax in doing so
> > > > for the way I use it and so posted:
> > > > https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
> > > > yesterday.
> > >
> > > Yes, that's great. It is a big help to have all the lab stuff in one
> > > place and not being able to get things into that repo was one reason I
> > > gave up with my local lab that used the hook scripts. Just too many
> > > local branches and hacks, and too hard.
> > >
> > > Anyway, yes I'd like to get it in u-boot-test-hooks. I last pushesd it
> > > to Labgrid 7 months ago [1]. I thought I did it at the end of November
> > > but must have imagined it. Perhaps I could put the config files, etc.
> > > into u-boot-test somewhere, like in a labgrid/sjg directory? Could you
> > > give me access to that repo?
> >
> > I've updated the permissions and you should update the "bin/ellesmere"
> > directory and add "py/ellesmere".
> 
> OK, thanks, will do. Where should the labgrid config go? It is three
> files and perhaps a few systemd scripts. See [2]

Well, we have a "udev" directory. So I guess a "labgrid-sjg" directory
and you can populate under that?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 26/28] test/py: Allow tests to be filtered by role

[Simon Glass]

Hi Tom,

On Tue, 18 Feb 2025 at 17:50, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 18, 2025 at 05:39:53PM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 18 Feb 2025 at 17:25, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Feb 18, 2025 at 05:04:09PM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Tue, 18 Feb 2025 at 08:12, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Tue, Feb 18, 2025 at 05:09:06AM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Mon, 17 Feb 2025 at 10:48, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Sun, Feb 16, 2025 at 01:44:11PM -0700, Simon Glass wrote:
> > > > > > >
> > > > > > > > Some test can only be run by a particular board in a lab, e.g. because
> > > > > > > > they are loaded with an OS image used by the test. Add a way to specify
> > > > > > > > this in tests.
> > > > > > > >
> > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Changes in v2:
> > > > > > > > - Add new patch to allow tests to be filtered by role
> > > > > > > >
> > > > > > > >  test/py/conftest.py | 22 ++++++++++++++++++++++
> > > > > > > >  test/py/pytest.ini  |  1 +
> > > > > > > >  2 files changed, 23 insertions(+)
> > > > > > >
> > > > > > > This seems duplicative of the existing functionality we have, all
> > > > > > > hardware tests can only be run on platforms which configure what the
> > > > > > > test requires, in the boardenv file.
> > > > > >
> > > > > > I was afraid you would say that, and yes you are right, it is duplicative.
> > > > > >
> > > > > > With my lab I am trying to use Labgrid's configuration and not need
> > > > > > anything else. For example, the params on how to launch QEMU and where
> > > > > > to find the OS file are there, rather than in u-boot-test-hooks. It
> > > > > > allows me to run a session interactively just as easily (and with the
> > > > > > same flow) as I run the pytests and gitlab.
> > > > > >
> > > > > > In my lab, I actually don't have any Python things, apart from what I
> > > > > > added for that tftp test.
> > > > > >
> > > > > > I wonder if I could create a way to pass the env__ stuff in from Labgrid?
> > > > >
> > > > > Well, I was also afraid that was going to be part of your answer. A
> > > > > challenge I see with the way you're using labgrid is that details are
> > > > > kept there, instead. Is there some way to publish your configs so
> > > > > they're also available for reference? I indeed have been lax in doing so
> > > > > for the way I use it and so posted:
> > > > > https://patchwork.ozlabs.org/project/uboot/patch/20250217200638.356583-1-trini@konsulko.com/
> > > > > yesterday.
> > > >
> > > > Yes, that's great. It is a big help to have all the lab stuff in one
> > > > place and not being able to get things into that repo was one reason I
> > > > gave up with my local lab that used the hook scripts. Just too many
> > > > local branches and hacks, and too hard.
> > > >
> > > > Anyway, yes I'd like to get it in u-boot-test-hooks. I last pushesd it
> > > > to Labgrid 7 months ago [1]. I thought I did it at the end of November
> > > > but must have imagined it. Perhaps I could put the config files, etc.
> > > > into u-boot-test somewhere, like in a labgrid/sjg directory? Could you
> > > > give me access to that repo?
> > >
> > > I've updated the permissions and you should update the "bin/ellesmere"
> > > directory and add "py/ellesmere".
> >
> > OK, thanks, will do. Where should the labgrid config go? It is three
> > files and perhaps a few systemd scripts. See [2]
>
> Well, we have a "udev" directory. So I guess a "labgrid-sjg" directory
> and you can populate under that?

OK. I need to do another round of cleaning up things, but will do it after that.

Regards,
SImon

[PATCH v2 27/28] RFC: test/py: Deal with timeouts

[Simon Glass]

The distro test takes a little longer to shut down and restart, so
add more time to this operation.

Disable the sleep command for now, as it seems to be unreliable on
QEMU with '-cpu host'.

More thought will be needed for both of these problems.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 test/py/console_base.py     | 8 ++++----
 test/py/tests/test_sleep.py | 1 +
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/test/py/console_base.py b/test/py/console_base.py
index 073ab0462dc..6d0f56f841d 100644
--- a/test/py/console_base.py
+++ b/test/py/console_base.py
@@ -464,7 +464,7 @@ class ConsoleBase(object):
         finally:
             self.p.timeout = orig_timeout
 
-    def ensure_spawned(self, expect_reset=False):
+    def ensure_spawned(self, expect_reset=False, timeout=None):
         """Ensure a connection to a correctly running U-Boot instance.
 
         This may require spawning a new Sandbox process or resetting target
@@ -485,7 +485,7 @@ class ConsoleBase(object):
             # Reset the console timeout value as some tests may change
             # its default value during the execution
             if not self.config.gdbserver:
-                self.p.timeout = TIMEOUT_MS
+                self.p.timeout = timeout or TIMEOUT_MS
             return
         try:
             self.log.start_section('Starting U-Boot')
@@ -496,7 +496,7 @@ class ConsoleBase(object):
             # future, possibly per-test to be optimal. This works for 'help'
             # on board 'seaboard'.
             if not self.config.gdbserver:
-                self.p.timeout = TIMEOUT_MS
+                self.p.timeout = timeout or TIMEOUT_MS
             self.p.logfile_read = self.logstream
             if self.config.use_running_system:
                 # Send an empty command to set up the 'expect' logic. This has
@@ -544,7 +544,7 @@ class ConsoleBase(object):
     def restart_uboot(self, expect_reset=False):
         """Shut down and restart U-Boot."""
         self.cleanup_spawn()
-        self.ensure_spawned(expect_reset)
+        self.ensure_spawned(expect_reset, 60 * 1000)
 
     def get_spawn_output(self):
         """Return the start-up output from U-Boot
diff --git a/test/py/tests/test_sleep.py b/test/py/tests/test_sleep.py
index f1bf34e05b2..2285a815813 100644
--- a/test/py/tests/test_sleep.py
+++ b/test/py/tests/test_sleep.py
@@ -19,6 +19,7 @@ env__sleep_margin = 0.25
 
 """
 
+@pytest.mark.role('!qemu-x86_64')
 def test_sleep(ubman):
     """Test the sleep command, and validate that it sleeps for approximately
     the correct amount of time."""
-- 
2.43.0

Re: [PATCH v2 27/28] RFC: test/py: Deal with timeouts

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

On Sun, Feb 16, 2025 at 01:44:12PM -0700, Simon Glass wrote:

> The distro test takes a little longer to shut down and restart, so
> add more time to this operation.
> 
> Disable the sleep command for now, as it seems to be unreliable on
> QEMU with '-cpu host'.
> 
> More thought will be needed for both of these problems.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

We already do:
    TEST_PY_TEST_SPEC: "not sleep"
for every QEMU target in CI. The test fuzz-factor is generally enough
for sandbox, but maybe we should just make that test opt-in.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 27/28] RFC: test/py: Deal with timeouts

[Simon Glass]

Hi Tom,

On Mon, 17 Feb 2025 at 17:46, Tom Rini <trini@konsulko.com> wrote:
>
> On Sun, Feb 16, 2025 at 01:44:12PM -0700, Simon Glass wrote:
>
> > The distro test takes a little longer to shut down and restart, so
> > add more time to this operation.
> >
> > Disable the sleep command for now, as it seems to be unreliable on
> > QEMU with '-cpu host'.
> >
> > More thought will be needed for both of these problems.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
>
> We already do:
>     TEST_PY_TEST_SPEC: "not sleep"
> for every QEMU target in CI. The test fuzz-factor is generally enough
> for sandbox, but maybe we should just make that test opt-in.

It is useful to have it on real boards. For QEMU there is actually a
bug on x86 where the timer is wrong in some cases. Still, we are
making progress.

Regards,
Simon

[PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Now that U-Boot can boot this quickly, using kvm, add a test that the
installer starts up correctly.

Use the qemu-x86_64 board in the SJG lab.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add more patches to support booting with kvm
- Add new patch with a test for booting Ubuntu 24.04

 .gitlab-ci.yml               |  5 ++++
 test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 test/py/tests/test_distro.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8c49d5b0a79..ec799e97c10 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -745,3 +745,8 @@ zybo:
   variables:
     ROLE: zybo
   <<: *lab_dfn
+
+qemu-x86_64:
+  variables:
+    ROLE: qemu-x86_64
+  <<: *lab_dfn
diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
new file mode 100644
index 00000000000..51eec45cecc
--- /dev/null
+++ b/test/py/tests/test_distro.py
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright 2025 Canonical Ltd.
+# Written by Simon Glass <simon.glass@canonical.com>
+
+import pytest
+
+DOWN = '\x1b\x5b\x42\x0d'
+
+# Enable early console so that the test can see if something goes wrong
+CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
+
+@pytest.mark.boardspec('qemu-x86_64')
+@pytest.mark.role('qemu-x86_64')
+def test_distro(ubman):
+    """Test that of-platdata can be generated and used in sandbox"""
+    with ubman.log.section('boot'):
+        ubman.run_command('boot', wait_for_prompt=False)
+
+    with ubman.log.section('Grub'):
+        # Wait for grub to come up and offset a menu
+        ubman.p.expect(['Try or Install Ubuntu'])
+
+        # Press 'e' to edit the command line
+        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
+
+        # Wait until we see the editor appear
+        ubman.p.expect(['/casper/initrd'])
+
+        # Go down to the 'linux' line
+        ubman.send(DOWN * 3)
+
+        # Go to end of line
+        ubman.ctrl('E')
+
+        # Backspace to remove 'quiet splash'
+        ubman.send('\b' * len('quiet splash'))
+
+        # Send our noisy console
+        ubman.send(CONSOLE)
+
+        # Tell grub to boot
+        ubman.ctrl('X')
+        ubman.p.expect(['Booting a command list'])
+
+    with ubman.log.section('Linux'):
+        # Linux should start immediately
+        ubman.p.expect(['Linux version'])
+
+    with ubman.log.section('Ubuntu'):
+        # Shortly later, we should see this banner
+        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
+
+    ubman.restart_uboot()
-- 
2.43.0

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 3528 bytes --]

On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> Now that U-Boot can boot this quickly, using kvm, add a test that the
> installer starts up correctly.
> 
> Use the qemu-x86_64 board in the SJG lab.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
> Changes in v2:
> - Add more patches to support booting with kvm
> - Add new patch with a test for booting Ubuntu 24.04
> 
>  .gitlab-ci.yml               |  5 ++++
>  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 58 insertions(+)
>  create mode 100644 test/py/tests/test_distro.py
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 8c49d5b0a79..ec799e97c10 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -745,3 +745,8 @@ zybo:
>    variables:
>      ROLE: zybo
>    <<: *lab_dfn
> +
> +qemu-x86_64:
> +  variables:
> +    ROLE: qemu-x86_64
> +  <<: *lab_dfn

I'm not sure why this is in your lab stanza, rather than the normal
test.py QEMU stanza.

> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> new file mode 100644
> index 00000000000..51eec45cecc
> --- /dev/null
> +++ b/test/py/tests/test_distro.py
> @@ -0,0 +1,53 @@
> +# SPDX-License-Identifier: GPL-2.0+
> +# Copyright 2025 Canonical Ltd.
> +# Written by Simon Glass <simon.glass@canonical.com>
> +
> +import pytest
> +
> +DOWN = '\x1b\x5b\x42\x0d'
> +
> +# Enable early console so that the test can see if something goes wrong
> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> +
> +@pytest.mark.boardspec('qemu-x86_64')
> +@pytest.mark.role('qemu-x86_64')
> +def test_distro(ubman):
> +    """Test that of-platdata can be generated and used in sandbox"""
> +    with ubman.log.section('boot'):
> +        ubman.run_command('boot', wait_for_prompt=False)
> +
> +    with ubman.log.section('Grub'):
> +        # Wait for grub to come up and offset a menu
> +        ubman.p.expect(['Try or Install Ubuntu'])
> +
> +        # Press 'e' to edit the command line
> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> +
> +        # Wait until we see the editor appear
> +        ubman.p.expect(['/casper/initrd'])
> +
> +        # Go down to the 'linux' line
> +        ubman.send(DOWN * 3)
> +
> +        # Go to end of line
> +        ubman.ctrl('E')
> +
> +        # Backspace to remove 'quiet splash'
> +        ubman.send('\b' * len('quiet splash'))
> +
> +        # Send our noisy console
> +        ubman.send(CONSOLE)
> +
> +        # Tell grub to boot
> +        ubman.ctrl('X')
> +        ubman.p.expect(['Booting a command list'])
> +
> +    with ubman.log.section('Linux'):
> +        # Linux should start immediately
> +        ubman.p.expect(['Linux version'])
> +
> +    with ubman.log.section('Ubuntu'):
> +        # Shortly later, we should see this banner
> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> +
> +    ubman.restart_uboot()

And this seems very inflexible. Please see
test/py/tests/test_net_boot.py for an example of how to have this be
configurable and work on arbitrary platforms. What I assume is tricky is
that the "role" part here is where you have a special disk image being
passed. That too could be dealt with in u-boot-test-hooks in a few ways,
and the images pre-fetched to the CI container. And if this was
configurable similar to the example I noted above, it could check real
hardware too.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
>
> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > installer starts up correctly.
> >
> > Use the qemu-x86_64 board in the SJG lab.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > Changes in v2:
> > - Add more patches to support booting with kvm
> > - Add new patch with a test for booting Ubuntu 24.04
> >
> >  .gitlab-ci.yml               |  5 ++++
> >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> >  2 files changed, 58 insertions(+)
> >  create mode 100644 test/py/tests/test_distro.py
> >
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 8c49d5b0a79..ec799e97c10 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -745,3 +745,8 @@ zybo:
> >    variables:
> >      ROLE: zybo
> >    <<: *lab_dfn
> > +
> > +qemu-x86_64:
> > +  variables:
> > +    ROLE: qemu-x86_64
> > +  <<: *lab_dfn
>
> I'm not sure why this is in your lab stanza, rather than the normal
> test.py QEMU stanza.

Are you wanting to add the Ubuntu image into CI? It is quite large.

>
> > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > new file mode 100644
> > index 00000000000..51eec45cecc
> > --- /dev/null
> > +++ b/test/py/tests/test_distro.py
> > @@ -0,0 +1,53 @@
> > +# SPDX-License-Identifier: GPL-2.0+
> > +# Copyright 2025 Canonical Ltd.
> > +# Written by Simon Glass <simon.glass@canonical.com>
> > +
> > +import pytest
> > +
> > +DOWN = '\x1b\x5b\x42\x0d'
> > +
> > +# Enable early console so that the test can see if something goes wrong
> > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > +
> > +@pytest.mark.boardspec('qemu-x86_64')
> > +@pytest.mark.role('qemu-x86_64')
> > +def test_distro(ubman):
> > +    """Test that of-platdata can be generated and used in sandbox"""
> > +    with ubman.log.section('boot'):
> > +        ubman.run_command('boot', wait_for_prompt=False)
> > +
> > +    with ubman.log.section('Grub'):
> > +        # Wait for grub to come up and offset a menu
> > +        ubman.p.expect(['Try or Install Ubuntu'])
> > +
> > +        # Press 'e' to edit the command line
> > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > +
> > +        # Wait until we see the editor appear
> > +        ubman.p.expect(['/casper/initrd'])
> > +
> > +        # Go down to the 'linux' line
> > +        ubman.send(DOWN * 3)
> > +
> > +        # Go to end of line
> > +        ubman.ctrl('E')
> > +
> > +        # Backspace to remove 'quiet splash'
> > +        ubman.send('\b' * len('quiet splash'))
> > +
> > +        # Send our noisy console
> > +        ubman.send(CONSOLE)
> > +
> > +        # Tell grub to boot
> > +        ubman.ctrl('X')
> > +        ubman.p.expect(['Booting a command list'])
> > +
> > +    with ubman.log.section('Linux'):
> > +        # Linux should start immediately
> > +        ubman.p.expect(['Linux version'])
> > +
> > +    with ubman.log.section('Ubuntu'):
> > +        # Shortly later, we should see this banner
> > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > +
> > +    ubman.restart_uboot()
>
> And this seems very inflexible. Please see
> test/py/tests/test_net_boot.py for an example of how to have this be
> configurable and work on arbitrary platforms. What I assume is tricky is
> that the "role" part here is where you have a special disk image being
> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> and the images pre-fetched to the CI container. And if this was
> configurable similar to the example I noted above, it could check real
> hardware too.

That wasn't the reaction I expected.

Yes, it is inflexible, but it is a starting point. Isn't it better
than what we have today?

I would like to generalise this test to work on at least one real
board, preferably one that doesn't use grub.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 5181 bytes --]

On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > installer starts up correctly.
> > >
> > > Use the qemu-x86_64 board in the SJG lab.
> > >
> > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > ---
> > >
> > > Changes in v2:
> > > - Add more patches to support booting with kvm
> > > - Add new patch with a test for booting Ubuntu 24.04
> > >
> > >  .gitlab-ci.yml               |  5 ++++
> > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 58 insertions(+)
> > >  create mode 100644 test/py/tests/test_distro.py
> > >
> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > index 8c49d5b0a79..ec799e97c10 100644
> > > --- a/.gitlab-ci.yml
> > > +++ b/.gitlab-ci.yml
> > > @@ -745,3 +745,8 @@ zybo:
> > >    variables:
> > >      ROLE: zybo
> > >    <<: *lab_dfn
> > > +
> > > +qemu-x86_64:
> > > +  variables:
> > > +    ROLE: qemu-x86_64
> > > +  <<: *lab_dfn
> >
> > I'm not sure why this is in your lab stanza, rather than the normal
> > test.py QEMU stanza.
> 
> Are you wanting to add the Ubuntu image into CI? It is quite large.

If we're going to be able to run it on N platforms, yes, we need to
think of a good way to cache the download. There's not a particular
reason we can't run the stock Ubuntu RISC-V image on the two sifive
targets and also qemu-riscv64, is there?

> > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > new file mode 100644
> > > index 00000000000..51eec45cecc
> > > --- /dev/null
> > > +++ b/test/py/tests/test_distro.py
> > > @@ -0,0 +1,53 @@
> > > +# SPDX-License-Identifier: GPL-2.0+
> > > +# Copyright 2025 Canonical Ltd.
> > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > +
> > > +import pytest
> > > +
> > > +DOWN = '\x1b\x5b\x42\x0d'
> > > +
> > > +# Enable early console so that the test can see if something goes wrong
> > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > +
> > > +@pytest.mark.boardspec('qemu-x86_64')
> > > +@pytest.mark.role('qemu-x86_64')
> > > +def test_distro(ubman):
> > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > +    with ubman.log.section('boot'):
> > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > +
> > > +    with ubman.log.section('Grub'):
> > > +        # Wait for grub to come up and offset a menu
> > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > +
> > > +        # Press 'e' to edit the command line
> > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > +
> > > +        # Wait until we see the editor appear
> > > +        ubman.p.expect(['/casper/initrd'])
> > > +
> > > +        # Go down to the 'linux' line
> > > +        ubman.send(DOWN * 3)
> > > +
> > > +        # Go to end of line
> > > +        ubman.ctrl('E')
> > > +
> > > +        # Backspace to remove 'quiet splash'
> > > +        ubman.send('\b' * len('quiet splash'))
> > > +
> > > +        # Send our noisy console
> > > +        ubman.send(CONSOLE)
> > > +
> > > +        # Tell grub to boot
> > > +        ubman.ctrl('X')
> > > +        ubman.p.expect(['Booting a command list'])
> > > +
> > > +    with ubman.log.section('Linux'):
> > > +        # Linux should start immediately
> > > +        ubman.p.expect(['Linux version'])
> > > +
> > > +    with ubman.log.section('Ubuntu'):
> > > +        # Shortly later, we should see this banner
> > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > +
> > > +    ubman.restart_uboot()
> >
> > And this seems very inflexible. Please see
> > test/py/tests/test_net_boot.py for an example of how to have this be
> > configurable and work on arbitrary platforms. What I assume is tricky is
> > that the "role" part here is where you have a special disk image being
> > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > and the images pre-fetched to the CI container. And if this was
> > configurable similar to the example I noted above, it could check real
> > hardware too.
> 
> That wasn't the reaction I expected.
> 
> Yes, it is inflexible, but it is a starting point. Isn't it better
> than what we have today?

Is your inflexible boot an OS test better than the flexible boot an OS
test that we have today? No, it's not.

> I would like to generalise this test to work on at least one real
> board, preferably one that doesn't use grub.

OK. The test we have today does that, if you check for the "Welcome to
..." string instead of the kernel has booted string. It also does
netboot rather than run default bootcmd. But that's an easy enough test
to write up. The only thing stopping me from doing that right now is I
need to find a board in the lab where we installed an OS to eMMC and not
SD card (some lab sd-mux issues).

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > installer starts up correctly.
> > > >
> > > > Use the qemu-x86_64 board in the SJG lab.
> > > >
> > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > ---
> > > >
> > > > Changes in v2:
> > > > - Add more patches to support booting with kvm
> > > > - Add new patch with a test for booting Ubuntu 24.04
> > > >
> > > >  .gitlab-ci.yml               |  5 ++++
> > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 58 insertions(+)
> > > >  create mode 100644 test/py/tests/test_distro.py
> > > >
> > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > --- a/.gitlab-ci.yml
> > > > +++ b/.gitlab-ci.yml
> > > > @@ -745,3 +745,8 @@ zybo:
> > > >    variables:
> > > >      ROLE: zybo
> > > >    <<: *lab_dfn
> > > > +
> > > > +qemu-x86_64:
> > > > +  variables:
> > > > +    ROLE: qemu-x86_64
> > > > +  <<: *lab_dfn
> > >
> > > I'm not sure why this is in your lab stanza, rather than the normal
> > > test.py QEMU stanza.
> >
> > Are you wanting to add the Ubuntu image into CI? It is quite large.
>
> If we're going to be able to run it on N platforms, yes, we need to
> think of a good way to cache the download. There's not a particular
> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> targets and also qemu-riscv64, is there?

Yes, we can do that. It is pretty simple to set up in Labgrid and it
doesn't require all the runners to download a much larger image, etc.

>
> > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > new file mode 100644
> > > > index 00000000000..51eec45cecc
> > > > --- /dev/null
> > > > +++ b/test/py/tests/test_distro.py
> > > > @@ -0,0 +1,53 @@
> > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > +# Copyright 2025 Canonical Ltd.
> > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > +
> > > > +import pytest
> > > > +
> > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > +
> > > > +# Enable early console so that the test can see if something goes wrong
> > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > +
> > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > +@pytest.mark.role('qemu-x86_64')
> > > > +def test_distro(ubman):
> > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > +    with ubman.log.section('boot'):
> > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > +
> > > > +    with ubman.log.section('Grub'):
> > > > +        # Wait for grub to come up and offset a menu
> > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > +
> > > > +        # Press 'e' to edit the command line
> > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > +
> > > > +        # Wait until we see the editor appear
> > > > +        ubman.p.expect(['/casper/initrd'])
> > > > +
> > > > +        # Go down to the 'linux' line
> > > > +        ubman.send(DOWN * 3)
> > > > +
> > > > +        # Go to end of line
> > > > +        ubman.ctrl('E')
> > > > +
> > > > +        # Backspace to remove 'quiet splash'
> > > > +        ubman.send('\b' * len('quiet splash'))
> > > > +
> > > > +        # Send our noisy console
> > > > +        ubman.send(CONSOLE)
> > > > +
> > > > +        # Tell grub to boot
> > > > +        ubman.ctrl('X')
> > > > +        ubman.p.expect(['Booting a command list'])
> > > > +
> > > > +    with ubman.log.section('Linux'):
> > > > +        # Linux should start immediately
> > > > +        ubman.p.expect(['Linux version'])
> > > > +
> > > > +    with ubman.log.section('Ubuntu'):
> > > > +        # Shortly later, we should see this banner
> > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > +
> > > > +    ubman.restart_uboot()
> > >
> > > And this seems very inflexible. Please see
> > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > that the "role" part here is where you have a special disk image being
> > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > and the images pre-fetched to the CI container. And if this was
> > > configurable similar to the example I noted above, it could check real
> > > hardware too.
> >
> > That wasn't the reaction I expected.
> >
> > Yes, it is inflexible, but it is a starting point. Isn't it better
> > than what we have today?
>
> Is your inflexible boot an OS test better than the flexible boot an OS
> test that we have today? No, it's not.

I didn't even know about it, or perhaps I forgot.

Perhaps this relates to getting the labgrid config published and
figuring out how to pass info from Labgrid to tests.

>
> > I would like to generalise this test to work on at least one real
> > board, preferably one that doesn't use grub.
>
> OK. The test we have today does that, if you check for the "Welcome to
> ..." string instead of the kernel has booted string. It also does
> netboot rather than run default bootcmd. But that's an easy enough test
> to write up. The only thing stopping me from doing that right now is I
> need to find a board in the lab where we installed an OS to eMMC and not
> SD card (some lab sd-mux issues).

OK. Labgrid has a 'features' thing which you can attach to targets, so
I should be able to use that to indicate that Ubuntu, Debian, Armbian,
etc. are available.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 7123 bytes --]

On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > installer starts up correctly.
> > > > >
> > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > >
> > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > ---
> > > > >
> > > > > Changes in v2:
> > > > > - Add more patches to support booting with kvm
> > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > >
> > > > >  .gitlab-ci.yml               |  5 ++++
> > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > >  2 files changed, 58 insertions(+)
> > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > >
> > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > --- a/.gitlab-ci.yml
> > > > > +++ b/.gitlab-ci.yml
> > > > > @@ -745,3 +745,8 @@ zybo:
> > > > >    variables:
> > > > >      ROLE: zybo
> > > > >    <<: *lab_dfn
> > > > > +
> > > > > +qemu-x86_64:
> > > > > +  variables:
> > > > > +    ROLE: qemu-x86_64
> > > > > +  <<: *lab_dfn
> > > >
> > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > test.py QEMU stanza.
> > >
> > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> >
> > If we're going to be able to run it on N platforms, yes, we need to
> > think of a good way to cache the download. There's not a particular
> > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > targets and also qemu-riscv64, is there?
> 
> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> doesn't require all the runners to download a much larger image, etc.

I don't quite understand why it's under "labgrid". These are generic CI
tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
that certain longer or possibly destructive tests are only run on tagged
releases or as requested rather than every time, as it will take longer.
But pretty much every platform under the qemu target list should be able
to Just Boot an off the shelf OS distribution is my point.

> > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > new file mode 100644
> > > > > index 00000000000..51eec45cecc
> > > > > --- /dev/null
> > > > > +++ b/test/py/tests/test_distro.py
> > > > > @@ -0,0 +1,53 @@
> > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > +# Copyright 2025 Canonical Ltd.
> > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > +
> > > > > +import pytest
> > > > > +
> > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > +
> > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > +
> > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > +def test_distro(ubman):
> > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > +    with ubman.log.section('boot'):
> > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > +
> > > > > +    with ubman.log.section('Grub'):
> > > > > +        # Wait for grub to come up and offset a menu
> > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > +
> > > > > +        # Press 'e' to edit the command line
> > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > +
> > > > > +        # Wait until we see the editor appear
> > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > +
> > > > > +        # Go down to the 'linux' line
> > > > > +        ubman.send(DOWN * 3)
> > > > > +
> > > > > +        # Go to end of line
> > > > > +        ubman.ctrl('E')
> > > > > +
> > > > > +        # Backspace to remove 'quiet splash'
> > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > +
> > > > > +        # Send our noisy console
> > > > > +        ubman.send(CONSOLE)
> > > > > +
> > > > > +        # Tell grub to boot
> > > > > +        ubman.ctrl('X')
> > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > +
> > > > > +    with ubman.log.section('Linux'):
> > > > > +        # Linux should start immediately
> > > > > +        ubman.p.expect(['Linux version'])
> > > > > +
> > > > > +    with ubman.log.section('Ubuntu'):
> > > > > +        # Shortly later, we should see this banner
> > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > +
> > > > > +    ubman.restart_uboot()
> > > >
> > > > And this seems very inflexible. Please see
> > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > that the "role" part here is where you have a special disk image being
> > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > and the images pre-fetched to the CI container. And if this was
> > > > configurable similar to the example I noted above, it could check real
> > > > hardware too.
> > >
> > > That wasn't the reaction I expected.
> > >
> > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > than what we have today?
> >
> > Is your inflexible boot an OS test better than the flexible boot an OS
> > test that we have today? No, it's not.
> 
> I didn't even know about it, or perhaps I forgot.

I believe I mentioned it every time you've said we should have an OS
test, so yes, I guess you forgot.

> Perhaps this relates to getting the labgrid config published and
> figuring out how to pass info from Labgrid to tests.
> 
> >
> > > I would like to generalise this test to work on at least one real
> > > board, preferably one that doesn't use grub.
> >
> > OK. The test we have today does that, if you check for the "Welcome to
> > ..." string instead of the kernel has booted string. It also does
> > netboot rather than run default bootcmd. But that's an easy enough test
> > to write up. The only thing stopping me from doing that right now is I
> > need to find a board in the lab where we installed an OS to eMMC and not
> > SD card (some lab sd-mux issues).
> 
> OK. Labgrid has a 'features' thing which you can attach to targets, so
> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> etc. are available.

OK, but that sounds like the opposite direction. These are generic tests
that can run in any / all of the labs, not just your labgrid
configuration. AMD has been contributing tests that run on hardware for
example.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > installer starts up correctly.
> > > > > >
> > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > >
> > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > ---
> > > > > >
> > > > > > Changes in v2:
> > > > > > - Add more patches to support booting with kvm
> > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > >
> > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > >  2 files changed, 58 insertions(+)
> > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > >
> > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > --- a/.gitlab-ci.yml
> > > > > > +++ b/.gitlab-ci.yml
> > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > >    variables:
> > > > > >      ROLE: zybo
> > > > > >    <<: *lab_dfn
> > > > > > +
> > > > > > +qemu-x86_64:
> > > > > > +  variables:
> > > > > > +    ROLE: qemu-x86_64
> > > > > > +  <<: *lab_dfn
> > > > >
> > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > test.py QEMU stanza.
> > > >
> > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > >
> > > If we're going to be able to run it on N platforms, yes, we need to
> > > think of a good way to cache the download. There's not a particular
> > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > targets and also qemu-riscv64, is there?
> >
> > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > doesn't require all the runners to download a much larger image, etc.
>
> I don't quite understand why it's under "labgrid". These are generic CI
> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> that certain longer or possibly destructive tests are only run on tagged
> releases or as requested rather than every time, as it will take longer.
> But pretty much every platform under the qemu target list should be able
> to Just Boot an off the shelf OS distribution is my point.

Sure, and I'm not suggesting we shouldn't do that as well.

>
> > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > new file mode 100644
> > > > > > index 00000000000..51eec45cecc
> > > > > > --- /dev/null
> > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > @@ -0,0 +1,53 @@
> > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > +
> > > > > > +import pytest
> > > > > > +
> > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > +
> > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > +
> > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > +def test_distro(ubman):
> > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > +    with ubman.log.section('boot'):
> > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > +
> > > > > > +    with ubman.log.section('Grub'):
> > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > +
> > > > > > +        # Press 'e' to edit the command line
> > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > +
> > > > > > +        # Wait until we see the editor appear
> > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > +
> > > > > > +        # Go down to the 'linux' line
> > > > > > +        ubman.send(DOWN * 3)
> > > > > > +
> > > > > > +        # Go to end of line
> > > > > > +        ubman.ctrl('E')
> > > > > > +
> > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > +
> > > > > > +        # Send our noisy console
> > > > > > +        ubman.send(CONSOLE)
> > > > > > +
> > > > > > +        # Tell grub to boot
> > > > > > +        ubman.ctrl('X')
> > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > +
> > > > > > +    with ubman.log.section('Linux'):
> > > > > > +        # Linux should start immediately
> > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > +
> > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > +        # Shortly later, we should see this banner
> > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > +
> > > > > > +    ubman.restart_uboot()
> > > > >
> > > > > And this seems very inflexible. Please see
> > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > that the "role" part here is where you have a special disk image being
> > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > and the images pre-fetched to the CI container. And if this was
> > > > > configurable similar to the example I noted above, it could check real
> > > > > hardware too.
> > > >
> > > > That wasn't the reaction I expected.
> > > >
> > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > than what we have today?
> > >
> > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > test that we have today? No, it's not.
> >
> > I didn't even know about it, or perhaps I forgot.
>
> I believe I mentioned it every time you've said we should have an OS
> test, so yes, I guess you forgot.

Well it was only added in May last year and it relies on board config
which I don't have...although I see that you have now posted yours.

>
> > Perhaps this relates to getting the labgrid config published and
> > figuring out how to pass info from Labgrid to tests.
> >
> > >
> > > > I would like to generalise this test to work on at least one real
> > > > board, preferably one that doesn't use grub.
> > >
> > > OK. The test we have today does that, if you check for the "Welcome to
> > > ..." string instead of the kernel has booted string. It also does
> > > netboot rather than run default bootcmd. But that's an easy enough test
> > > to write up. The only thing stopping me from doing that right now is I
> > > need to find a board in the lab where we installed an OS to eMMC and not
> > > SD card (some lab sd-mux issues).
> >
> > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > etc. are available.
>
> OK, but that sounds like the opposite direction. These are generic tests
> that can run in any / all of the labs, not just your labgrid
> configuration. AMD has been contributing tests that run on hardware for
> example.

That's great, the more tests we have the better. But those tests can't
and don't run in CI, whereas mine can and do.

We need an 'all of the above' strategy here.

BTW, having thought about how test/py works a bit, instead of the
env__net_tftp_bootable_file stuff, we should have code or data which
sets up the required test files (on a suitable server) before running
the test. That way, all the test code is in one Python file and we
don't have to spend ages trying to divine what each test needs.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 10082 bytes --]

On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > installer starts up correctly.
> > > > > > >
> > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > >
> > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > ---
> > > > > > >
> > > > > > > Changes in v2:
> > > > > > > - Add more patches to support booting with kvm
> > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > >
> > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > >  2 files changed, 58 insertions(+)
> > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > >
> > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > --- a/.gitlab-ci.yml
> > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > >    variables:
> > > > > > >      ROLE: zybo
> > > > > > >    <<: *lab_dfn
> > > > > > > +
> > > > > > > +qemu-x86_64:
> > > > > > > +  variables:
> > > > > > > +    ROLE: qemu-x86_64
> > > > > > > +  <<: *lab_dfn
> > > > > >
> > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > test.py QEMU stanza.
> > > > >
> > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > >
> > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > think of a good way to cache the download. There's not a particular
> > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > targets and also qemu-riscv64, is there?
> > >
> > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > doesn't require all the runners to download a much larger image, etc.
> >
> > I don't quite understand why it's under "labgrid". These are generic CI
> > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > that certain longer or possibly destructive tests are only run on tagged
> > releases or as requested rather than every time, as it will take longer.
> > But pretty much every platform under the qemu target list should be able
> > to Just Boot an off the shelf OS distribution is my point.
> 
> Sure, and I'm not suggesting we shouldn't do that as well.
> 
> >
> > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > new file mode 100644
> > > > > > > index 00000000000..51eec45cecc
> > > > > > > --- /dev/null
> > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > @@ -0,0 +1,53 @@
> > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > +
> > > > > > > +import pytest
> > > > > > > +
> > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > +
> > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > +
> > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > +def test_distro(ubman):
> > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > +    with ubman.log.section('boot'):
> > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > +
> > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > +
> > > > > > > +        # Press 'e' to edit the command line
> > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > +
> > > > > > > +        # Wait until we see the editor appear
> > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > +
> > > > > > > +        # Go down to the 'linux' line
> > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > +
> > > > > > > +        # Go to end of line
> > > > > > > +        ubman.ctrl('E')
> > > > > > > +
> > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > +
> > > > > > > +        # Send our noisy console
> > > > > > > +        ubman.send(CONSOLE)
> > > > > > > +
> > > > > > > +        # Tell grub to boot
> > > > > > > +        ubman.ctrl('X')
> > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > +
> > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > +        # Linux should start immediately
> > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > +
> > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > +        # Shortly later, we should see this banner
> > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > +
> > > > > > > +    ubman.restart_uboot()
> > > > > >
> > > > > > And this seems very inflexible. Please see
> > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > that the "role" part here is where you have a special disk image being
> > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > configurable similar to the example I noted above, it could check real
> > > > > > hardware too.
> > > > >
> > > > > That wasn't the reaction I expected.
> > > > >
> > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > than what we have today?
> > > >
> > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > test that we have today? No, it's not.
> > >
> > > I didn't even know about it, or perhaps I forgot.
> >
> > I believe I mentioned it every time you've said we should have an OS
> > test, so yes, I guess you forgot.
> 
> Well it was only added in May last year and it relies on board config
> which I don't have...although I see that you have now posted yours.

Yes, it was added not quite a year ago, and is documented within the
test, like most tests that rely on the real platform.

And do we need better documentation for test? Yes.

> > > Perhaps this relates to getting the labgrid config published and
> > > figuring out how to pass info from Labgrid to tests.
> > >
> > > >
> > > > > I would like to generalise this test to work on at least one real
> > > > > board, preferably one that doesn't use grub.
> > > >
> > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > ..." string instead of the kernel has booted string. It also does
> > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > to write up. The only thing stopping me from doing that right now is I
> > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > SD card (some lab sd-mux issues).
> > >
> > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > etc. are available.
> >
> > OK, but that sounds like the opposite direction. These are generic tests
> > that can run in any / all of the labs, not just your labgrid
> > configuration. AMD has been contributing tests that run on hardware for
> > example.
> 
> That's great, the more tests we have the better. But those tests can't
> and don't run in CI, whereas mine can and do.

AFAICT they're running on AMD's CI. They run on my CI. They don't run on
*your* lab because you took things, intentionally, in a direction to
minimize using u-boot-test-hooks and our existing per-board
configuration infrastructure.

> We need an 'all of the above' strategy here.

Sure. But I still want to see things as reusable as possible. What you
have above is *extremely* board and OS specific and non-configurable. I
also don't quite see why it's not a test of autoboot with the
pre-requisite of an OS being installed.

> BTW, having thought about how test/py works a bit, instead of the
> env__net_tftp_bootable_file stuff, we should have code or data which
> sets up the required test files (on a suitable server) before running
> the test. That way, all the test code is in one Python file and we
> don't have to spend ages trying to divine what each test needs.

That seems like a lot more work than documenting more what we have
today, and I'm not sure of the benefit. Given the contents of the pxe
test, yes, just having those files available to 'cp' in place would be
helpful. But that's not the case for booting a kernel (the FIT match
stuff doesn't work on the TI platforms atm). And if you look at the
config I posted it also includes bootstage configuration. It also won't
work well for the SPI tests, which I'm talking with Love about in
another thread.

In other words, the majority of py/<host>/u_boot_boardenv_ content is
configuration details, specific to both the platform / SoC first, some
lab specific details second and drop-in existing 3rd party files a
distant third.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > installer starts up correctly.
> > > > > > > >
> > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > >
> > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Changes in v2:
> > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > >
> > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > >
> > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > >    variables:
> > > > > > > >      ROLE: zybo
> > > > > > > >    <<: *lab_dfn
> > > > > > > > +
> > > > > > > > +qemu-x86_64:
> > > > > > > > +  variables:
> > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > +  <<: *lab_dfn
> > > > > > >
> > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > test.py QEMU stanza.
> > > > > >
> > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > >
> > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > think of a good way to cache the download. There's not a particular
> > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > targets and also qemu-riscv64, is there?
> > > >
> > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > doesn't require all the runners to download a much larger image, etc.
> > >
> > > I don't quite understand why it's under "labgrid". These are generic CI
> > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > that certain longer or possibly destructive tests are only run on tagged
> > > releases or as requested rather than every time, as it will take longer.
> > > But pretty much every platform under the qemu target list should be able
> > > to Just Boot an off the shelf OS distribution is my point.
> >
> > Sure, and I'm not suggesting we shouldn't do that as well.
> >
> > >
> > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > new file mode 100644
> > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > +
> > > > > > > > +import pytest
> > > > > > > > +
> > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > +
> > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > +
> > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > +def test_distro(ubman):
> > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > +
> > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > +
> > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > +
> > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > +
> > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > +
> > > > > > > > +        # Go to end of line
> > > > > > > > +        ubman.ctrl('E')
> > > > > > > > +
> > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > +
> > > > > > > > +        # Send our noisy console
> > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > +
> > > > > > > > +        # Tell grub to boot
> > > > > > > > +        ubman.ctrl('X')
> > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > +
> > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > +        # Linux should start immediately
> > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > +
> > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > +
> > > > > > > > +    ubman.restart_uboot()
> > > > > > >
> > > > > > > And this seems very inflexible. Please see
> > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > hardware too.
> > > > > >
> > > > > > That wasn't the reaction I expected.
> > > > > >
> > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > than what we have today?
> > > > >
> > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > test that we have today? No, it's not.
> > > >
> > > > I didn't even know about it, or perhaps I forgot.
> > >
> > > I believe I mentioned it every time you've said we should have an OS
> > > test, so yes, I guess you forgot.
> >
> > Well it was only added in May last year and it relies on board config
> > which I don't have...although I see that you have now posted yours.
>
> Yes, it was added not quite a year ago, and is documented within the
> test, like most tests that rely on the real platform.
>
> And do we need better documentation for test? Yes.

+1

I'll note that I did my bit!

>
> > > > Perhaps this relates to getting the labgrid config published and
> > > > figuring out how to pass info from Labgrid to tests.
> > > >
> > > > >
> > > > > > I would like to generalise this test to work on at least one real
> > > > > > board, preferably one that doesn't use grub.
> > > > >
> > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > ..." string instead of the kernel has booted string. It also does
> > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > SD card (some lab sd-mux issues).
> > > >
> > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > etc. are available.
> > >
> > > OK, but that sounds like the opposite direction. These are generic tests
> > > that can run in any / all of the labs, not just your labgrid
> > > configuration. AMD has been contributing tests that run on hardware for
> > > example.
> >
> > That's great, the more tests we have the better. But those tests can't
> > and don't run in CI, whereas mine can and do.
>
> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> *your* lab because you took things, intentionally, in a direction to
> minimize using u-boot-test-hooks and our existing per-board
> configuration infrastructure.

When I look at CI all I see is my lab. Which CI are you referring to
and how can I access it?

Here I would like to make a case for moving to using Labgrid across
the board, but unfortunately the project struggles to review PRs, so
it's probably not a good idea.

>
> > We need an 'all of the above' strategy here.
>
> Sure. But I still want to see things as reusable as possible. What you
> have above is *extremely* board and OS specific and non-configurable.

Yes, agreed.

> I
> also don't quite see why it's not a test of autoboot with the
> pre-requisite of an OS being installed.

Ah OK, my test is just for the installer itself. Both are useful, but
I hope eventually to have the installer run to completion and then
reboot to check all is well.

>
> > BTW, having thought about how test/py works a bit, instead of the
> > env__net_tftp_bootable_file stuff, we should have code or data which
> > sets up the required test files (on a suitable server) before running
> > the test. That way, all the test code is in one Python file and we
> > don't have to spend ages trying to divine what each test needs.
>
> That seems like a lot more work than documenting more what we have
> today, and I'm not sure of the benefit. Given the contents of the pxe
> test, yes, just having those files available to 'cp' in place would be
> helpful. But that's not the case for booting a kernel (the FIT match
> stuff doesn't work on the TI platforms atm). And if you look at the
> config I posted it also includes bootstage configuration. It also won't
> work well for the SPI tests, which I'm talking with Love about in
> another thread.

Yes, perhaps, but having self-contained tests would be a win.

>
> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> configuration details, specific to both the platform / SoC first, some
> lab specific details second and drop-in existing 3rd party files a
> distant third.

I think the u-boot-test-hooks was an amazing solution 9 years ago, but
we have outgrown it. We want people to be able to connect their lab to
CI (meaning gitlab), so testing is more automated.

We should move away from relying on maintainers getting around to
testing patches months after they are sent, when they have time, but
they don't. Things need to be more automated and I'd encourage you to
push this as well.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 14555 bytes --]

On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > installer starts up correctly.
> > > > > > > > >
> > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > Changes in v2:
> > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > >
> > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > >
> > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > >    variables:
> > > > > > > > >      ROLE: zybo
> > > > > > > > >    <<: *lab_dfn
> > > > > > > > > +
> > > > > > > > > +qemu-x86_64:
> > > > > > > > > +  variables:
> > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > +  <<: *lab_dfn
> > > > > > > >
> > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > test.py QEMU stanza.
> > > > > > >
> > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > >
> > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > think of a good way to cache the download. There's not a particular
> > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > targets and also qemu-riscv64, is there?
> > > > >
> > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > doesn't require all the runners to download a much larger image, etc.
> > > >
> > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > that certain longer or possibly destructive tests are only run on tagged
> > > > releases or as requested rather than every time, as it will take longer.
> > > > But pretty much every platform under the qemu target list should be able
> > > > to Just Boot an off the shelf OS distribution is my point.
> > >
> > > Sure, and I'm not suggesting we shouldn't do that as well.
> > >
> > > >
> > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > new file mode 100644
> > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > +
> > > > > > > > > +import pytest
> > > > > > > > > +
> > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > +
> > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > +
> > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > +def test_distro(ubman):
> > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > +
> > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > +
> > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > +
> > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > +
> > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > +
> > > > > > > > > +        # Go to end of line
> > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > +
> > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > +
> > > > > > > > > +        # Send our noisy console
> > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > +
> > > > > > > > > +        # Tell grub to boot
> > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > +
> > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > +        # Linux should start immediately
> > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > +
> > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > +
> > > > > > > > > +    ubman.restart_uboot()
> > > > > > > >
> > > > > > > > And this seems very inflexible. Please see
> > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > hardware too.
> > > > > > >
> > > > > > > That wasn't the reaction I expected.
> > > > > > >
> > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > than what we have today?
> > > > > >
> > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > test that we have today? No, it's not.
> > > > >
> > > > > I didn't even know about it, or perhaps I forgot.
> > > >
> > > > I believe I mentioned it every time you've said we should have an OS
> > > > test, so yes, I guess you forgot.
> > >
> > > Well it was only added in May last year and it relies on board config
> > > which I don't have...although I see that you have now posted yours.
> >
> > Yes, it was added not quite a year ago, and is documented within the
> > test, like most tests that rely on the real platform.
> >
> > And do we need better documentation for test? Yes.
> 
> +1
> 
> I'll note that I did my bit!
> 
> >
> > > > > Perhaps this relates to getting the labgrid config published and
> > > > > figuring out how to pass info from Labgrid to tests.
> > > > >
> > > > > >
> > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > board, preferably one that doesn't use grub.
> > > > > >
> > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > SD card (some lab sd-mux issues).
> > > > >
> > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > etc. are available.
> > > >
> > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > that can run in any / all of the labs, not just your labgrid
> > > > configuration. AMD has been contributing tests that run on hardware for
> > > > example.
> > >
> > > That's great, the more tests we have the better. But those tests can't
> > > and don't run in CI, whereas mine can and do.
> >
> > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > *your* lab because you took things, intentionally, in a direction to
> > minimize using u-boot-test-hooks and our existing per-board
> > configuration infrastructure.
> 
> When I look at CI all I see is my lab. Which CI are you referring to
> and how can I access it?

I'll point you at the notes for the first call we had recently:
https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
and note that there are many labs doing testing on / with U-Boot.

> Here I would like to make a case for moving to using Labgrid across
> the board, but unfortunately the project struggles to review PRs, so
> it's probably not a good idea.

It would also be counter to the feedback from the U-Boot community about
making it easier to contribute testing results from additional labs.

> > > We need an 'all of the above' strategy here.
> >
> > Sure. But I still want to see things as reusable as possible. What you
> > have above is *extremely* board and OS specific and non-configurable.
> 
> Yes, agreed.
> 
> > I
> > also don't quite see why it's not a test of autoboot with the
> > pre-requisite of an OS being installed.
> 
> Ah OK, my test is just for the installer itself. Both are useful, but
> I hope eventually to have the installer run to completion and then
> reboot to check all is well.

In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
testing Linaro has going now that automates I believe it was current
Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
full install via network in CI. So yes, a Canonical lab might also find
it useful to end to end test installing Ubuntu. My own personal dream is
that at least some of the existing kernelci labs see the utility in
adding "current U-Boot" as one of the matrix variables they test and not
just "U-Boot as delivered by vendor" as a static part of the testing.

> > > BTW, having thought about how test/py works a bit, instead of the
> > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > sets up the required test files (on a suitable server) before running
> > > the test. That way, all the test code is in one Python file and we
> > > don't have to spend ages trying to divine what each test needs.
> >
> > That seems like a lot more work than documenting more what we have
> > today, and I'm not sure of the benefit. Given the contents of the pxe
> > test, yes, just having those files available to 'cp' in place would be
> > helpful. But that's not the case for booting a kernel (the FIT match
> > stuff doesn't work on the TI platforms atm). And if you look at the
> > config I posted it also includes bootstage configuration. It also won't
> > work well for the SPI tests, which I'm talking with Love about in
> > another thread.
> 
> Yes, perhaps, but having self-contained tests would be a win.

With it's own set of technical and legal challenges / obligations and
difficulties depending on what you even mean by "self contained". And
how often what's run where, and all sorts of other challenges too.

Given the extreme depth that testing can go to, this is why I'm of the
position that we need to document things more and worry less about
prepackaged things. For example, making the documentation for the
current net based OS boot means that for bringing up a new board the
developer can just drop something in. Whereas if the tests expect a
functional OS image that has to also be messed with and is its own
challenge.

> > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > configuration details, specific to both the platform / SoC first, some
> > lab specific details second and drop-in existing 3rd party files a
> > distant third.
> 
> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> we have outgrown it. We want people to be able to connect their lab to
> CI (meaning gitlab), so testing is more automated.

More and more public testing would be great. The notes I linked above
explain one of the first problems there being that most companies will
not or can not hook a lab to a public CI instance. The next problem, as
both of our personal labs show, is that just maintaining the physical
lab takes time and resources. I've added Heiko here because I've been
talking with him off-list about expanding tbot coverage and plumbing
that in to gitlab.

> We should move away from relying on maintainers getting around to
> testing patches months after they are sent, when they have time, but
> they don't. Things need to be more automated and I'd encourage you to
> push this as well.

I have been, and the results I've gotten are that companies are testing
things internally but there's not any good way to publish results, and
that's the kind of framework we're entirely missing.

Which is another part of why I keep pushing against having U-Boot
configuration stuff inside of Labgrid as it makes it harder for any lab
that's not using labgrid to see how to configure things.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Heiko Schocher]

Hello Tom,

On 21.02.25 17:06, Tom Rini wrote:
> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
>> Hi Tom,
>>
>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
>>>
>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
>>>> Hi Tom,
>>>>
>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
>>>>>
>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
>>>>>> Hi Tom,
>>>>>>
>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>
>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
>>>>>>>> Hi Tom,
>>>>>>>>
>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>
>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
>>>>>>>>>> installer starts up correctly.
>>>>>>>>>>
>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> Changes in v2:
>>>>>>>>>> - Add more patches to support booting with kvm
>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
>>>>>>>>>>
>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>   2 files changed, 58 insertions(+)
>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
>>>>>>>>>>
>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
>>>>>>>>>> --- a/.gitlab-ci.yml
>>>>>>>>>> +++ b/.gitlab-ci.yml
>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
>>>>>>>>>>     variables:
>>>>>>>>>>       ROLE: zybo
>>>>>>>>>>     <<: *lab_dfn
>>>>>>>>>> +
>>>>>>>>>> +qemu-x86_64:
>>>>>>>>>> +  variables:
>>>>>>>>>> +    ROLE: qemu-x86_64
>>>>>>>>>> +  <<: *lab_dfn
>>>>>>>>>
>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
>>>>>>>>> test.py QEMU stanza.
>>>>>>>>
>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
>>>>>>>
>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
>>>>>>> think of a good way to cache the download. There's not a particular
>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
>>>>>>> targets and also qemu-riscv64, is there?
>>>>>>
>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
>>>>>> doesn't require all the runners to download a much larger image, etc.
>>>>>
>>>>> I don't quite understand why it's under "labgrid". These are generic CI
>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
>>>>> that certain longer or possibly destructive tests are only run on tagged
>>>>> releases or as requested rather than every time, as it will take longer.
>>>>> But pretty much every platform under the qemu target list should be able
>>>>> to Just Boot an off the shelf OS distribution is my point.
>>>>
>>>> Sure, and I'm not suggesting we shouldn't do that as well.
>>>>
>>>>>
>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
>>>>>>>>>> new file mode 100644
>>>>>>>>>> index 00000000000..51eec45cecc
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++ b/test/py/tests/test_distro.py
>>>>>>>>>> @@ -0,0 +1,53 @@
>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
>>>>>>>>>> +
>>>>>>>>>> +import pytest
>>>>>>>>>> +
>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
>>>>>>>>>> +
>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
>>>>>>>>>> +
>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
>>>>>>>>>> +def test_distro(ubman):
>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
>>>>>>>>>> +    with ubman.log.section('boot'):
>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
>>>>>>>>>> +
>>>>>>>>>> +    with ubman.log.section('Grub'):
>>>>>>>>>> +        # Wait for grub to come up and offset a menu
>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
>>>>>>>>>> +
>>>>>>>>>> +        # Press 'e' to edit the command line
>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
>>>>>>>>>> +
>>>>>>>>>> +        # Wait until we see the editor appear
>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
>>>>>>>>>> +
>>>>>>>>>> +        # Go down to the 'linux' line
>>>>>>>>>> +        ubman.send(DOWN * 3)
>>>>>>>>>> +
>>>>>>>>>> +        # Go to end of line
>>>>>>>>>> +        ubman.ctrl('E')
>>>>>>>>>> +
>>>>>>>>>> +        # Backspace to remove 'quiet splash'
>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
>>>>>>>>>> +
>>>>>>>>>> +        # Send our noisy console
>>>>>>>>>> +        ubman.send(CONSOLE)
>>>>>>>>>> +
>>>>>>>>>> +        # Tell grub to boot
>>>>>>>>>> +        ubman.ctrl('X')
>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
>>>>>>>>>> +
>>>>>>>>>> +    with ubman.log.section('Linux'):
>>>>>>>>>> +        # Linux should start immediately
>>>>>>>>>> +        ubman.p.expect(['Linux version'])
>>>>>>>>>> +
>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
>>>>>>>>>> +        # Shortly later, we should see this banner
>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
>>>>>>>>>> +
>>>>>>>>>> +    ubman.restart_uboot()
>>>>>>>>>
>>>>>>>>> And this seems very inflexible. Please see
>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
>>>>>>>>> that the "role" part here is where you have a special disk image being
>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
>>>>>>>>> and the images pre-fetched to the CI container. And if this was
>>>>>>>>> configurable similar to the example I noted above, it could check real
>>>>>>>>> hardware too.

Ah, and now I got the trick, when looking into test/py/tests/test_net_boot.py
how to configure more tests (I think)!

>>>>>>>>
>>>>>>>> That wasn't the reaction I expected.
>>>>>>>>
>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
>>>>>>>> than what we have today?
>>>>>>>
>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
>>>>>>> test that we have today? No, it's not.
>>>>>>
>>>>>> I didn't even know about it, or perhaps I forgot.
>>>>>
>>>>> I believe I mentioned it every time you've said we should have an OS
>>>>> test, so yes, I guess you forgot.
>>>>
>>>> Well it was only added in May last year and it relies on board config
>>>> which I don't have...although I see that you have now posted yours.
>>>
>>> Yes, it was added not quite a year ago, and is documented within the
>>> test, like most tests that rely on the real platform.
>>>
>>> And do we need better documentation for test? Yes.
>>
>> +1
>>
>> I'll note that I did my bit!
>>
>>>
>>>>>> Perhaps this relates to getting the labgrid config published and
>>>>>> figuring out how to pass info from Labgrid to tests.
>>>>>>
>>>>>>>
>>>>>>>> I would like to generalise this test to work on at least one real
>>>>>>>> board, preferably one that doesn't use grub.
>>>>>>>
>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
>>>>>>> ..." string instead of the kernel has booted string. It also does
>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
>>>>>>> to write up. The only thing stopping me from doing that right now is I
>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
>>>>>>> SD card (some lab sd-mux issues).
>>>>>>
>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
>>>>>> etc. are available.
>>>>>
>>>>> OK, but that sounds like the opposite direction. These are generic tests
>>>>> that can run in any / all of the labs, not just your labgrid
>>>>> configuration. AMD has been contributing tests that run on hardware for
>>>>> example.
>>>>
>>>> That's great, the more tests we have the better. But those tests can't
>>>> and don't run in CI, whereas mine can and do.
>>>
>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
>>> *your* lab because you took things, intentionally, in a direction to
>>> minimize using u-boot-test-hooks and our existing per-board
>>> configuration infrastructure.
>>
>> When I look at CI all I see is my lab. Which CI are you referring to
>> and how can I access it?
> 
> I'll point you at the notes for the first call we had recently:
> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> and note that there are many labs doing testing on / with U-Boot.
> 
>> Here I would like to make a case for moving to using Labgrid across
>> the board, but unfortunately the project struggles to review PRs, so
>> it's probably not a good idea.
> 
> It would also be counter to the feedback from the U-Boot community about
> making it easier to contribute testing results from additional labs.

I had years ago a nightly U-Boot build/install/test tbot setup for some
boards I had access to, and collected while the test run data with tbot,
and pushed that data to a DB ... and had a "blog" based webpage which
showed the data from that DB (I am soory, not up and running currently)

(IIRC, I collected, base commit, boardname, resulting binary sizes,
  used toolchain, testresult good/bad,...)

May we should discuss first which data we are interested in, and than define
where and how we store this data. Than we should define an API for adding
results... easiest way could be simple txt emails to ML (generated with tbot
or at least a shell script), so we get the results in a well defined format
and than we can write a script which extracts the info from such an EMail
and pushes it into a DB. So it should be easy for people to create such
a test reoport email, and easy for us to parse this email...

And based on such a DB we can make a small webpage which shows the results...
(May add queries like binary size over the last 100 builds and make a nice
  image, which shows the size growth ... which I had in my old approach)

>>>> We need an 'all of the above' strategy here.
>>>
>>> Sure. But I still want to see things as reusable as possible. What you
>>> have above is *extremely* board and OS specific and non-configurable.
>>
>> Yes, agreed.
>>
>>> I
>>> also don't quite see why it's not a test of autoboot with the
>>> pre-requisite of an OS being installed.
>>
>> Ah OK, my test is just for the installer itself. Both are useful, but
>> I hope eventually to have the installer run to completion and then
>> reboot to check all is well.
> 
> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> testing Linaro has going now that automates I believe it was current
> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> full install via network in CI. So yes, a Canonical lab might also find
> it useful to end to end test installing Ubuntu. My own personal dream is
> that at least some of the existing kernelci labs see the utility in
> adding "current U-Boot" as one of the matrix variables they test and not
> just "U-Boot as delivered by vendor" as a static part of the testing.

Yep, that would be nice.

>>>> BTW, having thought about how test/py works a bit, instead of the
>>>> env__net_tftp_bootable_file stuff, we should have code or data which
>>>> sets up the required test files (on a suitable server) before running
>>>> the test. That way, all the test code is in one Python file and we
>>>> don't have to spend ages trying to divine what each test needs.
>>>
>>> That seems like a lot more work than documenting more what we have
>>> today, and I'm not sure of the benefit. Given the contents of the pxe
>>> test, yes, just having those files available to 'cp' in place would be
>>> helpful. But that's not the case for booting a kernel (the FIT match
>>> stuff doesn't work on the TI platforms atm). And if you look at the
>>> config I posted it also includes bootstage configuration. It also won't
>>> work well for the SPI tests, which I'm talking with Love about in
>>> another thread.
>>
>> Yes, perhaps, but having self-contained tests would be a win.
> 
> With it's own set of technical and legal challenges / obligations and
> difficulties depending on what you even mean by "self contained". And
> how often what's run where, and all sorts of other challenges too.
> 
> Given the extreme depth that testing can go to, this is why I'm of the
> position that we need to document things more and worry less about
> prepackaged things. For example, making the documentation for the
> current net based OS boot means that for bringing up a new board the
> developer can just drop something in. Whereas if the tests expect a
> functional OS image that has to also be messed with and is its own
> challenge.

Just my 1 cent.. without to much knwoledge about current situation/problem,
or test/py at all, hope I write not too big nonsense ... just from a tbot
point of view.

In tbot you request/enter an U-Boot machine, and than tbot for example setups
Environment variables defined for this board/machine.

Example:
https://github.com/hsdenx/u-boot-test/blob/tbottesting/tbottesting/tbotconfig-hs/hs/cxg3.ini#L37

So we can define for each board the Environment which fits for the
tests we want to call (tbot setup should now it!)

So lets, say, we want to boot different linux images, from different
sources with different mounts of rootfs (nfs, emmc,...) we can define
different linux bootcommands (U-Boot environment variables at the end),
like:

tftp_fit_nfs (load fitimage from tftp and boot with nfs rootfs)
tftp_raw_nfs (load raw kernel image with tftp and boot with nfs as rootfs)
emmc_fit_emmc (load fit iamge from emmc and boot with rootfs on emmc)
<your fancy command> ...
[...]

Than we can select in tbot which command we run when we request the
linux machine. May dependend (means not implemented yet) on what specifc
test/py test needs!

When entered linux commandline, you can search strings in the linux
bootlog, you can access with machine.bootlog, or call linux commands
and analyse the output... but I think, that is not to much interesting
in U-Boot tests... but I can think for example of a i2c RTC test, which
sets a date in U-Boot, than enters linux, check the date, set in linux
another date, power off/on (or reboot) to enter U-Boot and check again
the date... no problem with tbot...

So I think, we should define in test/py "stuff we need for a test"
which than lab implementations can parse and provide/setup dependend
on the specific board. test/py may can request such a "feature" from
the lab and if not availiable skip the test?

As I now (after reading this Email often, and look around in test/py)
it seems to me, we already have such "request feature" in test/py by
defining env__xyz variables? Is this correct?

If so, it seems no problem to me, to create that stuff from within tbot,
before it calls test/py!...  as tbot (see below) creates this u-boot hook
scripts already *before* starting test/py! It would be good to have an
overview which variables should be set for which test... do we have such
a doc?

Think about that there are boards in labs where we can do a specific test
but in other labs, the same board skips this test, because missing images
for this tests in that lab! ... may some lab has can or rs485 testing
hardware, the other lab misses this feature ... and tbot "knows the lab/board"
combination and can generate the specifc env__xyz variables (or not) in
hook files...

so as much flexible as possible approach I think ...

 From my side, yes, test/py can use u-boot hook scripts and depend
only on them and a lab integration simply should create that files
before starting test/py and we are fine for both worlds... or?

>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
>>> configuration details, specific to both the platform / SoC first, some
>>> lab specific details second and drop-in existing 3rd party files a
>>> distant third.
>>
>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
>> we have outgrown it. We want people to be able to connect their lab to
>> CI (meaning gitlab), so testing is more automated.
> 
> More and more public testing would be great. The notes I linked above
> explain one of the first problems there being that most companies will
> not or can not hook a lab to a public CI instance. The next problem, as
> both of our personal labs show, is that just maintaining the physical
> lab takes time and resources. I've added Heiko here because I've been
> talking with him off-list about expanding tbot coverage and plumbing
> that in to gitlab.

And yes, as above mentioned, I had this stuff up and running >5 years ago,
but when you test, you find bugs, boards break, you have to fix, you need
time... customers do not want to pay, so it is all done in free time at
the end...

BTW: I added in that old setup that I downloaded the patches from my patchwork
  ToDo list, run checkpatch and apply them to current HEAD, and run the tests
  (which means build/install/test) on the boards... so I only had to look in
  the morning on my webpage, and if all is green, all patches in my patchwork
  todo list are fine... so that makes maintainers work a lot easier...

  May I find time to reactivate this setup again (of course with the new tbot,
  not my old crap...) if people are interested... but yes, we should define
  how to report testresults... (speculating without knowledge ... may we can
  use kernelCI code and start an u-bootCI server?)

  heh,  found a video on youtube from me... 8 years ago! ... wow time is
  running...

  https://www.youtube.com/watch?v=PhaYfqOrQOg

  It also shows a complete git bisect session to find out which patch
  on my patchwork todo list breaks U-Boot ... fully automated...
  (But yes, use the fast forward button, as looking at logs is not that
   interesting, but you can see, that it really worked)

YFI:
My current approach for integrating tbot into gitlab Tom mentioned above:
(Attention: still WIP!)

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/pipelines

unfortunately test/py does not work yet on gitlab:

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/jobs/1032932#L2105

but this is some problem with python module versions, hope I can soon fix it.

*same* tbottesting code works on github fine with test/py

https://github.com/hsdenx/u-boot-test/actions/runs/13412895540
https://github.com/hsdenx/u-boot-test/actions/runs/13412895540/job/37467281787#step:6:2122

(I see a lot of skipped tests... how can I activate them? It seems now
  to me, that I need such u-boot hook files with env__xyz variables in, is
  this correct?)

In short, tbot runs on gitlab or github, board is in my lab in hungary. It is
the imx8qxp based capricorn board, test fetches from lab host the binary blobs,
needed for getting a working flash.bin and get some downstream patches from
lab host and applies them), build, copy the resutling binary flash.bin to the
lab host.

Than set bootmode "USB SDP" and load the flash.bin with the uuu tool, when reached
U-Boots shell, install with U-Boots fastboot mode the flash.bin onto the emmc again
with uuu tool. Set bootmode emmc, powercylce and U-Boot boots from emmc (I check
here that the correct bootmode is in U-Boots bootlog!)

Than I do a small ping test and call ut command, and as Tom requested,
start test/py... more I hopefully can add.

work still on a README for it:

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/blob/tbottesting/tbottesting/README.md?ref_type=heads

The good thing is, I can use the same tbot commands/code during my
daily work... so setting up CI is not that hard or a seperated task.
This is the main goal from tbot, to automate daily developers work,
setting up a CI is than easy... I should have called tbot preferably "abot"
(automation bot ... as it automates machines and interaction between them)

>> We should move away from relying on maintainers getting around to
>> testing patches months after they are sent, when they have time, but
>> they don't. Things need to be more automated and I'd encourage you to
>> push this as well.
> 
> I have been, and the results I've gotten are that companies are testing
> things internally but there's not any good way to publish results, and
> that's the kind of framework we're entirely missing.

Indeed.

> Which is another part of why I keep pushing against having U-Boot
> configuration stuff inside of Labgrid as it makes it harder for any lab
> that's not using labgrid to see how to configure things.

I must admit, that I currently have to learn how to setup test/py
stuff, to get rid of a lot of skipped tests! But first I want to
get it up and running @gitlab.

Hmm.. I also have the complete config in tbot setup... see above, but
I generate the u-boot hooks from within tbot, example:

https://github.com/hsdenx/u-boot-test/actions/runs/13412895540/job/37467281787#step:6:1878

with:
https://tbot.tools/contrib/uboot.html#tbot-contrib-uboot

So, have the test/py config seperated in u-boot-hook scripts is good!
and no showstopper for lab integrations... simply the lab integration
should generate the hook scripts with the correct settings for the
lab and board combination before calling test/py...

I have to admit, that I must dig deeper into test/py as I had the last
years not that much time for it ... hope I wrote not to much nonsense
(sorry in advance if so...)

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs@denx.de

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Heiko Schocher]

Hello Tom,

just a fast update, testing with tbot running @gitlab works now too:

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/jobs/1034474#L2199

problem was:

https://source.denx.de/u-boot/custodians/u-boot-i2c/-/commit/e32624a1ffff320a5e75a7360903f6de75b14a61

So, I try to clean this up, work on the README and hope to add one more
board soon... and try to enable more test/py tests, but first I wait
for some response to previous mail...

Have a nice weekend.

bye,
Heiko

On 22.02.25 09:27, Heiko Schocher wrote:
> Hello Tom,
> 
> On 21.02.25 17:06, Tom Rini wrote:
>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
>>> Hi Tom,
>>>
>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
>>>>
>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
>>>>> Hi Tom,
>>>>>
>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
>>>>>>
>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
>>>>>>> Hi Tom,
>>>>>>>
>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>
>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
>>>>>>>>> Hi Tom,
>>>>>>>>>
>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
>>>>>>>>>>> installer starts up correctly.
>>>>>>>>>>>
>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> Changes in v2:
>>>>>>>>>>> - Add more patches to support booting with kvm
>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
>>>>>>>>>>>
>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>   2 files changed, 58 insertions(+)
>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
>>>>>>>>>>> --- a/.gitlab-ci.yml
>>>>>>>>>>> +++ b/.gitlab-ci.yml
>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
>>>>>>>>>>>     variables:
>>>>>>>>>>>       ROLE: zybo
>>>>>>>>>>>     <<: *lab_dfn
>>>>>>>>>>> +
>>>>>>>>>>> +qemu-x86_64:
>>>>>>>>>>> +  variables:
>>>>>>>>>>> +    ROLE: qemu-x86_64
>>>>>>>>>>> +  <<: *lab_dfn
>>>>>>>>>>
>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
>>>>>>>>>> test.py QEMU stanza.
>>>>>>>>>
>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
>>>>>>>>
>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
>>>>>>>> think of a good way to cache the download. There's not a particular
>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
>>>>>>>> targets and also qemu-riscv64, is there?
>>>>>>>
>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
>>>>>>> doesn't require all the runners to download a much larger image, etc.
>>>>>>
>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
>>>>>> that certain longer or possibly destructive tests are only run on tagged
>>>>>> releases or as requested rather than every time, as it will take longer.
>>>>>> But pretty much every platform under the qemu target list should be able
>>>>>> to Just Boot an off the shelf OS distribution is my point.
>>>>>
>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
>>>>>
>>>>>>
>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
>>>>>>>>>>> new file mode 100644
>>>>>>>>>>> index 00000000000..51eec45cecc
>>>>>>>>>>> --- /dev/null
>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
>>>>>>>>>>> @@ -0,0 +1,53 @@
>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
>>>>>>>>>>> +
>>>>>>>>>>> +import pytest
>>>>>>>>>>> +
>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
>>>>>>>>>>> +
>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
>>>>>>>>>>> +
>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
>>>>>>>>>>> +def test_distro(ubman):
>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
>>>>>>>>>>> +    with ubman.log.section('boot'):
>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
>>>>>>>>>>> +
>>>>>>>>>>> +    with ubman.log.section('Grub'):
>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
>>>>>>>>>>> +
>>>>>>>>>>> +        # Press 'e' to edit the command line
>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
>>>>>>>>>>> +
>>>>>>>>>>> +        # Wait until we see the editor appear
>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
>>>>>>>>>>> +
>>>>>>>>>>> +        # Go down to the 'linux' line
>>>>>>>>>>> +        ubman.send(DOWN * 3)
>>>>>>>>>>> +
>>>>>>>>>>> +        # Go to end of line
>>>>>>>>>>> +        ubman.ctrl('E')
>>>>>>>>>>> +
>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
>>>>>>>>>>> +
>>>>>>>>>>> +        # Send our noisy console
>>>>>>>>>>> +        ubman.send(CONSOLE)
>>>>>>>>>>> +
>>>>>>>>>>> +        # Tell grub to boot
>>>>>>>>>>> +        ubman.ctrl('X')
>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
>>>>>>>>>>> +
>>>>>>>>>>> +    with ubman.log.section('Linux'):
>>>>>>>>>>> +        # Linux should start immediately
>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
>>>>>>>>>>> +
>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
>>>>>>>>>>> +        # Shortly later, we should see this banner
>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
>>>>>>>>>>> +
>>>>>>>>>>> +    ubman.restart_uboot()
>>>>>>>>>>
>>>>>>>>>> And this seems very inflexible. Please see
>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
>>>>>>>>>> that the "role" part here is where you have a special disk image being
>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
>>>>>>>>>> configurable similar to the example I noted above, it could check real
>>>>>>>>>> hardware too.
> 
> Ah, and now I got the trick, when looking into test/py/tests/test_net_boot.py
> how to configure more tests (I think)!
> 
>>>>>>>>>
>>>>>>>>> That wasn't the reaction I expected.
>>>>>>>>>
>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
>>>>>>>>> than what we have today?
>>>>>>>>
>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
>>>>>>>> test that we have today? No, it's not.
>>>>>>>
>>>>>>> I didn't even know about it, or perhaps I forgot.
>>>>>>
>>>>>> I believe I mentioned it every time you've said we should have an OS
>>>>>> test, so yes, I guess you forgot.
>>>>>
>>>>> Well it was only added in May last year and it relies on board config
>>>>> which I don't have...although I see that you have now posted yours.
>>>>
>>>> Yes, it was added not quite a year ago, and is documented within the
>>>> test, like most tests that rely on the real platform.
>>>>
>>>> And do we need better documentation for test? Yes.
>>>
>>> +1
>>>
>>> I'll note that I did my bit!
>>>
>>>>
>>>>>>> Perhaps this relates to getting the labgrid config published and
>>>>>>> figuring out how to pass info from Labgrid to tests.
>>>>>>>
>>>>>>>>
>>>>>>>>> I would like to generalise this test to work on at least one real
>>>>>>>>> board, preferably one that doesn't use grub.
>>>>>>>>
>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
>>>>>>>> ..." string instead of the kernel has booted string. It also does
>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
>>>>>>>> to write up. The only thing stopping me from doing that right now is I
>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
>>>>>>>> SD card (some lab sd-mux issues).
>>>>>>>
>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
>>>>>>> etc. are available.
>>>>>>
>>>>>> OK, but that sounds like the opposite direction. These are generic tests
>>>>>> that can run in any / all of the labs, not just your labgrid
>>>>>> configuration. AMD has been contributing tests that run on hardware for
>>>>>> example.
>>>>>
>>>>> That's great, the more tests we have the better. But those tests can't
>>>>> and don't run in CI, whereas mine can and do.
>>>>
>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
>>>> *your* lab because you took things, intentionally, in a direction to
>>>> minimize using u-boot-test-hooks and our existing per-board
>>>> configuration infrastructure.
>>>
>>> When I look at CI all I see is my lab. Which CI are you referring to
>>> and how can I access it?
>>
>> I'll point you at the notes for the first call we had recently:
>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
>> and note that there are many labs doing testing on / with U-Boot.
>>
>>> Here I would like to make a case for moving to using Labgrid across
>>> the board, but unfortunately the project struggles to review PRs, so
>>> it's probably not a good idea.
>>
>> It would also be counter to the feedback from the U-Boot community about
>> making it easier to contribute testing results from additional labs.
> 
> I had years ago a nightly U-Boot build/install/test tbot setup for some
> boards I had access to, and collected while the test run data with tbot,
> and pushed that data to a DB ... and had a "blog" based webpage which
> showed the data from that DB (I am soory, not up and running currently)
> 
> (IIRC, I collected, base commit, boardname, resulting binary sizes,
>   used toolchain, testresult good/bad,...)
> 
> May we should discuss first which data we are interested in, and than define
> where and how we store this data. Than we should define an API for adding
> results... easiest way could be simple txt emails to ML (generated with tbot
> or at least a shell script), so we get the results in a well defined format
> and than we can write a script which extracts the info from such an EMail
> and pushes it into a DB. So it should be easy for people to create such
> a test reoport email, and easy for us to parse this email...
> 
> And based on such a DB we can make a small webpage which shows the results...
> (May add queries like binary size over the last 100 builds and make a nice
>   image, which shows the size growth ... which I had in my old approach)
> 
>>>>> We need an 'all of the above' strategy here.
>>>>
>>>> Sure. But I still want to see things as reusable as possible. What you
>>>> have above is *extremely* board and OS specific and non-configurable.
>>>
>>> Yes, agreed.
>>>
>>>> I
>>>> also don't quite see why it's not a test of autoboot with the
>>>> pre-requisite of an OS being installed.
>>>
>>> Ah OK, my test is just for the installer itself. Both are useful, but
>>> I hope eventually to have the installer run to completion and then
>>> reboot to check all is well.
>>
>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
>> testing Linaro has going now that automates I believe it was current
>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
>> full install via network in CI. So yes, a Canonical lab might also find
>> it useful to end to end test installing Ubuntu. My own personal dream is
>> that at least some of the existing kernelci labs see the utility in
>> adding "current U-Boot" as one of the matrix variables they test and not
>> just "U-Boot as delivered by vendor" as a static part of the testing.
> 
> Yep, that would be nice.
> 
>>>>> BTW, having thought about how test/py works a bit, instead of the
>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
>>>>> sets up the required test files (on a suitable server) before running
>>>>> the test. That way, all the test code is in one Python file and we
>>>>> don't have to spend ages trying to divine what each test needs.
>>>>
>>>> That seems like a lot more work than documenting more what we have
>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
>>>> test, yes, just having those files available to 'cp' in place would be
>>>> helpful. But that's not the case for booting a kernel (the FIT match
>>>> stuff doesn't work on the TI platforms atm). And if you look at the
>>>> config I posted it also includes bootstage configuration. It also won't
>>>> work well for the SPI tests, which I'm talking with Love about in
>>>> another thread.
>>>
>>> Yes, perhaps, but having self-contained tests would be a win.
>>
>> With it's own set of technical and legal challenges / obligations and
>> difficulties depending on what you even mean by "self contained". And
>> how often what's run where, and all sorts of other challenges too.
>>
>> Given the extreme depth that testing can go to, this is why I'm of the
>> position that we need to document things more and worry less about
>> prepackaged things. For example, making the documentation for the
>> current net based OS boot means that for bringing up a new board the
>> developer can just drop something in. Whereas if the tests expect a
>> functional OS image that has to also be messed with and is its own
>> challenge.
> 
> Just my 1 cent.. without to much knwoledge about current situation/problem,
> or test/py at all, hope I write not too big nonsense ... just from a tbot
> point of view.
> 
> In tbot you request/enter an U-Boot machine, and than tbot for example setups
> Environment variables defined for this board/machine.
> 
> Example:
> https://github.com/hsdenx/u-boot-test/blob/tbottesting/tbottesting/tbotconfig-hs/hs/cxg3.ini#L37
> 
> So we can define for each board the Environment which fits for the
> tests we want to call (tbot setup should now it!)
> 
> So lets, say, we want to boot different linux images, from different
> sources with different mounts of rootfs (nfs, emmc,...) we can define
> different linux bootcommands (U-Boot environment variables at the end),
> like:
> 
> tftp_fit_nfs (load fitimage from tftp and boot with nfs rootfs)
> tftp_raw_nfs (load raw kernel image with tftp and boot with nfs as rootfs)
> emmc_fit_emmc (load fit iamge from emmc and boot with rootfs on emmc)
> <your fancy command> ...
> [...]
> 
> Than we can select in tbot which command we run when we request the
> linux machine. May dependend (means not implemented yet) on what specifc
> test/py test needs!
> 
> When entered linux commandline, you can search strings in the linux
> bootlog, you can access with machine.bootlog, or call linux commands
> and analyse the output... but I think, that is not to much interesting
> in U-Boot tests... but I can think for example of a i2c RTC test, which
> sets a date in U-Boot, than enters linux, check the date, set in linux
> another date, power off/on (or reboot) to enter U-Boot and check again
> the date... no problem with tbot...
> 
> So I think, we should define in test/py "stuff we need for a test"
> which than lab implementations can parse and provide/setup dependend
> on the specific board. test/py may can request such a "feature" from
> the lab and if not availiable skip the test?
> 
> As I now (after reading this Email often, and look around in test/py)
> it seems to me, we already have such "request feature" in test/py by
> defining env__xyz variables? Is this correct?
> 
> If so, it seems no problem to me, to create that stuff from within tbot,
> before it calls test/py!...  as tbot (see below) creates this u-boot hook
> scripts already *before* starting test/py! It would be good to have an
> overview which variables should be set for which test... do we have such
> a doc?
> 
> Think about that there are boards in labs where we can do a specific test
> but in other labs, the same board skips this test, because missing images
> for this tests in that lab! ... may some lab has can or rs485 testing
> hardware, the other lab misses this feature ... and tbot "knows the lab/board"
> combination and can generate the specifc env__xyz variables (or not) in
> hook files...
> 
> so as much flexible as possible approach I think ...
> 
>  From my side, yes, test/py can use u-boot hook scripts and depend
> only on them and a lab integration simply should create that files
> before starting test/py and we are fine for both worlds... or?
> 
>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
>>>> configuration details, specific to both the platform / SoC first, some
>>>> lab specific details second and drop-in existing 3rd party files a
>>>> distant third.
>>>
>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
>>> we have outgrown it. We want people to be able to connect their lab to
>>> CI (meaning gitlab), so testing is more automated.
>>
>> More and more public testing would be great. The notes I linked above
>> explain one of the first problems there being that most companies will
>> not or can not hook a lab to a public CI instance. The next problem, as
>> both of our personal labs show, is that just maintaining the physical
>> lab takes time and resources. I've added Heiko here because I've been
>> talking with him off-list about expanding tbot coverage and plumbing
>> that in to gitlab.
> 
> And yes, as above mentioned, I had this stuff up and running >5 years ago,
> but when you test, you find bugs, boards break, you have to fix, you need
> time... customers do not want to pay, so it is all done in free time at
> the end...
> 
> BTW: I added in that old setup that I downloaded the patches from my patchwork
>   ToDo list, run checkpatch and apply them to current HEAD, and run the tests
>   (which means build/install/test) on the boards... so I only had to look in
>   the morning on my webpage, and if all is green, all patches in my patchwork
>   todo list are fine... so that makes maintainers work a lot easier...
> 
>   May I find time to reactivate this setup again (of course with the new tbot,
>   not my old crap...) if people are interested... but yes, we should define
>   how to report testresults... (speculating without knowledge ... may we can
>   use kernelCI code and start an u-bootCI server?)
> 
>   heh,  found a video on youtube from me... 8 years ago! ... wow time is
>   running...
> 
>   https://www.youtube.com/watch?v=PhaYfqOrQOg
> 
>   It also shows a complete git bisect session to find out which patch
>   on my patchwork todo list breaks U-Boot ... fully automated...
>   (But yes, use the fast forward button, as looking at logs is not that
>    interesting, but you can see, that it really worked)
> 
> YFI:
> My current approach for integrating tbot into gitlab Tom mentioned above:
> (Attention: still WIP!)
> 
> https://source.denx.de/u-boot/custodians/u-boot-i2c/-/pipelines
> 
> unfortunately test/py does not work yet on gitlab:
> 
> https://source.denx.de/u-boot/custodians/u-boot-i2c/-/jobs/1032932#L2105
> 
> but this is some problem with python module versions, hope I can soon fix it.
> 
> *same* tbottesting code works on github fine with test/py
> 
> https://github.com/hsdenx/u-boot-test/actions/runs/13412895540
> https://github.com/hsdenx/u-boot-test/actions/runs/13412895540/job/37467281787#step:6:2122
> 
> (I see a lot of skipped tests... how can I activate them? It seems now
>   to me, that I need such u-boot hook files with env__xyz variables in, is
>   this correct?)
> 
> In short, tbot runs on gitlab or github, board is in my lab in hungary. It is
> the imx8qxp based capricorn board, test fetches from lab host the binary blobs,
> needed for getting a working flash.bin and get some downstream patches from
> lab host and applies them), build, copy the resutling binary flash.bin to the
> lab host.
> 
> Than set bootmode "USB SDP" and load the flash.bin with the uuu tool, when reached
> U-Boots shell, install with U-Boots fastboot mode the flash.bin onto the emmc again
> with uuu tool. Set bootmode emmc, powercylce and U-Boot boots from emmc (I check
> here that the correct bootmode is in U-Boots bootlog!)
> 
> Than I do a small ping test and call ut command, and as Tom requested,
> start test/py... more I hopefully can add.
> 
> work still on a README for it:
> 
> https://source.denx.de/u-boot/custodians/u-boot-i2c/-/blob/tbottesting/tbottesting/README.md?ref_type=heads 
> 
> 
> The good thing is, I can use the same tbot commands/code during my
> daily work... so setting up CI is not that hard or a seperated task.
> This is the main goal from tbot, to automate daily developers work,
> setting up a CI is than easy... I should have called tbot preferably "abot"
> (automation bot ... as it automates machines and interaction between them)
> 
>>> We should move away from relying on maintainers getting around to
>>> testing patches months after they are sent, when they have time, but
>>> they don't. Things need to be more automated and I'd encourage you to
>>> push this as well.
>>
>> I have been, and the results I've gotten are that companies are testing
>> things internally but there's not any good way to publish results, and
>> that's the kind of framework we're entirely missing.
> 
> Indeed.
> 
>> Which is another part of why I keep pushing against having U-Boot
>> configuration stuff inside of Labgrid as it makes it harder for any lab
>> that's not using labgrid to see how to configure things.
> 
> I must admit, that I currently have to learn how to setup test/py
> stuff, to get rid of a lot of skipped tests! But first I want to
> get it up and running @gitlab.
> 
> Hmm.. I also have the complete config in tbot setup... see above, but
> I generate the u-boot hooks from within tbot, example:
> 
> https://github.com/hsdenx/u-boot-test/actions/runs/13412895540/job/37467281787#step:6:1878
> 
> with:
> https://tbot.tools/contrib/uboot.html#tbot-contrib-uboot
> 
> So, have the test/py config seperated in u-boot-hook scripts is good!
> and no showstopper for lab integrations... simply the lab integration
> should generate the hook scripts with the correct settings for the
> lab and board combination before calling test/py...
> 
> I have to admit, that I must dig deeper into test/py as I had the last
> years not that much time for it ... hope I wrote not to much nonsense
> (sorry in advance if so...)
> 
> bye,
> Heiko

-- 
DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs@denx.de

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
>
> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > >
> > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > installer starts up correctly.
> > > > > > > > > >
> > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > Changes in v2:
> > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > >
> > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > >
> > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > >    variables:
> > > > > > > > > >      ROLE: zybo
> > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > +
> > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > +  variables:
> > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > >
> > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > test.py QEMU stanza.
> > > > > > > >
> > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > >
> > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > targets and also qemu-riscv64, is there?
> > > > > >
> > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > >
> > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > releases or as requested rather than every time, as it will take longer.
> > > > > But pretty much every platform under the qemu target list should be able
> > > > > to Just Boot an off the shelf OS distribution is my point.
> > > >
> > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > >
> > > > >
> > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > new file mode 100644
> > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > --- /dev/null
> > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > +
> > > > > > > > > > +import pytest
> > > > > > > > > > +
> > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > +
> > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > +
> > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > +
> > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > +
> > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > +
> > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > +
> > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > +
> > > > > > > > > > +        # Go to end of line
> > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > +
> > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > +
> > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > +
> > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > +
> > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > +
> > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > +
> > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > >
> > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > hardware too.
> > > > > > > >
> > > > > > > > That wasn't the reaction I expected.
> > > > > > > >
> > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > than what we have today?
> > > > > > >
> > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > test that we have today? No, it's not.
> > > > > >
> > > > > > I didn't even know about it, or perhaps I forgot.
> > > > >
> > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > test, so yes, I guess you forgot.
> > > >
> > > > Well it was only added in May last year and it relies on board config
> > > > which I don't have...although I see that you have now posted yours.
> > >
> > > Yes, it was added not quite a year ago, and is documented within the
> > > test, like most tests that rely on the real platform.
> > >
> > > And do we need better documentation for test? Yes.
> >
> > +1
> >
> > I'll note that I did my bit!
> >
> > >
> > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > >
> > > > > > >
> > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > board, preferably one that doesn't use grub.
> > > > > > >
> > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > SD card (some lab sd-mux issues).
> > > > > >
> > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > etc. are available.
> > > > >
> > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > that can run in any / all of the labs, not just your labgrid
> > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > example.
> > > >
> > > > That's great, the more tests we have the better. But those tests can't
> > > > and don't run in CI, whereas mine can and do.
> > >
> > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > *your* lab because you took things, intentionally, in a direction to
> > > minimize using u-boot-test-hooks and our existing per-board
> > > configuration infrastructure.
> >
> > When I look at CI all I see is my lab. Which CI are you referring to
> > and how can I access it?
>
> I'll point you at the notes for the first call we had recently:
> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> and note that there are many labs doing testing on / with U-Boot.

That's all good, but it isn't as good as having the lab in gitlab.

>
> > Here I would like to make a case for moving to using Labgrid across
> > the board, but unfortunately the project struggles to review PRs, so
> > it's probably not a good idea.
>
> It would also be counter to the feedback from the U-Boot community about
> making it easier to contribute testing results from additional labs.

I really don't think the test hooks are a good setup, though. It is
OK-ish for small labs, but it is so fiddly to use that I wrote a tool
(Labman) to deal with all the confusion.

Labgrid (which you suggested I use for my lab, if you recall),
provides for two yaml configuration files so that everything is in one
place. Apart from its primitive support for USB hubs, it is much
easier to maintain that dozens of little files all over the palce.

>
> > > > We need an 'all of the above' strategy here.
> > >
> > > Sure. But I still want to see things as reusable as possible. What you
> > > have above is *extremely* board and OS specific and non-configurable.
> >
> > Yes, agreed.
> >
> > > I
> > > also don't quite see why it's not a test of autoboot with the
> > > pre-requisite of an OS being installed.
> >
> > Ah OK, my test is just for the installer itself. Both are useful, but
> > I hope eventually to have the installer run to completion and then
> > reboot to check all is well.
>
> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> testing Linaro has going now that automates I believe it was current
> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> full install via network in CI. So yes, a Canonical lab might also find
> it useful to end to end test installing Ubuntu. My own personal dream is
> that at least some of the existing kernelci labs see the utility in
> adding "current U-Boot" as one of the matrix variables they test and not
> just "U-Boot as delivered by vendor" as a static part of the testing.

OK.

>
> > > > BTW, having thought about how test/py works a bit, instead of the
> > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > sets up the required test files (on a suitable server) before running
> > > > the test. That way, all the test code is in one Python file and we
> > > > don't have to spend ages trying to divine what each test needs.
> > >
> > > That seems like a lot more work than documenting more what we have
> > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > test, yes, just having those files available to 'cp' in place would be
> > > helpful. But that's not the case for booting a kernel (the FIT match
> > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > config I posted it also includes bootstage configuration. It also won't
> > > work well for the SPI tests, which I'm talking with Love about in
> > > another thread.
> >
> > Yes, perhaps, but having self-contained tests would be a win.
>
> With it's own set of technical and legal challenges / obligations and
> difficulties depending on what you even mean by "self contained". And
> how often what's run where, and all sorts of other challenges too.
>
> Given the extreme depth that testing can go to, this is why I'm of the
> position that we need to document things more and worry less about
> prepackaged things. For example, making the documentation for the
> current net based OS boot means that for bringing up a new board the
> developer can just drop something in. Whereas if the tests expect a
> functional OS image that has to also be messed with and is its own
> challenge.

Yes

>
> > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > configuration details, specific to both the platform / SoC first, some
> > > lab specific details second and drop-in existing 3rd party files a
> > > distant third.
> >
> > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > we have outgrown it. We want people to be able to connect their lab to
> > CI (meaning gitlab), so testing is more automated.
>
> More and more public testing would be great. The notes I linked above
> explain one of the first problems there being that most companies will
> not or can not hook a lab to a public CI instance.

Well corporate IT is what it is.

That means that their boards will not be testing in CI, unless they do
it themselves, right?

> The next problem, as
> both of our personal labs show, is that just maintaining the physical
> lab takes time and resources. I've added Heiko here because I've been
> talking with him off-list about expanding tbot coverage and plumbing
> that in to gitlab.

OK

>
> > We should move away from relying on maintainers getting around to
> > testing patches months after they are sent, when they have time, but
> > they don't. Things need to be more automated and I'd encourage you to
> > push this as well.
>
> I have been, and the results I've gotten are that companies are testing
> things internally but there's not any good way to publish results, and
> that's the kind of framework we're entirely missing.

If you like, but from my side, I like to see the results in gitlab.

>
> Which is another part of why I keep pushing against having U-Boot
> configuration stuff inside of Labgrid as it makes it harder for any lab
> that's not using labgrid to see how to configure things.

Well, as you requested, I looked at Labgrid and now my lab uses it. I
am happy to publish the config[1], but I still hold my view that all
the shell scripts in u-boot-test-hooks are limiting and painful to
work with.

Anyway, back to this patch, I believe it is a useful test.

Regards,
Simon

[1] https://patchwork.ozlabs.org/project/uboot/list/?series=445563

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 18572 bytes --]

On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > >
> > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > ---
> > > > > > > > > > >
> > > > > > > > > > > Changes in v2:
> > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > >
> > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > >
> > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > >    variables:
> > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > +
> > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > +  variables:
> > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > >
> > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > test.py QEMU stanza.
> > > > > > > > >
> > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > >
> > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > >
> > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > >
> > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > >
> > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > >
> > > > > >
> > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > new file mode 100644
> > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > --- /dev/null
> > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > +
> > > > > > > > > > > +import pytest
> > > > > > > > > > > +
> > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > +
> > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > +
> > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > +
> > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > +
> > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > +
> > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > +
> > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > +
> > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > +
> > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > +
> > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > +
> > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > +
> > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > +
> > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > +
> > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > >
> > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > hardware too.
> > > > > > > > >
> > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > >
> > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > than what we have today?
> > > > > > > >
> > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > test that we have today? No, it's not.
> > > > > > >
> > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > >
> > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > test, so yes, I guess you forgot.
> > > > >
> > > > > Well it was only added in May last year and it relies on board config
> > > > > which I don't have...although I see that you have now posted yours.
> > > >
> > > > Yes, it was added not quite a year ago, and is documented within the
> > > > test, like most tests that rely on the real platform.
> > > >
> > > > And do we need better documentation for test? Yes.
> > >
> > > +1
> > >
> > > I'll note that I did my bit!
> > >
> > > >
> > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > >
> > > > > > > >
> > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > >
> > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > SD card (some lab sd-mux issues).
> > > > > > >
> > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > etc. are available.
> > > > > >
> > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > example.
> > > > >
> > > > > That's great, the more tests we have the better. But those tests can't
> > > > > and don't run in CI, whereas mine can and do.
> > > >
> > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > *your* lab because you took things, intentionally, in a direction to
> > > > minimize using u-boot-test-hooks and our existing per-board
> > > > configuration infrastructure.
> > >
> > > When I look at CI all I see is my lab. Which CI are you referring to
> > > and how can I access it?
> >
> > I'll point you at the notes for the first call we had recently:
> > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > and note that there are many labs doing testing on / with U-Boot.
> 
> That's all good, but it isn't as good as having the lab in gitlab.

Strongly disagree. Especially since having it in the mainline gitlab
isn't feasible.

> > > Here I would like to make a case for moving to using Labgrid across
> > > the board, but unfortunately the project struggles to review PRs, so
> > > it's probably not a good idea.
> >
> > It would also be counter to the feedback from the U-Boot community about
> > making it easier to contribute testing results from additional labs.
> 
> I really don't think the test hooks are a good setup, though. It is
> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> (Labman) to deal with all the confusion.

Yes, I don't know how hard you evaluated all of the then-current lab
management tooling and wrote your own.

> Labgrid (which you suggested I use for my lab, if you recall),

Yes, and I think you forgot the aim was to make it easy to show all of
the existing Labgrid based labs that do Linux kernel testing they could
easily add U-Boot to the mix. I've been trying to get feedback from
other people with existing labgrid setups to look at what you've done.

> provides for two yaml configuration files so that everything is in one
> place. Apart from its primitive support for USB hubs, it is much
> easier to maintain that dozens of little files all over the palce.

I mean, I looked at what you posted and strongly disagree, but I think
both cases here are personal preference and not some sort of objective
and easily evaluated thing.

> > > > > We need an 'all of the above' strategy here.
> > > >
> > > > Sure. But I still want to see things as reusable as possible. What you
> > > > have above is *extremely* board and OS specific and non-configurable.
> > >
> > > Yes, agreed.
> > >
> > > > I
> > > > also don't quite see why it's not a test of autoboot with the
> > > > pre-requisite of an OS being installed.
> > >
> > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > I hope eventually to have the installer run to completion and then
> > > reboot to check all is well.
> >
> > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > testing Linaro has going now that automates I believe it was current
> > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > full install via network in CI. So yes, a Canonical lab might also find
> > it useful to end to end test installing Ubuntu. My own personal dream is
> > that at least some of the existing kernelci labs see the utility in
> > adding "current U-Boot" as one of the matrix variables they test and not
> > just "U-Boot as delivered by vendor" as a static part of the testing.
> 
> OK.
> 
> >
> > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > sets up the required test files (on a suitable server) before running
> > > > > the test. That way, all the test code is in one Python file and we
> > > > > don't have to spend ages trying to divine what each test needs.
> > > >
> > > > That seems like a lot more work than documenting more what we have
> > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > test, yes, just having those files available to 'cp' in place would be
> > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > config I posted it also includes bootstage configuration. It also won't
> > > > work well for the SPI tests, which I'm talking with Love about in
> > > > another thread.
> > >
> > > Yes, perhaps, but having self-contained tests would be a win.
> >
> > With it's own set of technical and legal challenges / obligations and
> > difficulties depending on what you even mean by "self contained". And
> > how often what's run where, and all sorts of other challenges too.
> >
> > Given the extreme depth that testing can go to, this is why I'm of the
> > position that we need to document things more and worry less about
> > prepackaged things. For example, making the documentation for the
> > current net based OS boot means that for bringing up a new board the
> > developer can just drop something in. Whereas if the tests expect a
> > functional OS image that has to also be messed with and is its own
> > challenge.
> 
> Yes
> 
> >
> > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > configuration details, specific to both the platform / SoC first, some
> > > > lab specific details second and drop-in existing 3rd party files a
> > > > distant third.
> > >
> > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > we have outgrown it. We want people to be able to connect their lab to
> > > CI (meaning gitlab), so testing is more automated.
> >
> > More and more public testing would be great. The notes I linked above
> > explain one of the first problems there being that most companies will
> > not or can not hook a lab to a public CI instance.
> 
> Well corporate IT is what it is.
> 
> That means that their boards will not be testing in CI, unless they do
> it themselves, right?

I'm not sure what you mean here. It's a solved problem for them (monitor
tree at URL) and something that's been being done since the beginning
even for U-Boot (it's how the original nvidia lab worked).

> > The next problem, as
> > both of our personal labs show, is that just maintaining the physical
> > lab takes time and resources. I've added Heiko here because I've been
> > talking with him off-list about expanding tbot coverage and plumbing
> > that in to gitlab.
> 
> OK
> 
> >
> > > We should move away from relying on maintainers getting around to
> > > testing patches months after they are sent, when they have time, but
> > > they don't. Things need to be more automated and I'd encourage you to
> > > push this as well.
> >
> > I have been, and the results I've gotten are that companies are testing
> > things internally but there's not any good way to publish results, and
> > that's the kind of framework we're entirely missing.
> 
> If you like, but from my side, I like to see the results in gitlab.

Depends on what you mean by gitlab. I assume you mean "triggered by a
push and visible in the main pipeline". Which isn't possible. It's not
going to happen. External collection is how it's handled for the linux
kernel and that community has far more sway than we do. If we ride their
coattails here so to speak, we can get results. If we push for something
completely different we aren't likely to have success.

> > Which is another part of why I keep pushing against having U-Boot
> > configuration stuff inside of Labgrid as it makes it harder for any lab
> > that's not using labgrid to see how to configure things.
> 
> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> am happy to publish the config[1], but I still hold my view that all
> the shell scripts in u-boot-test-hooks are limiting and painful to
> work with.

Yes, and as I've shown, you can also use labgrid without going down the
same path you took, and we can also support other lab management methods
too, which is important to get as much testing as possible without
needing to centralize everything.

> Anyway, back to this patch, I believe it is a useful test.

It's just a shame it's so specifically written.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
>
> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > >
> > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > >
> > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > ---
> > > > > > > > > > > >
> > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > >
> > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > >
> > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > >    variables:
> > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > +
> > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > +  variables:
> > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > >
> > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > >
> > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > >
> > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > >
> > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > >
> > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > >
> > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > >
> > > > > > >
> > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > +
> > > > > > > > > > > > +import pytest
> > > > > > > > > > > > +
> > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > +
> > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > +
> > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > +
> > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > +
> > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > +
> > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > +
> > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > +
> > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > >
> > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > hardware too.
> > > > > > > > > >
> > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > >
> > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > than what we have today?
> > > > > > > > >
> > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > test that we have today? No, it's not.
> > > > > > > >
> > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > >
> > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > test, so yes, I guess you forgot.
> > > > > >
> > > > > > Well it was only added in May last year and it relies on board config
> > > > > > which I don't have...although I see that you have now posted yours.
> > > > >
> > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > test, like most tests that rely on the real platform.
> > > > >
> > > > > And do we need better documentation for test? Yes.
> > > >
> > > > +1
> > > >
> > > > I'll note that I did my bit!
> > > >
> > > > >
> > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > >
> > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > >
> > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > etc. are available.
> > > > > > >
> > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > example.
> > > > > >
> > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > and don't run in CI, whereas mine can and do.
> > > > >
> > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > configuration infrastructure.
> > > >
> > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > and how can I access it?
> > >
> > > I'll point you at the notes for the first call we had recently:
> > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > and note that there are many labs doing testing on / with U-Boot.
> >
> > That's all good, but it isn't as good as having the lab in gitlab.
>
> Strongly disagree. Especially since having it in the mainline gitlab
> isn't feasible.
>
> > > > Here I would like to make a case for moving to using Labgrid across
> > > > the board, but unfortunately the project struggles to review PRs, so
> > > > it's probably not a good idea.
> > >
> > > It would also be counter to the feedback from the U-Boot community about
> > > making it easier to contribute testing results from additional labs.
> >
> > I really don't think the test hooks are a good setup, though. It is
> > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > (Labman) to deal with all the confusion.
>
> Yes, I don't know how hard you evaluated all of the then-current lab
> management tooling and wrote your own.
>
> > Labgrid (which you suggested I use for my lab, if you recall),
>
> Yes, and I think you forgot the aim was to make it easy to show all of
> the existing Labgrid based labs that do Linux kernel testing they could
> easily add U-Boot to the mix. I've been trying to get feedback from
> other people with existing labgrid setups to look at what you've done.
>
> > provides for two yaml configuration files so that everything is in one
> > place. Apart from its primitive support for USB hubs, it is much
> > easier to maintain that dozens of little files all over the palce.
>
> I mean, I looked at what you posted and strongly disagree, but I think
> both cases here are personal preference and not some sort of objective
> and easily evaluated thing.
>
> > > > > > We need an 'all of the above' strategy here.
> > > > >
> > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > have above is *extremely* board and OS specific and non-configurable.
> > > >
> > > > Yes, agreed.
> > > >
> > > > > I
> > > > > also don't quite see why it's not a test of autoboot with the
> > > > > pre-requisite of an OS being installed.
> > > >
> > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > I hope eventually to have the installer run to completion and then
> > > > reboot to check all is well.
> > >
> > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > testing Linaro has going now that automates I believe it was current
> > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > full install via network in CI. So yes, a Canonical lab might also find
> > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > that at least some of the existing kernelci labs see the utility in
> > > adding "current U-Boot" as one of the matrix variables they test and not
> > > just "U-Boot as delivered by vendor" as a static part of the testing.
> >
> > OK.
> >
> > >
> > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > sets up the required test files (on a suitable server) before running
> > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > don't have to spend ages trying to divine what each test needs.
> > > > >
> > > > > That seems like a lot more work than documenting more what we have
> > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > another thread.
> > > >
> > > > Yes, perhaps, but having self-contained tests would be a win.
> > >
> > > With it's own set of technical and legal challenges / obligations and
> > > difficulties depending on what you even mean by "self contained". And
> > > how often what's run where, and all sorts of other challenges too.
> > >
> > > Given the extreme depth that testing can go to, this is why I'm of the
> > > position that we need to document things more and worry less about
> > > prepackaged things. For example, making the documentation for the
> > > current net based OS boot means that for bringing up a new board the
> > > developer can just drop something in. Whereas if the tests expect a
> > > functional OS image that has to also be messed with and is its own
> > > challenge.
> >
> > Yes
> >
> > >
> > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > configuration details, specific to both the platform / SoC first, some
> > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > distant third.
> > > >
> > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > we have outgrown it. We want people to be able to connect their lab to
> > > > CI (meaning gitlab), so testing is more automated.
> > >
> > > More and more public testing would be great. The notes I linked above
> > > explain one of the first problems there being that most companies will
> > > not or can not hook a lab to a public CI instance.
> >
> > Well corporate IT is what it is.
> >
> > That means that their boards will not be testing in CI, unless they do
> > it themselves, right?
>
> I'm not sure what you mean here. It's a solved problem for them (monitor
> tree at URL) and something that's been being done since the beginning
> even for U-Boot (it's how the original nvidia lab worked).
>
> > > The next problem, as
> > > both of our personal labs show, is that just maintaining the physical
> > > lab takes time and resources. I've added Heiko here because I've been
> > > talking with him off-list about expanding tbot coverage and plumbing
> > > that in to gitlab.
> >
> > OK
> >
> > >
> > > > We should move away from relying on maintainers getting around to
> > > > testing patches months after they are sent, when they have time, but
> > > > they don't. Things need to be more automated and I'd encourage you to
> > > > push this as well.
> > >
> > > I have been, and the results I've gotten are that companies are testing
> > > things internally but there's not any good way to publish results, and
> > > that's the kind of framework we're entirely missing.
> >
> > If you like, but from my side, I like to see the results in gitlab.
>
> Depends on what you mean by gitlab. I assume you mean "triggered by a
> push and visible in the main pipeline". Which isn't possible. It's not
> going to happen. External collection is how it's handled for the linux
> kernel and that community has far more sway than we do. If we ride their
> coattails here so to speak, we can get results. If we push for something
> completely different we aren't likely to have success.
>
> > > Which is another part of why I keep pushing against having U-Boot
> > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > that's not using labgrid to see how to configure things.
> >
> > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > am happy to publish the config[1], but I still hold my view that all
> > the shell scripts in u-boot-test-hooks are limiting and painful to
> > work with.
>
> Yes, and as I've shown, you can also use labgrid without going down the
> same path you took, and we can also support other lab management methods
> too, which is important to get as much testing as possible without
> needing to centralize everything.

In summary, I suppose we just have different visions and ideas, which
should be a good thing.

>
> > Anyway, back to this patch, I believe it is a useful test.
>
> It's just a shame it's so specifically written.

It will expand over time, it's just a starting point.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 20316 bytes --]

On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > > Hi Tom,
> > > > > > > > > > >
> > > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > > ---
> > > > > > > > > > > > >
> > > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > > >
> > > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > > >
> > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > > >    variables:
> > > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > > +  variables:
> > > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > > >
> > > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > > >
> > > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > > >
> > > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > > >
> > > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > > >
> > > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > > >
> > > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > >
> > > > > > > >
> > > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +import pytest
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > > >
> > > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > > hardware too.
> > > > > > > > > > >
> > > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > > >
> > > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > > than what we have today?
> > > > > > > > > >
> > > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > > test that we have today? No, it's not.
> > > > > > > > >
> > > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > > >
> > > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > > test, so yes, I guess you forgot.
> > > > > > >
> > > > > > > Well it was only added in May last year and it relies on board config
> > > > > > > which I don't have...although I see that you have now posted yours.
> > > > > >
> > > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > > test, like most tests that rely on the real platform.
> > > > > >
> > > > > > And do we need better documentation for test? Yes.
> > > > >
> > > > > +1
> > > > >
> > > > > I'll note that I did my bit!
> > > > >
> > > > > >
> > > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > > >
> > > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > > >
> > > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > > etc. are available.
> > > > > > > >
> > > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > > example.
> > > > > > >
> > > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > > and don't run in CI, whereas mine can and do.
> > > > > >
> > > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > > configuration infrastructure.
> > > > >
> > > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > > and how can I access it?
> > > >
> > > > I'll point you at the notes for the first call we had recently:
> > > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > and note that there are many labs doing testing on / with U-Boot.
> > >
> > > That's all good, but it isn't as good as having the lab in gitlab.
> >
> > Strongly disagree. Especially since having it in the mainline gitlab
> > isn't feasible.
> >
> > > > > Here I would like to make a case for moving to using Labgrid across
> > > > > the board, but unfortunately the project struggles to review PRs, so
> > > > > it's probably not a good idea.
> > > >
> > > > It would also be counter to the feedback from the U-Boot community about
> > > > making it easier to contribute testing results from additional labs.
> > >
> > > I really don't think the test hooks are a good setup, though. It is
> > > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > (Labman) to deal with all the confusion.
> >
> > Yes, I don't know how hard you evaluated all of the then-current lab
> > management tooling and wrote your own.
> >
> > > Labgrid (which you suggested I use for my lab, if you recall),
> >
> > Yes, and I think you forgot the aim was to make it easy to show all of
> > the existing Labgrid based labs that do Linux kernel testing they could
> > easily add U-Boot to the mix. I've been trying to get feedback from
> > other people with existing labgrid setups to look at what you've done.
> >
> > > provides for two yaml configuration files so that everything is in one
> > > place. Apart from its primitive support for USB hubs, it is much
> > > easier to maintain that dozens of little files all over the palce.
> >
> > I mean, I looked at what you posted and strongly disagree, but I think
> > both cases here are personal preference and not some sort of objective
> > and easily evaluated thing.
> >
> > > > > > > We need an 'all of the above' strategy here.
> > > > > >
> > > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > > have above is *extremely* board and OS specific and non-configurable.
> > > > >
> > > > > Yes, agreed.
> > > > >
> > > > > > I
> > > > > > also don't quite see why it's not a test of autoboot with the
> > > > > > pre-requisite of an OS being installed.
> > > > >
> > > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > I hope eventually to have the installer run to completion and then
> > > > > reboot to check all is well.
> > > >
> > > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > testing Linaro has going now that automates I believe it was current
> > > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > full install via network in CI. So yes, a Canonical lab might also find
> > > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > > that at least some of the existing kernelci labs see the utility in
> > > > adding "current U-Boot" as one of the matrix variables they test and not
> > > > just "U-Boot as delivered by vendor" as a static part of the testing.
> > >
> > > OK.
> > >
> > > >
> > > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > > sets up the required test files (on a suitable server) before running
> > > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > > don't have to spend ages trying to divine what each test needs.
> > > > > >
> > > > > > That seems like a lot more work than documenting more what we have
> > > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > > another thread.
> > > > >
> > > > > Yes, perhaps, but having self-contained tests would be a win.
> > > >
> > > > With it's own set of technical and legal challenges / obligations and
> > > > difficulties depending on what you even mean by "self contained". And
> > > > how often what's run where, and all sorts of other challenges too.
> > > >
> > > > Given the extreme depth that testing can go to, this is why I'm of the
> > > > position that we need to document things more and worry less about
> > > > prepackaged things. For example, making the documentation for the
> > > > current net based OS boot means that for bringing up a new board the
> > > > developer can just drop something in. Whereas if the tests expect a
> > > > functional OS image that has to also be messed with and is its own
> > > > challenge.
> > >
> > > Yes
> > >
> > > >
> > > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > configuration details, specific to both the platform / SoC first, some
> > > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > > distant third.
> > > > >
> > > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > we have outgrown it. We want people to be able to connect their lab to
> > > > > CI (meaning gitlab), so testing is more automated.
> > > >
> > > > More and more public testing would be great. The notes I linked above
> > > > explain one of the first problems there being that most companies will
> > > > not or can not hook a lab to a public CI instance.
> > >
> > > Well corporate IT is what it is.
> > >
> > > That means that their boards will not be testing in CI, unless they do
> > > it themselves, right?
> >
> > I'm not sure what you mean here. It's a solved problem for them (monitor
> > tree at URL) and something that's been being done since the beginning
> > even for U-Boot (it's how the original nvidia lab worked).
> >
> > > > The next problem, as
> > > > both of our personal labs show, is that just maintaining the physical
> > > > lab takes time and resources. I've added Heiko here because I've been
> > > > talking with him off-list about expanding tbot coverage and plumbing
> > > > that in to gitlab.
> > >
> > > OK
> > >
> > > >
> > > > > We should move away from relying on maintainers getting around to
> > > > > testing patches months after they are sent, when they have time, but
> > > > > they don't. Things need to be more automated and I'd encourage you to
> > > > > push this as well.
> > > >
> > > > I have been, and the results I've gotten are that companies are testing
> > > > things internally but there's not any good way to publish results, and
> > > > that's the kind of framework we're entirely missing.
> > >
> > > If you like, but from my side, I like to see the results in gitlab.
> >
> > Depends on what you mean by gitlab. I assume you mean "triggered by a
> > push and visible in the main pipeline". Which isn't possible. It's not
> > going to happen. External collection is how it's handled for the linux
> > kernel and that community has far more sway than we do. If we ride their
> > coattails here so to speak, we can get results. If we push for something
> > completely different we aren't likely to have success.
> >
> > > > Which is another part of why I keep pushing against having U-Boot
> > > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > that's not using labgrid to see how to configure things.
> > >
> > > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > am happy to publish the config[1], but I still hold my view that all
> > > the shell scripts in u-boot-test-hooks are limiting and painful to
> > > work with.
> >
> > Yes, and as I've shown, you can also use labgrid without going down the
> > same path you took, and we can also support other lab management methods
> > too, which is important to get as much testing as possible without
> > needing to centralize everything.
> 
> In summary, I suppose we just have different visions and ideas, which
> should be a good thing.

So long as the project can speak with one voice, yes. Which all circles
back to why I do not think what you're doing with u-boot.org is at all
helpful.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Heiko Schocher]

Hello Tom,

On 26.02.25 15:35, Tom Rini wrote:
>>> Yes, and as I've shown, you can also use labgrid without going down the
>>> same path you took, and we can also support other lab management methods
>>> too, which is important to get as much testing as possible without
>>> needing to centralize everything.
>> In summary, I suppose we just have different visions and ideas, which
>> should be a good thing.
> So long as the project can speak with one voice, yes. Which all circles
> back to why I do not think what you're doing with u-boot.org is at all
> helpful.

dummy question ... what is the plan for this webaddress?

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs@denx.de

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On Thu, Feb 27, 2025 at 11:11:32AM +0100, Heiko Schocher wrote:
> Hello Tom,
> 
> On 26.02.25 15:35, Tom Rini wrote:
> > > > Yes, and as I've shown, you can also use labgrid without going down the
> > > > same path you took, and we can also support other lab management methods
> > > > too, which is important to get as much testing as possible without
> > > > needing to centralize everything.
> > > In summary, I suppose we just have different visions and ideas, which
> > > should be a good thing.
> > So long as the project can speak with one voice, yes. Which all circles
> > back to why I do not think what you're doing with u-boot.org is at all
> > helpful.
> 
> dummy question ... what is the plan for this webaddress?

It's a very good question. Simon owns the domain. ci.u-boot.org
currently points to his personal gitlab instance and tree and is not a
CNAME for source.denx.de.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
>
> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > > > ---
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > > > >    variables:
> > > > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > > > +  variables:
> > > > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > > > >
> > > > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > > > >
> > > > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > > > >
> > > > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > > > >
> > > > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > > > >
> > > > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +import pytest
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > > > +
> > > > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > > > >
> > > > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > > > hardware too.
> > > > > > > > > > > >
> > > > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > > > than what we have today?
> > > > > > > > > > >
> > > > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > > > test that we have today? No, it's not.
> > > > > > > > > >
> > > > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > > > >
> > > > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > > > test, so yes, I guess you forgot.
> > > > > > > >
> > > > > > > > Well it was only added in May last year and it relies on board config
> > > > > > > > which I don't have...although I see that you have now posted yours.
> > > > > > >
> > > > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > > > test, like most tests that rely on the real platform.
> > > > > > >
> > > > > > > And do we need better documentation for test? Yes.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > I'll note that I did my bit!
> > > > > >
> > > > > > >
> > > > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > > > >
> > > > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > > > >
> > > > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > > > etc. are available.
> > > > > > > > >
> > > > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > > > example.
> > > > > > > >
> > > > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > > > and don't run in CI, whereas mine can and do.
> > > > > > >
> > > > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > > > configuration infrastructure.
> > > > > >
> > > > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > > > and how can I access it?
> > > > >
> > > > > I'll point you at the notes for the first call we had recently:
> > > > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > and note that there are many labs doing testing on / with U-Boot.
> > > >
> > > > That's all good, but it isn't as good as having the lab in gitlab.
> > >
> > > Strongly disagree. Especially since having it in the mainline gitlab
> > > isn't feasible.
> > >
> > > > > > Here I would like to make a case for moving to using Labgrid across
> > > > > > the board, but unfortunately the project struggles to review PRs, so
> > > > > > it's probably not a good idea.
> > > > >
> > > > > It would also be counter to the feedback from the U-Boot community about
> > > > > making it easier to contribute testing results from additional labs.
> > > >
> > > > I really don't think the test hooks are a good setup, though. It is
> > > > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > (Labman) to deal with all the confusion.
> > >
> > > Yes, I don't know how hard you evaluated all of the then-current lab
> > > management tooling and wrote your own.
> > >
> > > > Labgrid (which you suggested I use for my lab, if you recall),
> > >
> > > Yes, and I think you forgot the aim was to make it easy to show all of
> > > the existing Labgrid based labs that do Linux kernel testing they could
> > > easily add U-Boot to the mix. I've been trying to get feedback from
> > > other people with existing labgrid setups to look at what you've done.
> > >
> > > > provides for two yaml configuration files so that everything is in one
> > > > place. Apart from its primitive support for USB hubs, it is much
> > > > easier to maintain that dozens of little files all over the palce.
> > >
> > > I mean, I looked at what you posted and strongly disagree, but I think
> > > both cases here are personal preference and not some sort of objective
> > > and easily evaluated thing.
> > >
> > > > > > > > We need an 'all of the above' strategy here.
> > > > > > >
> > > > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > > > have above is *extremely* board and OS specific and non-configurable.
> > > > > >
> > > > > > Yes, agreed.
> > > > > >
> > > > > > > I
> > > > > > > also don't quite see why it's not a test of autoboot with the
> > > > > > > pre-requisite of an OS being installed.
> > > > > >
> > > > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > > I hope eventually to have the installer run to completion and then
> > > > > > reboot to check all is well.
> > > > >
> > > > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > testing Linaro has going now that automates I believe it was current
> > > > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > full install via network in CI. So yes, a Canonical lab might also find
> > > > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > that at least some of the existing kernelci labs see the utility in
> > > > > adding "current U-Boot" as one of the matrix variables they test and not
> > > > > just "U-Boot as delivered by vendor" as a static part of the testing.
> > > >
> > > > OK.
> > > >
> > > > >
> > > > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > > > sets up the required test files (on a suitable server) before running
> > > > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > > > don't have to spend ages trying to divine what each test needs.
> > > > > > >
> > > > > > > That seems like a lot more work than documenting more what we have
> > > > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > > > another thread.
> > > > > >
> > > > > > Yes, perhaps, but having self-contained tests would be a win.
> > > > >
> > > > > With it's own set of technical and legal challenges / obligations and
> > > > > difficulties depending on what you even mean by "self contained". And
> > > > > how often what's run where, and all sorts of other challenges too.
> > > > >
> > > > > Given the extreme depth that testing can go to, this is why I'm of the
> > > > > position that we need to document things more and worry less about
> > > > > prepackaged things. For example, making the documentation for the
> > > > > current net based OS boot means that for bringing up a new board the
> > > > > developer can just drop something in. Whereas if the tests expect a
> > > > > functional OS image that has to also be messed with and is its own
> > > > > challenge.
> > > >
> > > > Yes
> > > >
> > > > >
> > > > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > > configuration details, specific to both the platform / SoC first, some
> > > > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > > > distant third.
> > > > > >
> > > > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > > we have outgrown it. We want people to be able to connect their lab to
> > > > > > CI (meaning gitlab), so testing is more automated.
> > > > >
> > > > > More and more public testing would be great. The notes I linked above
> > > > > explain one of the first problems there being that most companies will
> > > > > not or can not hook a lab to a public CI instance.
> > > >
> > > > Well corporate IT is what it is.
> > > >
> > > > That means that their boards will not be testing in CI, unless they do
> > > > it themselves, right?
> > >
> > > I'm not sure what you mean here. It's a solved problem for them (monitor
> > > tree at URL) and something that's been being done since the beginning
> > > even for U-Boot (it's how the original nvidia lab worked).
> > >
> > > > > The next problem, as
> > > > > both of our personal labs show, is that just maintaining the physical
> > > > > lab takes time and resources. I've added Heiko here because I've been
> > > > > talking with him off-list about expanding tbot coverage and plumbing
> > > > > that in to gitlab.
> > > >
> > > > OK
> > > >
> > > > >
> > > > > > We should move away from relying on maintainers getting around to
> > > > > > testing patches months after they are sent, when they have time, but
> > > > > > they don't. Things need to be more automated and I'd encourage you to
> > > > > > push this as well.
> > > > >
> > > > > I have been, and the results I've gotten are that companies are testing
> > > > > things internally but there's not any good way to publish results, and
> > > > > that's the kind of framework we're entirely missing.
> > > >
> > > > If you like, but from my side, I like to see the results in gitlab.
> > >
> > > Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > push and visible in the main pipeline". Which isn't possible. It's not
> > > going to happen. External collection is how it's handled for the linux
> > > kernel and that community has far more sway than we do. If we ride their
> > > coattails here so to speak, we can get results. If we push for something
> > > completely different we aren't likely to have success.
> > >
> > > > > Which is another part of why I keep pushing against having U-Boot
> > > > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > that's not using labgrid to see how to configure things.
> > > >
> > > > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > am happy to publish the config[1], but I still hold my view that all
> > > > the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > work with.
> > >
> > > Yes, and as I've shown, you can also use labgrid without going down the
> > > same path you took, and we can also support other lab management methods
> > > too, which is important to get as much testing as possible without
> > > needing to centralize everything.
> >
> > In summary, I suppose we just have different visions and ideas, which
> > should be a good thing.
>
> So long as the project can speak with one voice, yes. Which all circles
> back to why I do not think what you're doing with u-boot.org is at all
> helpful.

I do need a relief valve for when my efforts are blocked.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 22189 bytes --]

On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > > > > Hi Tom,
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > > > > >    variables:
> > > > > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > > > > +  variables:
> > > > > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > > > > >
> > > > > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > > > > >
> > > > > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > > > > >
> > > > > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > > > > >
> > > > > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +import pytest
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > > > > hardware too.
> > > > > > > > > > > > >
> > > > > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > > > > than what we have today?
> > > > > > > > > > > >
> > > > > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > > > > test that we have today? No, it's not.
> > > > > > > > > > >
> > > > > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > > > > >
> > > > > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > > > > test, so yes, I guess you forgot.
> > > > > > > > >
> > > > > > > > > Well it was only added in May last year and it relies on board config
> > > > > > > > > which I don't have...although I see that you have now posted yours.
> > > > > > > >
> > > > > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > > > > test, like most tests that rely on the real platform.
> > > > > > > >
> > > > > > > > And do we need better documentation for test? Yes.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > I'll note that I did my bit!
> > > > > > >
> > > > > > > >
> > > > > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > > > > >
> > > > > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > > > > >
> > > > > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > > > > etc. are available.
> > > > > > > > > >
> > > > > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > > > > example.
> > > > > > > > >
> > > > > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > > > > and don't run in CI, whereas mine can and do.
> > > > > > > >
> > > > > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > > > > configuration infrastructure.
> > > > > > >
> > > > > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > > > > and how can I access it?
> > > > > >
> > > > > > I'll point you at the notes for the first call we had recently:
> > > > > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > > and note that there are many labs doing testing on / with U-Boot.
> > > > >
> > > > > That's all good, but it isn't as good as having the lab in gitlab.
> > > >
> > > > Strongly disagree. Especially since having it in the mainline gitlab
> > > > isn't feasible.
> > > >
> > > > > > > Here I would like to make a case for moving to using Labgrid across
> > > > > > > the board, but unfortunately the project struggles to review PRs, so
> > > > > > > it's probably not a good idea.
> > > > > >
> > > > > > It would also be counter to the feedback from the U-Boot community about
> > > > > > making it easier to contribute testing results from additional labs.
> > > > >
> > > > > I really don't think the test hooks are a good setup, though. It is
> > > > > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > > (Labman) to deal with all the confusion.
> > > >
> > > > Yes, I don't know how hard you evaluated all of the then-current lab
> > > > management tooling and wrote your own.
> > > >
> > > > > Labgrid (which you suggested I use for my lab, if you recall),
> > > >
> > > > Yes, and I think you forgot the aim was to make it easy to show all of
> > > > the existing Labgrid based labs that do Linux kernel testing they could
> > > > easily add U-Boot to the mix. I've been trying to get feedback from
> > > > other people with existing labgrid setups to look at what you've done.
> > > >
> > > > > provides for two yaml configuration files so that everything is in one
> > > > > place. Apart from its primitive support for USB hubs, it is much
> > > > > easier to maintain that dozens of little files all over the palce.
> > > >
> > > > I mean, I looked at what you posted and strongly disagree, but I think
> > > > both cases here are personal preference and not some sort of objective
> > > > and easily evaluated thing.
> > > >
> > > > > > > > > We need an 'all of the above' strategy here.
> > > > > > > >
> > > > > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > > > > have above is *extremely* board and OS specific and non-configurable.
> > > > > > >
> > > > > > > Yes, agreed.
> > > > > > >
> > > > > > > > I
> > > > > > > > also don't quite see why it's not a test of autoboot with the
> > > > > > > > pre-requisite of an OS being installed.
> > > > > > >
> > > > > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > > > I hope eventually to have the installer run to completion and then
> > > > > > > reboot to check all is well.
> > > > > >
> > > > > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > > testing Linaro has going now that automates I believe it was current
> > > > > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > > full install via network in CI. So yes, a Canonical lab might also find
> > > > > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > > that at least some of the existing kernelci labs see the utility in
> > > > > > adding "current U-Boot" as one of the matrix variables they test and not
> > > > > > just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > >
> > > > > OK.
> > > > >
> > > > > >
> > > > > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > > > > sets up the required test files (on a suitable server) before running
> > > > > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > > > > don't have to spend ages trying to divine what each test needs.
> > > > > > > >
> > > > > > > > That seems like a lot more work than documenting more what we have
> > > > > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > > > > another thread.
> > > > > > >
> > > > > > > Yes, perhaps, but having self-contained tests would be a win.
> > > > > >
> > > > > > With it's own set of technical and legal challenges / obligations and
> > > > > > difficulties depending on what you even mean by "self contained". And
> > > > > > how often what's run where, and all sorts of other challenges too.
> > > > > >
> > > > > > Given the extreme depth that testing can go to, this is why I'm of the
> > > > > > position that we need to document things more and worry less about
> > > > > > prepackaged things. For example, making the documentation for the
> > > > > > current net based OS boot means that for bringing up a new board the
> > > > > > developer can just drop something in. Whereas if the tests expect a
> > > > > > functional OS image that has to also be messed with and is its own
> > > > > > challenge.
> > > > >
> > > > > Yes
> > > > >
> > > > > >
> > > > > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > > > configuration details, specific to both the platform / SoC first, some
> > > > > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > > > > distant third.
> > > > > > >
> > > > > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > > > we have outgrown it. We want people to be able to connect their lab to
> > > > > > > CI (meaning gitlab), so testing is more automated.
> > > > > >
> > > > > > More and more public testing would be great. The notes I linked above
> > > > > > explain one of the first problems there being that most companies will
> > > > > > not or can not hook a lab to a public CI instance.
> > > > >
> > > > > Well corporate IT is what it is.
> > > > >
> > > > > That means that their boards will not be testing in CI, unless they do
> > > > > it themselves, right?
> > > >
> > > > I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > tree at URL) and something that's been being done since the beginning
> > > > even for U-Boot (it's how the original nvidia lab worked).
> > > >
> > > > > > The next problem, as
> > > > > > both of our personal labs show, is that just maintaining the physical
> > > > > > lab takes time and resources. I've added Heiko here because I've been
> > > > > > talking with him off-list about expanding tbot coverage and plumbing
> > > > > > that in to gitlab.
> > > > >
> > > > > OK
> > > > >
> > > > > >
> > > > > > > We should move away from relying on maintainers getting around to
> > > > > > > testing patches months after they are sent, when they have time, but
> > > > > > > they don't. Things need to be more automated and I'd encourage you to
> > > > > > > push this as well.
> > > > > >
> > > > > > I have been, and the results I've gotten are that companies are testing
> > > > > > things internally but there's not any good way to publish results, and
> > > > > > that's the kind of framework we're entirely missing.
> > > > >
> > > > > If you like, but from my side, I like to see the results in gitlab.
> > > >
> > > > Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > push and visible in the main pipeline". Which isn't possible. It's not
> > > > going to happen. External collection is how it's handled for the linux
> > > > kernel and that community has far more sway than we do. If we ride their
> > > > coattails here so to speak, we can get results. If we push for something
> > > > completely different we aren't likely to have success.
> > > >
> > > > > > Which is another part of why I keep pushing against having U-Boot
> > > > > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > > that's not using labgrid to see how to configure things.
> > > > >
> > > > > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > > am happy to publish the config[1], but I still hold my view that all
> > > > > the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > > work with.
> > > >
> > > > Yes, and as I've shown, you can also use labgrid without going down the
> > > > same path you took, and we can also support other lab management methods
> > > > too, which is important to get as much testing as possible without
> > > > needing to centralize everything.
> > >
> > > In summary, I suppose we just have different visions and ideas, which
> > > should be a good thing.
> >
> > So long as the project can speak with one voice, yes. Which all circles
> > back to why I do not think what you're doing with u-boot.org is at all
> > helpful.
> 
> I do need a relief valve for when my efforts are blocked.

And I do not see how using the project domain is appropriate. Stop doing
this. If you can't stop I'm going to reach my own limit here.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > >
> > > > > > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > > > > > Hi Tom,
> > > > > > > > > >
> > > > > > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > > > > > >    variables:
> > > > > > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > > > > > +  variables:
> > > > > > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > > > > > >
> > > > > > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > > > > > >
> > > > > > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +import pytest
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > > > > > hardware too.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > > > > > than what we have today?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > > > > > test that we have today? No, it's not.
> > > > > > > > > > > >
> > > > > > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > > > > > >
> > > > > > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > > > > > test, so yes, I guess you forgot.
> > > > > > > > > >
> > > > > > > > > > Well it was only added in May last year and it relies on board config
> > > > > > > > > > which I don't have...although I see that you have now posted yours.
> > > > > > > > >
> > > > > > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > > > > > test, like most tests that rely on the real platform.
> > > > > > > > >
> > > > > > > > > And do we need better documentation for test? Yes.
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > I'll note that I did my bit!
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > > > > > >
> > > > > > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > > > > > >
> > > > > > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > > > > > etc. are available.
> > > > > > > > > > >
> > > > > > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > > > > > example.
> > > > > > > > > >
> > > > > > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > > > > > and don't run in CI, whereas mine can and do.
> > > > > > > > >
> > > > > > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > > > > > configuration infrastructure.
> > > > > > > >
> > > > > > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > > > > > and how can I access it?
> > > > > > >
> > > > > > > I'll point you at the notes for the first call we had recently:
> > > > > > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > > > and note that there are many labs doing testing on / with U-Boot.
> > > > > >
> > > > > > That's all good, but it isn't as good as having the lab in gitlab.
> > > > >
> > > > > Strongly disagree. Especially since having it in the mainline gitlab
> > > > > isn't feasible.
> > > > >
> > > > > > > > Here I would like to make a case for moving to using Labgrid across
> > > > > > > > the board, but unfortunately the project struggles to review PRs, so
> > > > > > > > it's probably not a good idea.
> > > > > > >
> > > > > > > It would also be counter to the feedback from the U-Boot community about
> > > > > > > making it easier to contribute testing results from additional labs.
> > > > > >
> > > > > > I really don't think the test hooks are a good setup, though. It is
> > > > > > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > > > (Labman) to deal with all the confusion.
> > > > >
> > > > > Yes, I don't know how hard you evaluated all of the then-current lab
> > > > > management tooling and wrote your own.
> > > > >
> > > > > > Labgrid (which you suggested I use for my lab, if you recall),
> > > > >
> > > > > Yes, and I think you forgot the aim was to make it easy to show all of
> > > > > the existing Labgrid based labs that do Linux kernel testing they could
> > > > > easily add U-Boot to the mix. I've been trying to get feedback from
> > > > > other people with existing labgrid setups to look at what you've done.
> > > > >
> > > > > > provides for two yaml configuration files so that everything is in one
> > > > > > place. Apart from its primitive support for USB hubs, it is much
> > > > > > easier to maintain that dozens of little files all over the palce.
> > > > >
> > > > > I mean, I looked at what you posted and strongly disagree, but I think
> > > > > both cases here are personal preference and not some sort of objective
> > > > > and easily evaluated thing.
> > > > >
> > > > > > > > > > We need an 'all of the above' strategy here.
> > > > > > > > >
> > > > > > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > > > > > have above is *extremely* board and OS specific and non-configurable.
> > > > > > > >
> > > > > > > > Yes, agreed.
> > > > > > > >
> > > > > > > > > I
> > > > > > > > > also don't quite see why it's not a test of autoboot with the
> > > > > > > > > pre-requisite of an OS being installed.
> > > > > > > >
> > > > > > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > > > > I hope eventually to have the installer run to completion and then
> > > > > > > > reboot to check all is well.
> > > > > > >
> > > > > > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > > > testing Linaro has going now that automates I believe it was current
> > > > > > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > > > full install via network in CI. So yes, a Canonical lab might also find
> > > > > > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > > > that at least some of the existing kernelci labs see the utility in
> > > > > > > adding "current U-Boot" as one of the matrix variables they test and not
> > > > > > > just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > > >
> > > > > > OK.
> > > > > >
> > > > > > >
> > > > > > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > > > > > sets up the required test files (on a suitable server) before running
> > > > > > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > > > > > don't have to spend ages trying to divine what each test needs.
> > > > > > > > >
> > > > > > > > > That seems like a lot more work than documenting more what we have
> > > > > > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > > > > > another thread.
> > > > > > > >
> > > > > > > > Yes, perhaps, but having self-contained tests would be a win.
> > > > > > >
> > > > > > > With it's own set of technical and legal challenges / obligations and
> > > > > > > difficulties depending on what you even mean by "self contained". And
> > > > > > > how often what's run where, and all sorts of other challenges too.
> > > > > > >
> > > > > > > Given the extreme depth that testing can go to, this is why I'm of the
> > > > > > > position that we need to document things more and worry less about
> > > > > > > prepackaged things. For example, making the documentation for the
> > > > > > > current net based OS boot means that for bringing up a new board the
> > > > > > > developer can just drop something in. Whereas if the tests expect a
> > > > > > > functional OS image that has to also be messed with and is its own
> > > > > > > challenge.
> > > > > >
> > > > > > Yes
> > > > > >
> > > > > > >
> > > > > > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > > > > configuration details, specific to both the platform / SoC first, some
> > > > > > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > > > > > distant third.
> > > > > > > >
> > > > > > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > > > > we have outgrown it. We want people to be able to connect their lab to
> > > > > > > > CI (meaning gitlab), so testing is more automated.
> > > > > > >
> > > > > > > More and more public testing would be great. The notes I linked above
> > > > > > > explain one of the first problems there being that most companies will
> > > > > > > not or can not hook a lab to a public CI instance.
> > > > > >
> > > > > > Well corporate IT is what it is.
> > > > > >
> > > > > > That means that their boards will not be testing in CI, unless they do
> > > > > > it themselves, right?
> > > > >
> > > > > I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > > tree at URL) and something that's been being done since the beginning
> > > > > even for U-Boot (it's how the original nvidia lab worked).
> > > > >
> > > > > > > The next problem, as
> > > > > > > both of our personal labs show, is that just maintaining the physical
> > > > > > > lab takes time and resources. I've added Heiko here because I've been
> > > > > > > talking with him off-list about expanding tbot coverage and plumbing
> > > > > > > that in to gitlab.
> > > > > >
> > > > > > OK
> > > > > >
> > > > > > >
> > > > > > > > We should move away from relying on maintainers getting around to
> > > > > > > > testing patches months after they are sent, when they have time, but
> > > > > > > > they don't. Things need to be more automated and I'd encourage you to
> > > > > > > > push this as well.
> > > > > > >
> > > > > > > I have been, and the results I've gotten are that companies are testing
> > > > > > > things internally but there's not any good way to publish results, and
> > > > > > > that's the kind of framework we're entirely missing.
> > > > > >
> > > > > > If you like, but from my side, I like to see the results in gitlab.
> > > > >
> > > > > Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > > push and visible in the main pipeline". Which isn't possible. It's not
> > > > > going to happen. External collection is how it's handled for the linux
> > > > > kernel and that community has far more sway than we do. If we ride their
> > > > > coattails here so to speak, we can get results. If we push for something
> > > > > completely different we aren't likely to have success.
> > > > >
> > > > > > > Which is another part of why I keep pushing against having U-Boot
> > > > > > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > > > that's not using labgrid to see how to configure things.
> > > > > >
> > > > > > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > > > am happy to publish the config[1], but I still hold my view that all
> > > > > > the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > > > work with.
> > > > >
> > > > > Yes, and as I've shown, you can also use labgrid without going down the
> > > > > same path you took, and we can also support other lab management methods
> > > > > too, which is important to get as much testing as possible without
> > > > > needing to centralize everything.
> > > >
> > > > In summary, I suppose we just have different visions and ideas, which
> > > > should be a good thing.
> > >
> > > So long as the project can speak with one voice, yes. Which all circles
> > > back to why I do not think what you're doing with u-boot.org is at all
> > > helpful.
> >
> > I do need a relief valve for when my efforts are blocked.
>
> And I do not see how using the project domain is appropriate. Stop doing
> this. If you can't stop I'm going to reach my own limit here.

I'm happy to have source.u-boot.org or similar point to Denx, but I do
need my own tree while my work is blocked from submission. I suggest
we discuss it on a call and try to figure out a compromise while we
are in this state.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 24308 bytes --]

On Thu, Feb 27, 2025 at 12:26:10PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > >
> > > > > > > > On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > > > > > > Hi Tom,
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > > Hi Tom,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > > > > > > > > > > > > Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > > > > > > > > > > > > installer starts up correctly.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Use the qemu-x86_64 board in the SJG lab.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Changes in v2:
> > > > > > > > > > > > > > > > > - Add more patches to support booting with kvm
> > > > > > > > > > > > > > > > > - Add new patch with a test for booting Ubuntu 24.04
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >  .gitlab-ci.yml               |  5 ++++
> > > > > > > > > > > > > > > > >  test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > > > > > > > > > > > >  2 files changed, 58 insertions(+)
> > > > > > > > > > > > > > > > >  create mode 100644 test/py/tests/test_distro.py
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > > > > > > > > > > > > index 8c49d5b0a79..ec799e97c10 100644
> > > > > > > > > > > > > > > > > --- a/.gitlab-ci.yml
> > > > > > > > > > > > > > > > > +++ b/.gitlab-ci.yml
> > > > > > > > > > > > > > > > > @@ -745,3 +745,8 @@ zybo:
> > > > > > > > > > > > > > > > >    variables:
> > > > > > > > > > > > > > > > >      ROLE: zybo
> > > > > > > > > > > > > > > > >    <<: *lab_dfn
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +qemu-x86_64:
> > > > > > > > > > > > > > > > > +  variables:
> > > > > > > > > > > > > > > > > +    ROLE: qemu-x86_64
> > > > > > > > > > > > > > > > > +  <<: *lab_dfn
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > > > > > > > > > > > test.py QEMU stanza.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we're going to be able to run it on N platforms, yes, we need to
> > > > > > > > > > > > > > think of a good way to cache the download. There's not a particular
> > > > > > > > > > > > > > reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > > > > > > > > > targets and also qemu-riscv64, is there?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > > > > > > > > doesn't require all the runners to download a much larger image, etc.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > > > > > > > tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > > > > > > > that certain longer or possibly destructive tests are only run on tagged
> > > > > > > > > > > > releases or as requested rather than every time, as it will take longer.
> > > > > > > > > > > > But pretty much every platform under the qemu target list should be able
> > > > > > > > > > > > to Just Boot an off the shelf OS distribution is my point.
> > > > > > > > > > >
> > > > > > > > > > > Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > > > > > diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > > > new file mode 100644
> > > > > > > > > > > > > > > > > index 00000000000..51eec45cecc
> > > > > > > > > > > > > > > > > --- /dev/null
> > > > > > > > > > > > > > > > > +++ b/test/py/tests/test_distro.py
> > > > > > > > > > > > > > > > > @@ -0,0 +1,53 @@
> > > > > > > > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0+
> > > > > > > > > > > > > > > > > +# Copyright 2025 Canonical Ltd.
> > > > > > > > > > > > > > > > > +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +import pytest
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +# Enable early console so that the test can see if something goes wrong
> > > > > > > > > > > > > > > > > +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > > > > > > > > > > > > +@pytest.mark.role('qemu-x86_64')
> > > > > > > > > > > > > > > > > +def test_distro(ubman):
> > > > > > > > > > > > > > > > > +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > > > > > > > > > > > > +    with ubman.log.section('boot'):
> > > > > > > > > > > > > > > > > +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +    with ubman.log.section('Grub'):
> > > > > > > > > > > > > > > > > +        # Wait for grub to come up and offset a menu
> > > > > > > > > > > > > > > > > +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Press 'e' to edit the command line
> > > > > > > > > > > > > > > > > +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Wait until we see the editor appear
> > > > > > > > > > > > > > > > > +        ubman.p.expect(['/casper/initrd'])
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Go down to the 'linux' line
> > > > > > > > > > > > > > > > > +        ubman.send(DOWN * 3)
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Go to end of line
> > > > > > > > > > > > > > > > > +        ubman.ctrl('E')
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Backspace to remove 'quiet splash'
> > > > > > > > > > > > > > > > > +        ubman.send('\b' * len('quiet splash'))
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Send our noisy console
> > > > > > > > > > > > > > > > > +        ubman.send(CONSOLE)
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +        # Tell grub to boot
> > > > > > > > > > > > > > > > > +        ubman.ctrl('X')
> > > > > > > > > > > > > > > > > +        ubman.p.expect(['Booting a command list'])
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +    with ubman.log.section('Linux'):
> > > > > > > > > > > > > > > > > +        # Linux should start immediately
> > > > > > > > > > > > > > > > > +        ubman.p.expect(['Linux version'])
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +    with ubman.log.section('Ubuntu'):
> > > > > > > > > > > > > > > > > +        # Shortly later, we should see this banner
> > > > > > > > > > > > > > > > > +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > > > > > > > > > > > > +
> > > > > > > > > > > > > > > > > +    ubman.restart_uboot()
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > And this seems very inflexible. Please see
> > > > > > > > > > > > > > > > test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > > > > > > > > > > > configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > > > > > > > > > > > that the "role" part here is where you have a special disk image being
> > > > > > > > > > > > > > > > passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > > > > > > > > > > > and the images pre-fetched to the CI container. And if this was
> > > > > > > > > > > > > > > > configurable similar to the example I noted above, it could check real
> > > > > > > > > > > > > > > > hardware too.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > That wasn't the reaction I expected.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > > > > > > > > > > than what we have today?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > > > > > > > > > test that we have today? No, it's not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I didn't even know about it, or perhaps I forgot.
> > > > > > > > > > > >
> > > > > > > > > > > > I believe I mentioned it every time you've said we should have an OS
> > > > > > > > > > > > test, so yes, I guess you forgot.
> > > > > > > > > > >
> > > > > > > > > > > Well it was only added in May last year and it relies on board config
> > > > > > > > > > > which I don't have...although I see that you have now posted yours.
> > > > > > > > > >
> > > > > > > > > > Yes, it was added not quite a year ago, and is documented within the
> > > > > > > > > > test, like most tests that rely on the real platform.
> > > > > > > > > >
> > > > > > > > > > And do we need better documentation for test? Yes.
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > I'll note that I did my bit!
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > Perhaps this relates to getting the labgrid config published and
> > > > > > > > > > > > > figuring out how to pass info from Labgrid to tests.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I would like to generalise this test to work on at least one real
> > > > > > > > > > > > > > > board, preferably one that doesn't use grub.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > OK. The test we have today does that, if you check for the "Welcome to
> > > > > > > > > > > > > > ..." string instead of the kernel has booted string. It also does
> > > > > > > > > > > > > > netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > > > > > > > > > to write up. The only thing stopping me from doing that right now is I
> > > > > > > > > > > > > > need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > > > > > > > > > SD card (some lab sd-mux issues).
> > > > > > > > > > > > >
> > > > > > > > > > > > > OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > > > > > > > > I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > > > > > > > > etc. are available.
> > > > > > > > > > > >
> > > > > > > > > > > > OK, but that sounds like the opposite direction. These are generic tests
> > > > > > > > > > > > that can run in any / all of the labs, not just your labgrid
> > > > > > > > > > > > configuration. AMD has been contributing tests that run on hardware for
> > > > > > > > > > > > example.
> > > > > > > > > > >
> > > > > > > > > > > That's great, the more tests we have the better. But those tests can't
> > > > > > > > > > > and don't run in CI, whereas mine can and do.
> > > > > > > > > >
> > > > > > > > > > AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > > > > > *your* lab because you took things, intentionally, in a direction to
> > > > > > > > > > minimize using u-boot-test-hooks and our existing per-board
> > > > > > > > > > configuration infrastructure.
> > > > > > > > >
> > > > > > > > > When I look at CI all I see is my lab. Which CI are you referring to
> > > > > > > > > and how can I access it?
> > > > > > > >
> > > > > > > > I'll point you at the notes for the first call we had recently:
> > > > > > > > https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > > > > and note that there are many labs doing testing on / with U-Boot.
> > > > > > >
> > > > > > > That's all good, but it isn't as good as having the lab in gitlab.
> > > > > >
> > > > > > Strongly disagree. Especially since having it in the mainline gitlab
> > > > > > isn't feasible.
> > > > > >
> > > > > > > > > Here I would like to make a case for moving to using Labgrid across
> > > > > > > > > the board, but unfortunately the project struggles to review PRs, so
> > > > > > > > > it's probably not a good idea.
> > > > > > > >
> > > > > > > > It would also be counter to the feedback from the U-Boot community about
> > > > > > > > making it easier to contribute testing results from additional labs.
> > > > > > >
> > > > > > > I really don't think the test hooks are a good setup, though. It is
> > > > > > > OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > > > > (Labman) to deal with all the confusion.
> > > > > >
> > > > > > Yes, I don't know how hard you evaluated all of the then-current lab
> > > > > > management tooling and wrote your own.
> > > > > >
> > > > > > > Labgrid (which you suggested I use for my lab, if you recall),
> > > > > >
> > > > > > Yes, and I think you forgot the aim was to make it easy to show all of
> > > > > > the existing Labgrid based labs that do Linux kernel testing they could
> > > > > > easily add U-Boot to the mix. I've been trying to get feedback from
> > > > > > other people with existing labgrid setups to look at what you've done.
> > > > > >
> > > > > > > provides for two yaml configuration files so that everything is in one
> > > > > > > place. Apart from its primitive support for USB hubs, it is much
> > > > > > > easier to maintain that dozens of little files all over the palce.
> > > > > >
> > > > > > I mean, I looked at what you posted and strongly disagree, but I think
> > > > > > both cases here are personal preference and not some sort of objective
> > > > > > and easily evaluated thing.
> > > > > >
> > > > > > > > > > > We need an 'all of the above' strategy here.
> > > > > > > > > >
> > > > > > > > > > Sure. But I still want to see things as reusable as possible. What you
> > > > > > > > > > have above is *extremely* board and OS specific and non-configurable.
> > > > > > > > >
> > > > > > > > > Yes, agreed.
> > > > > > > > >
> > > > > > > > > > I
> > > > > > > > > > also don't quite see why it's not a test of autoboot with the
> > > > > > > > > > pre-requisite of an OS being installed.
> > > > > > > > >
> > > > > > > > > Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > > > > > I hope eventually to have the installer run to completion and then
> > > > > > > > > reboot to check all is well.
> > > > > > > >
> > > > > > > > In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > > > > testing Linaro has going now that automates I believe it was current
> > > > > > > > Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > > > > full install via network in CI. So yes, a Canonical lab might also find
> > > > > > > > it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > > > > that at least some of the existing kernelci labs see the utility in
> > > > > > > > adding "current U-Boot" as one of the matrix variables they test and not
> > > > > > > > just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > > > >
> > > > > > > OK.
> > > > > > >
> > > > > > > >
> > > > > > > > > > > BTW, having thought about how test/py works a bit, instead of the
> > > > > > > > > > > env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > > > > > > sets up the required test files (on a suitable server) before running
> > > > > > > > > > > the test. That way, all the test code is in one Python file and we
> > > > > > > > > > > don't have to spend ages trying to divine what each test needs.
> > > > > > > > > >
> > > > > > > > > > That seems like a lot more work than documenting more what we have
> > > > > > > > > > today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > > > > > test, yes, just having those files available to 'cp' in place would be
> > > > > > > > > > helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > > > > > stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > > > > > config I posted it also includes bootstage configuration. It also won't
> > > > > > > > > > work well for the SPI tests, which I'm talking with Love about in
> > > > > > > > > > another thread.
> > > > > > > > >
> > > > > > > > > Yes, perhaps, but having self-contained tests would be a win.
> > > > > > > >
> > > > > > > > With it's own set of technical and legal challenges / obligations and
> > > > > > > > difficulties depending on what you even mean by "self contained". And
> > > > > > > > how often what's run where, and all sorts of other challenges too.
> > > > > > > >
> > > > > > > > Given the extreme depth that testing can go to, this is why I'm of the
> > > > > > > > position that we need to document things more and worry less about
> > > > > > > > prepackaged things. For example, making the documentation for the
> > > > > > > > current net based OS boot means that for bringing up a new board the
> > > > > > > > developer can just drop something in. Whereas if the tests expect a
> > > > > > > > functional OS image that has to also be messed with and is its own
> > > > > > > > challenge.
> > > > > > >
> > > > > > > Yes
> > > > > > >
> > > > > > > >
> > > > > > > > > > In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > > > > > configuration details, specific to both the platform / SoC first, some
> > > > > > > > > > lab specific details second and drop-in existing 3rd party files a
> > > > > > > > > > distant third.
> > > > > > > > >
> > > > > > > > > I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > > > > > we have outgrown it. We want people to be able to connect their lab to
> > > > > > > > > CI (meaning gitlab), so testing is more automated.
> > > > > > > >
> > > > > > > > More and more public testing would be great. The notes I linked above
> > > > > > > > explain one of the first problems there being that most companies will
> > > > > > > > not or can not hook a lab to a public CI instance.
> > > > > > >
> > > > > > > Well corporate IT is what it is.
> > > > > > >
> > > > > > > That means that their boards will not be testing in CI, unless they do
> > > > > > > it themselves, right?
> > > > > >
> > > > > > I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > > > tree at URL) and something that's been being done since the beginning
> > > > > > even for U-Boot (it's how the original nvidia lab worked).
> > > > > >
> > > > > > > > The next problem, as
> > > > > > > > both of our personal labs show, is that just maintaining the physical
> > > > > > > > lab takes time and resources. I've added Heiko here because I've been
> > > > > > > > talking with him off-list about expanding tbot coverage and plumbing
> > > > > > > > that in to gitlab.
> > > > > > >
> > > > > > > OK
> > > > > > >
> > > > > > > >
> > > > > > > > > We should move away from relying on maintainers getting around to
> > > > > > > > > testing patches months after they are sent, when they have time, but
> > > > > > > > > they don't. Things need to be more automated and I'd encourage you to
> > > > > > > > > push this as well.
> > > > > > > >
> > > > > > > > I have been, and the results I've gotten are that companies are testing
> > > > > > > > things internally but there's not any good way to publish results, and
> > > > > > > > that's the kind of framework we're entirely missing.
> > > > > > >
> > > > > > > If you like, but from my side, I like to see the results in gitlab.
> > > > > >
> > > > > > Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > > > push and visible in the main pipeline". Which isn't possible. It's not
> > > > > > going to happen. External collection is how it's handled for the linux
> > > > > > kernel and that community has far more sway than we do. If we ride their
> > > > > > coattails here so to speak, we can get results. If we push for something
> > > > > > completely different we aren't likely to have success.
> > > > > >
> > > > > > > > Which is another part of why I keep pushing against having U-Boot
> > > > > > > > configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > > > > that's not using labgrid to see how to configure things.
> > > > > > >
> > > > > > > Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > > > > am happy to publish the config[1], but I still hold my view that all
> > > > > > > the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > > > > work with.
> > > > > >
> > > > > > Yes, and as I've shown, you can also use labgrid without going down the
> > > > > > same path you took, and we can also support other lab management methods
> > > > > > too, which is important to get as much testing as possible without
> > > > > > needing to centralize everything.
> > > > >
> > > > > In summary, I suppose we just have different visions and ideas, which
> > > > > should be a good thing.
> > > >
> > > > So long as the project can speak with one voice, yes. Which all circles
> > > > back to why I do not think what you're doing with u-boot.org is at all
> > > > helpful.
> > >
> > > I do need a relief valve for when my efforts are blocked.
> >
> > And I do not see how using the project domain is appropriate. Stop doing
> > this. If you can't stop I'm going to reach my own limit here.
> 
> I'm happy to have source.u-boot.org or similar point to Denx, but I do
> need my own tree while my work is blocked from submission. I suggest
> we discuss it on a call and try to figure out a compromise while we
> are in this state.

If you'll remove ci.u-boot.org and just have sjg.u-boot.org I'll stop
feeling like you're holding the domain hostage over your rejected
changes, and we can talk.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Heiko Schocher]

Hi Simon,

On 27.02.25 20:26, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
>>
>> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
>>> Hi Tom,
>>>
>>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
>>>>
>>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
>>>>> Hi Tom,
>>>>>
>>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
>>>>>>
>>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
>>>>>>> Hi Tom,
>>>>>>>
>>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>
>>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
>>>>>>>>> Hi Tom,
>>>>>>>>>
>>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>
>>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
>>>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
>>>>>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
>>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
>>>>>>>>>>>>>>>>> installer starts up correctly.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Changes in v2:
>>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
>>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
>>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
>>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
>>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
>>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
>>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
>>>>>>>>>>>>>>>>>     variables:
>>>>>>>>>>>>>>>>>       ROLE: zybo
>>>>>>>>>>>>>>>>>     <<: *lab_dfn
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +qemu-x86_64:
>>>>>>>>>>>>>>>>> +  variables:
>>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
>>>>>>>>>>>>>>>>> +  <<: *lab_dfn
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
>>>>>>>>>>>>>>>> test.py QEMU stanza.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
>>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
>>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
>>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
>>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
>>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
>>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
>>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
>>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
>>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
>>>>>>>>>>>
>>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
>>>>>>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
>>>>>>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
>>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
>>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
>>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
>>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +import pytest
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
>>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
>>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
>>>>>>>>>>>>>>>>> +def test_distro(ubman):
>>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
>>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
>>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
>>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
>>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
>>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
>>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
>>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Go to end of line
>>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
>>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Send our noisy console
>>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +        # Tell grub to boot
>>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
>>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
>>>>>>>>>>>>>>>>> +        # Linux should start immediately
>>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
>>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
>>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And this seems very inflexible. Please see
>>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
>>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
>>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
>>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
>>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
>>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
>>>>>>>>>>>>>>>> hardware too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That wasn't the reaction I expected.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
>>>>>>>>>>>>>>> than what we have today?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
>>>>>>>>>>>>>> test that we have today? No, it's not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
>>>>>>>>>>>>
>>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
>>>>>>>>>>>> test, so yes, I guess you forgot.
>>>>>>>>>>>
>>>>>>>>>>> Well it was only added in May last year and it relies on board config
>>>>>>>>>>> which I don't have...although I see that you have now posted yours.
>>>>>>>>>>
>>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
>>>>>>>>>> test, like most tests that rely on the real platform.
>>>>>>>>>>
>>>>>>>>>> And do we need better documentation for test? Yes.
>>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> I'll note that I did my bit!
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
>>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
>>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
>>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
>>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
>>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
>>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
>>>>>>>>>>>>>> SD card (some lab sd-mux issues).
>>>>>>>>>>>>>
>>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
>>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
>>>>>>>>>>>>> etc. are available.
>>>>>>>>>>>>
>>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
>>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
>>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
>>>>>>>>>>>> example.
>>>>>>>>>>>
>>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
>>>>>>>>>>> and don't run in CI, whereas mine can and do.
>>>>>>>>>>
>>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
>>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
>>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
>>>>>>>>>> configuration infrastructure.
>>>>>>>>>
>>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
>>>>>>>>> and how can I access it?
>>>>>>>>
>>>>>>>> I'll point you at the notes for the first call we had recently:
>>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
>>>>>>>> and note that there are many labs doing testing on / with U-Boot.
>>>>>>>
>>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
>>>>>>
>>>>>> Strongly disagree. Especially since having it in the mainline gitlab
>>>>>> isn't feasible.
>>>>>>
>>>>>>>>> Here I would like to make a case for moving to using Labgrid across
>>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
>>>>>>>>> it's probably not a good idea.
>>>>>>>>
>>>>>>>> It would also be counter to the feedback from the U-Boot community about
>>>>>>>> making it easier to contribute testing results from additional labs.
>>>>>>>
>>>>>>> I really don't think the test hooks are a good setup, though. It is
>>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
>>>>>>> (Labman) to deal with all the confusion.
>>>>>>
>>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
>>>>>> management tooling and wrote your own.
>>>>>>
>>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
>>>>>>
>>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
>>>>>> the existing Labgrid based labs that do Linux kernel testing they could
>>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
>>>>>> other people with existing labgrid setups to look at what you've done.
>>>>>>
>>>>>>> provides for two yaml configuration files so that everything is in one
>>>>>>> place. Apart from its primitive support for USB hubs, it is much
>>>>>>> easier to maintain that dozens of little files all over the palce.
>>>>>>
>>>>>> I mean, I looked at what you posted and strongly disagree, but I think
>>>>>> both cases here are personal preference and not some sort of objective
>>>>>> and easily evaluated thing.
>>>>>>
>>>>>>>>>>> We need an 'all of the above' strategy here.
>>>>>>>>>>
>>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
>>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
>>>>>>>>>
>>>>>>>>> Yes, agreed.
>>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>> also don't quite see why it's not a test of autoboot with the
>>>>>>>>>> pre-requisite of an OS being installed.
>>>>>>>>>
>>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
>>>>>>>>> I hope eventually to have the installer run to completion and then
>>>>>>>>> reboot to check all is well.
>>>>>>>>
>>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
>>>>>>>> testing Linaro has going now that automates I believe it was current
>>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
>>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
>>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
>>>>>>>> that at least some of the existing kernelci labs see the utility in
>>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
>>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
>>>>>>>
>>>>>>> OK.
>>>>>>>
>>>>>>>>
>>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
>>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
>>>>>>>>>>> sets up the required test files (on a suitable server) before running
>>>>>>>>>>> the test. That way, all the test code is in one Python file and we
>>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
>>>>>>>>>>
>>>>>>>>>> That seems like a lot more work than documenting more what we have
>>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
>>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
>>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
>>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
>>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
>>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
>>>>>>>>>> another thread.
>>>>>>>>>
>>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
>>>>>>>>
>>>>>>>> With it's own set of technical and legal challenges / obligations and
>>>>>>>> difficulties depending on what you even mean by "self contained". And
>>>>>>>> how often what's run where, and all sorts of other challenges too.
>>>>>>>>
>>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
>>>>>>>> position that we need to document things more and worry less about
>>>>>>>> prepackaged things. For example, making the documentation for the
>>>>>>>> current net based OS boot means that for bringing up a new board the
>>>>>>>> developer can just drop something in. Whereas if the tests expect a
>>>>>>>> functional OS image that has to also be messed with and is its own
>>>>>>>> challenge.
>>>>>>>
>>>>>>> Yes
>>>>>>>
>>>>>>>>
>>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
>>>>>>>>>> configuration details, specific to both the platform / SoC first, some
>>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
>>>>>>>>>> distant third.
>>>>>>>>>
>>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
>>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
>>>>>>>>> CI (meaning gitlab), so testing is more automated.
>>>>>>>>
>>>>>>>> More and more public testing would be great. The notes I linked above
>>>>>>>> explain one of the first problems there being that most companies will
>>>>>>>> not or can not hook a lab to a public CI instance.
>>>>>>>
>>>>>>> Well corporate IT is what it is.
>>>>>>>
>>>>>>> That means that their boards will not be testing in CI, unless they do
>>>>>>> it themselves, right?
>>>>>>
>>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
>>>>>> tree at URL) and something that's been being done since the beginning
>>>>>> even for U-Boot (it's how the original nvidia lab worked).
>>>>>>
>>>>>>>> The next problem, as
>>>>>>>> both of our personal labs show, is that just maintaining the physical
>>>>>>>> lab takes time and resources. I've added Heiko here because I've been
>>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
>>>>>>>> that in to gitlab.
>>>>>>>
>>>>>>> OK
>>>>>>>
>>>>>>>>
>>>>>>>>> We should move away from relying on maintainers getting around to
>>>>>>>>> testing patches months after they are sent, when they have time, but
>>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
>>>>>>>>> push this as well.
>>>>>>>>
>>>>>>>> I have been, and the results I've gotten are that companies are testing
>>>>>>>> things internally but there's not any good way to publish results, and
>>>>>>>> that's the kind of framework we're entirely missing.
>>>>>>>
>>>>>>> If you like, but from my side, I like to see the results in gitlab.
>>>>>>
>>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
>>>>>> push and visible in the main pipeline". Which isn't possible. It's not
>>>>>> going to happen. External collection is how it's handled for the linux
>>>>>> kernel and that community has far more sway than we do. If we ride their
>>>>>> coattails here so to speak, we can get results. If we push for something
>>>>>> completely different we aren't likely to have success.
>>>>>>
>>>>>>>> Which is another part of why I keep pushing against having U-Boot
>>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
>>>>>>>> that's not using labgrid to see how to configure things.
>>>>>>>
>>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
>>>>>>> am happy to publish the config[1], but I still hold my view that all
>>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
>>>>>>> work with.
>>>>>>
>>>>>> Yes, and as I've shown, you can also use labgrid without going down the
>>>>>> same path you took, and we can also support other lab management methods
>>>>>> too, which is important to get as much testing as possible without
>>>>>> needing to centralize everything.
>>>>>
>>>>> In summary, I suppose we just have different visions and ideas, which
>>>>> should be a good thing.
>>>>
>>>> So long as the project can speak with one voice, yes. Which all circles
>>>> back to why I do not think what you're doing with u-boot.org is at all
>>>> helpful.
>>>
>>> I do need a relief valve for when my efforts are blocked.
>>
>> And I do not see how using the project domain is appropriate. Stop doing
>> this. If you can't stop I'm going to reach my own limit here.
> 
> I'm happy to have source.u-boot.org or similar point to Denx, but I do
> need my own tree while my work is blocked from submission. I suggest
> we discuss it on a call and try to figure out a compromise while we
> are in this state.

Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?

or u-boot-ci ?

I currently also work on a CI integration of my testsetup, and made a
simple branch in u-boot-i2c for doing this tests...

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs@denx.de

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Heiko,

On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
>
> Hi Simon,
>
> On 27.02.25 20:26, Simon Glass wrote:
> > Hi Tom,
> >
> > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> >>
> >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> >>> Hi Tom,
> >>>
> >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> >>>>
> >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> >>>>> Hi Tom,
> >>>>>
> >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>
> >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> >>>>>>> Hi Tom,
> >>>>>>>
> >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>>>
> >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> >>>>>>>>> Hi Tom,
> >>>>>>>>>
> >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> >>>>>>>>>>> Hi Tom,
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> >>>>>>>>>>>>> Hi Tom,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> >>>>>>>>>>>>>>> Hi Tom,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> >>>>>>>>>>>>>>>>> installer starts up correctly.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> >>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Changes in v2:
> >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> >>>>>>>>>>>>>>>>>     variables:
> >>>>>>>>>>>>>>>>>       ROLE: zybo
> >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +qemu-x86_64:
> >>>>>>>>>>>>>>>>> +  variables:
> >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> >>>>>>>>>>>>>>>> test.py QEMU stanza.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> >>>>>>>>>>>
> >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> >>>>>>>>>>>>>>>>> new file mode 100644
> >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> >>>>>>>>>>>>>>>>> --- /dev/null
> >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +import pytest
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Go to end of line
> >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Send our noisy console
> >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> >>>>>>>>>>>>>>>>> +
> >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> >>>>>>>>>>>>>>>> hardware too.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> >>>>>>>>>>>>>>> than what we have today?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> >>>>>>>>>>>>>> test that we have today? No, it's not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> >>>>>>>>>>>> test, so yes, I guess you forgot.
> >>>>>>>>>>>
> >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> >>>>>>>>>>
> >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> >>>>>>>>>> test, like most tests that rely on the real platform.
> >>>>>>>>>>
> >>>>>>>>>> And do we need better documentation for test? Yes.
> >>>>>>>>>
> >>>>>>>>> +1
> >>>>>>>>>
> >>>>>>>>> I'll note that I did my bit!
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> >>>>>>>>>>>>> etc. are available.
> >>>>>>>>>>>>
> >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> >>>>>>>>>>>> example.
> >>>>>>>>>>>
> >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> >>>>>>>>>>
> >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> >>>>>>>>>> configuration infrastructure.
> >>>>>>>>>
> >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> >>>>>>>>> and how can I access it?
> >>>>>>>>
> >>>>>>>> I'll point you at the notes for the first call we had recently:
> >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> >>>>>>>
> >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> >>>>>>
> >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> >>>>>> isn't feasible.
> >>>>>>
> >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> >>>>>>>>> it's probably not a good idea.
> >>>>>>>>
> >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> >>>>>>>> making it easier to contribute testing results from additional labs.
> >>>>>>>
> >>>>>>> I really don't think the test hooks are a good setup, though. It is
> >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> >>>>>>> (Labman) to deal with all the confusion.
> >>>>>>
> >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> >>>>>> management tooling and wrote your own.
> >>>>>>
> >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> >>>>>>
> >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> >>>>>> other people with existing labgrid setups to look at what you've done.
> >>>>>>
> >>>>>>> provides for two yaml configuration files so that everything is in one
> >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> >>>>>>> easier to maintain that dozens of little files all over the palce.
> >>>>>>
> >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> >>>>>> both cases here are personal preference and not some sort of objective
> >>>>>> and easily evaluated thing.
> >>>>>>
> >>>>>>>>>>> We need an 'all of the above' strategy here.
> >>>>>>>>>>
> >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> >>>>>>>>>
> >>>>>>>>> Yes, agreed.
> >>>>>>>>>
> >>>>>>>>>> I
> >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> >>>>>>>>>> pre-requisite of an OS being installed.
> >>>>>>>>>
> >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> >>>>>>>>> I hope eventually to have the installer run to completion and then
> >>>>>>>>> reboot to check all is well.
> >>>>>>>>
> >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> >>>>>>>> testing Linaro has going now that automates I believe it was current
> >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> >>>>>>>> that at least some of the existing kernelci labs see the utility in
> >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> >>>>>>>
> >>>>>>> OK.
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> >>>>>>>>>>
> >>>>>>>>>> That seems like a lot more work than documenting more what we have
> >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> >>>>>>>>>> another thread.
> >>>>>>>>>
> >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> >>>>>>>>
> >>>>>>>> With it's own set of technical and legal challenges / obligations and
> >>>>>>>> difficulties depending on what you even mean by "self contained". And
> >>>>>>>> how often what's run where, and all sorts of other challenges too.
> >>>>>>>>
> >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> >>>>>>>> position that we need to document things more and worry less about
> >>>>>>>> prepackaged things. For example, making the documentation for the
> >>>>>>>> current net based OS boot means that for bringing up a new board the
> >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> >>>>>>>> functional OS image that has to also be messed with and is its own
> >>>>>>>> challenge.
> >>>>>>>
> >>>>>>> Yes
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> >>>>>>>>>> distant third.
> >>>>>>>>>
> >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> >>>>>>>>
> >>>>>>>> More and more public testing would be great. The notes I linked above
> >>>>>>>> explain one of the first problems there being that most companies will
> >>>>>>>> not or can not hook a lab to a public CI instance.
> >>>>>>>
> >>>>>>> Well corporate IT is what it is.
> >>>>>>>
> >>>>>>> That means that their boards will not be testing in CI, unless they do
> >>>>>>> it themselves, right?
> >>>>>>
> >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> >>>>>> tree at URL) and something that's been being done since the beginning
> >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> >>>>>>
> >>>>>>>> The next problem, as
> >>>>>>>> both of our personal labs show, is that just maintaining the physical
> >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> >>>>>>>> that in to gitlab.
> >>>>>>>
> >>>>>>> OK
> >>>>>>>
> >>>>>>>>
> >>>>>>>>> We should move away from relying on maintainers getting around to
> >>>>>>>>> testing patches months after they are sent, when they have time, but
> >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> >>>>>>>>> push this as well.
> >>>>>>>>
> >>>>>>>> I have been, and the results I've gotten are that companies are testing
> >>>>>>>> things internally but there's not any good way to publish results, and
> >>>>>>>> that's the kind of framework we're entirely missing.
> >>>>>>>
> >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> >>>>>>
> >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> >>>>>> going to happen. External collection is how it's handled for the linux
> >>>>>> kernel and that community has far more sway than we do. If we ride their
> >>>>>> coattails here so to speak, we can get results. If we push for something
> >>>>>> completely different we aren't likely to have success.
> >>>>>>
> >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> >>>>>>>> that's not using labgrid to see how to configure things.
> >>>>>>>
> >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> >>>>>>> am happy to publish the config[1], but I still hold my view that all
> >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> >>>>>>> work with.
> >>>>>>
> >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> >>>>>> same path you took, and we can also support other lab management methods
> >>>>>> too, which is important to get as much testing as possible without
> >>>>>> needing to centralize everything.
> >>>>>
> >>>>> In summary, I suppose we just have different visions and ideas, which
> >>>>> should be a good thing.
> >>>>
> >>>> So long as the project can speak with one voice, yes. Which all circles
> >>>> back to why I do not think what you're doing with u-boot.org is at all
> >>>> helpful.
> >>>
> >>> I do need a relief valve for when my efforts are blocked.
> >>
> >> And I do not see how using the project domain is appropriate. Stop doing
> >> this. If you can't stop I'm going to reach my own limit here.
> >
> > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > need my own tree while my work is blocked from submission. I suggest
> > we discuss it on a call and try to figure out a compromise while we
> > are in this state.
>
> Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
>
> or u-boot-ci ?

I don't have permission to do that. With my tree I am able to have CI
running in half the time, use my full lab (not just the subset Tom's
tree has), push patches and ideas that others have blocked, point
people to my work, etc. It is a much better environment for me to
continue (what I see as) necessary innovation.

That said, if you wish to add a top-level project for me, then I would
push things to it, yes.

>
> I currently also work on a CI integration of my testsetup, and made a
> simple branch in u-boot-i2c for doing this tests...

Yes, that's what I had been doing.

>
> bye,
> Heiko
> --
> DENX Software Engineering GmbH,      Managing Director: Erika Unter
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: hs@denx.de

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 22722 bytes --]

On Thu, Mar 06, 2025 at 07:16:08AM -0700, Simon Glass wrote:
> Hi Heiko,
> 
> On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
> >
> > Hi Simon,
> >
> > On 27.02.25 20:26, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> > >>
> > >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > >>> Hi Tom,
> > >>>
> > >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > >>>>
> > >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > >>>>> Hi Tom,
> > >>>>>
> > >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>
> > >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > >>>>>>> Hi Tom,
> > >>>>>>>
> > >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>>>
> > >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > >>>>>>>>> Hi Tom,
> > >>>>>>>>>
> > >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > >>>>>>>>>>> Hi Tom,
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > >>>>>>>>>>>>> Hi Tom,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > >>>>>>>>>>>>>>> Hi Tom,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> > >>>>>>>>>>>>>>>>> installer starts up correctly.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> > >>>>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Changes in v2:
> > >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> > >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> > >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> > >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> > >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> > >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> > >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> > >>>>>>>>>>>>>>>>>     variables:
> > >>>>>>>>>>>>>>>>>       ROLE: zybo
> > >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +qemu-x86_64:
> > >>>>>>>>>>>>>>>>> +  variables:
> > >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> > >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> > >>>>>>>>>>>>>>>> test.py QEMU stanza.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> > >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> > >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> > >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> > >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> > >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> > >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> > >>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > >>>>>>>>>>>>>>>>> new file mode 100644
> > >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> > >>>>>>>>>>>>>>>>> --- /dev/null
> > >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> > >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> > >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> > >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> > >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +import pytest
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> > >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> > >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> > >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> > >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> > >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> > >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> > >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> > >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> > >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Go to end of line
> > >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> > >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Send our noisy console
> > >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> > >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> > >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> > >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > >>>>>>>>>>>>>>>>> +
> > >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> > >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> > >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> > >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> > >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> > >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> > >>>>>>>>>>>>>>>> hardware too.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> > >>>>>>>>>>>>>>> than what we have today?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> > >>>>>>>>>>>>>> test that we have today? No, it's not.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> > >>>>>>>>>>>> test, so yes, I guess you forgot.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> > >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> > >>>>>>>>>>
> > >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> > >>>>>>>>>> test, like most tests that rely on the real platform.
> > >>>>>>>>>>
> > >>>>>>>>>> And do we need better documentation for test? Yes.
> > >>>>>>>>>
> > >>>>>>>>> +1
> > >>>>>>>>>
> > >>>>>>>>> I'll note that I did my bit!
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> > >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> > >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> > >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> > >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> > >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> > >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> > >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> > >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > >>>>>>>>>>>>> etc. are available.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> > >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> > >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> > >>>>>>>>>>>> example.
> > >>>>>>>>>>>
> > >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> > >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> > >>>>>>>>>>
> > >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> > >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> > >>>>>>>>>> configuration infrastructure.
> > >>>>>>>>>
> > >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> > >>>>>>>>> and how can I access it?
> > >>>>>>>>
> > >>>>>>>> I'll point you at the notes for the first call we had recently:
> > >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> > >>>>>>>
> > >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> > >>>>>>
> > >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> > >>>>>> isn't feasible.
> > >>>>>>
> > >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> > >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> > >>>>>>>>> it's probably not a good idea.
> > >>>>>>>>
> > >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> > >>>>>>>> making it easier to contribute testing results from additional labs.
> > >>>>>>>
> > >>>>>>> I really don't think the test hooks are a good setup, though. It is
> > >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > >>>>>>> (Labman) to deal with all the confusion.
> > >>>>>>
> > >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> > >>>>>> management tooling and wrote your own.
> > >>>>>>
> > >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> > >>>>>>
> > >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> > >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> > >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> > >>>>>> other people with existing labgrid setups to look at what you've done.
> > >>>>>>
> > >>>>>>> provides for two yaml configuration files so that everything is in one
> > >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> > >>>>>>> easier to maintain that dozens of little files all over the palce.
> > >>>>>>
> > >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> > >>>>>> both cases here are personal preference and not some sort of objective
> > >>>>>> and easily evaluated thing.
> > >>>>>>
> > >>>>>>>>>>> We need an 'all of the above' strategy here.
> > >>>>>>>>>>
> > >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> > >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> > >>>>>>>>>
> > >>>>>>>>> Yes, agreed.
> > >>>>>>>>>
> > >>>>>>>>>> I
> > >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> > >>>>>>>>>> pre-requisite of an OS being installed.
> > >>>>>>>>>
> > >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> > >>>>>>>>> I hope eventually to have the installer run to completion and then
> > >>>>>>>>> reboot to check all is well.
> > >>>>>>>>
> > >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > >>>>>>>> testing Linaro has going now that automates I believe it was current
> > >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> > >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> > >>>>>>>> that at least some of the existing kernelci labs see the utility in
> > >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> > >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> > >>>>>>>
> > >>>>>>> OK.
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> > >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> > >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> > >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> > >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> > >>>>>>>>>>
> > >>>>>>>>>> That seems like a lot more work than documenting more what we have
> > >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> > >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> > >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> > >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> > >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> > >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> > >>>>>>>>>> another thread.
> > >>>>>>>>>
> > >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> > >>>>>>>>
> > >>>>>>>> With it's own set of technical and legal challenges / obligations and
> > >>>>>>>> difficulties depending on what you even mean by "self contained". And
> > >>>>>>>> how often what's run where, and all sorts of other challenges too.
> > >>>>>>>>
> > >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> > >>>>>>>> position that we need to document things more and worry less about
> > >>>>>>>> prepackaged things. For example, making the documentation for the
> > >>>>>>>> current net based OS boot means that for bringing up a new board the
> > >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> > >>>>>>>> functional OS image that has to also be messed with and is its own
> > >>>>>>>> challenge.
> > >>>>>>>
> > >>>>>>> Yes
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> > >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> > >>>>>>>>>> distant third.
> > >>>>>>>>>
> > >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> > >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> > >>>>>>>>
> > >>>>>>>> More and more public testing would be great. The notes I linked above
> > >>>>>>>> explain one of the first problems there being that most companies will
> > >>>>>>>> not or can not hook a lab to a public CI instance.
> > >>>>>>>
> > >>>>>>> Well corporate IT is what it is.
> > >>>>>>>
> > >>>>>>> That means that their boards will not be testing in CI, unless they do
> > >>>>>>> it themselves, right?
> > >>>>>>
> > >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> > >>>>>> tree at URL) and something that's been being done since the beginning
> > >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> > >>>>>>
> > >>>>>>>> The next problem, as
> > >>>>>>>> both of our personal labs show, is that just maintaining the physical
> > >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> > >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> > >>>>>>>> that in to gitlab.
> > >>>>>>>
> > >>>>>>> OK
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>>> We should move away from relying on maintainers getting around to
> > >>>>>>>>> testing patches months after they are sent, when they have time, but
> > >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> > >>>>>>>>> push this as well.
> > >>>>>>>>
> > >>>>>>>> I have been, and the results I've gotten are that companies are testing
> > >>>>>>>> things internally but there's not any good way to publish results, and
> > >>>>>>>> that's the kind of framework we're entirely missing.
> > >>>>>>>
> > >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> > >>>>>>
> > >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> > >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> > >>>>>> going to happen. External collection is how it's handled for the linux
> > >>>>>> kernel and that community has far more sway than we do. If we ride their
> > >>>>>> coattails here so to speak, we can get results. If we push for something
> > >>>>>> completely different we aren't likely to have success.
> > >>>>>>
> > >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> > >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> > >>>>>>>> that's not using labgrid to see how to configure things.
> > >>>>>>>
> > >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > >>>>>>> am happy to publish the config[1], but I still hold my view that all
> > >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> > >>>>>>> work with.
> > >>>>>>
> > >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> > >>>>>> same path you took, and we can also support other lab management methods
> > >>>>>> too, which is important to get as much testing as possible without
> > >>>>>> needing to centralize everything.
> > >>>>>
> > >>>>> In summary, I suppose we just have different visions and ideas, which
> > >>>>> should be a good thing.
> > >>>>
> > >>>> So long as the project can speak with one voice, yes. Which all circles
> > >>>> back to why I do not think what you're doing with u-boot.org is at all
> > >>>> helpful.
> > >>>
> > >>> I do need a relief valve for when my efforts are blocked.
> > >>
> > >> And I do not see how using the project domain is appropriate. Stop doing
> > >> this. If you can't stop I'm going to reach my own limit here.
> > >
> > > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > > need my own tree while my work is blocked from submission. I suggest
> > > we discuss it on a call and try to figure out a compromise while we
> > > are in this state.
> >
> > Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
> >
> > or u-boot-ci ?
> 
> I don't have permission to do that. With my tree I am able to have CI
> running in half the time, use my full lab (not just the subset Tom's
> tree has), push patches and ideas that others have blocked, point
> people to my work, etc. It is a much better environment for me to
> continue (what I see as) necessary innovation.
> 
> That said, if you wish to add a top-level project for me, then I would
> push things to it, yes.

To be clear, you can have
https://source.denx.de/u-boot/contributors/sjg/ but you can't host your
downstream fork on source.denx.de itself and I wish you wouldn't abuse
the project domain for it either.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Thu, 6 Mar 2025 at 07:32, Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Mar 06, 2025 at 07:16:08AM -0700, Simon Glass wrote:
> > Hi Heiko,
> >
> > On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
> > >
> > > Hi Simon,
> > >
> > > On 27.02.25 20:26, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> > > >>
> > > >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > > >>> Hi Tom,
> > > >>>
> > > >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>
> > > >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > >>>>> Hi Tom,
> > > >>>>>
> > > >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>
> > > >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > >>>>>>> Hi Tom,
> > > >>>>>>>
> > > >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > >>>>>>>>> Hi Tom,
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > >>>>>>>>>>> Hi Tom,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > >>>>>>>>>>>>> Hi Tom,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > >>>>>>>>>>>>>>> Hi Tom,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > >>>>>>>>>>>>>>>>> installer starts up correctly.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> > > >>>>>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Changes in v2:
> > > >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> > > >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> > > >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> > > >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> > > >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> > > >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> > > >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> > > >>>>>>>>>>>>>>>>>     variables:
> > > >>>>>>>>>>>>>>>>>       ROLE: zybo
> > > >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +qemu-x86_64:
> > > >>>>>>>>>>>>>>>>> +  variables:
> > > >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> > > >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> > > >>>>>>>>>>>>>>>> test.py QEMU stanza.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> > > >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> > > >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> > > >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> > > >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> > > >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> > > >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > >>>>>>>>>>>>>>>>> new file mode 100644
> > > >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> > > >>>>>>>>>>>>>>>>> --- /dev/null
> > > >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> > > >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> > > >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> > > >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> > > >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +import pytest
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> > > >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> > > >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> > > >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> > > >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> > > >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> > > >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> > > >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> > > >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Go to end of line
> > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> > > >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Send our noisy console
> > > >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> > > >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> > > >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > >>>>>>>>>>>>>>>>> +
> > > >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> > > >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> > > >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> > > >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> > > >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> > > >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> > > >>>>>>>>>>>>>>>> hardware too.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> > > >>>>>>>>>>>>>>> than what we have today?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> > > >>>>>>>>>>>>>> test that we have today? No, it's not.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> > > >>>>>>>>>>>> test, so yes, I guess you forgot.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> > > >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> > > >>>>>>>>>> test, like most tests that rely on the real platform.
> > > >>>>>>>>>>
> > > >>>>>>>>>> And do we need better documentation for test? Yes.
> > > >>>>>>>>>
> > > >>>>>>>>> +1
> > > >>>>>>>>>
> > > >>>>>>>>> I'll note that I did my bit!
> > > >>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> > > >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> > > >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> > > >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> > > >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> > > >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> > > >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> > > >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > >>>>>>>>>>>>> etc. are available.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> > > >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> > > >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> > > >>>>>>>>>>>> example.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> > > >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> > > >>>>>>>>>>
> > > >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> > > >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> > > >>>>>>>>>> configuration infrastructure.
> > > >>>>>>>>>
> > > >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> > > >>>>>>>>> and how can I access it?
> > > >>>>>>>>
> > > >>>>>>>> I'll point you at the notes for the first call we had recently:
> > > >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> > > >>>>>>>
> > > >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> > > >>>>>>
> > > >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> > > >>>>>> isn't feasible.
> > > >>>>>>
> > > >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> > > >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> > > >>>>>>>>> it's probably not a good idea.
> > > >>>>>>>>
> > > >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> > > >>>>>>>> making it easier to contribute testing results from additional labs.
> > > >>>>>>>
> > > >>>>>>> I really don't think the test hooks are a good setup, though. It is
> > > >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > >>>>>>> (Labman) to deal with all the confusion.
> > > >>>>>>
> > > >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> > > >>>>>> management tooling and wrote your own.
> > > >>>>>>
> > > >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> > > >>>>>>
> > > >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> > > >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> > > >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> > > >>>>>> other people with existing labgrid setups to look at what you've done.
> > > >>>>>>
> > > >>>>>>> provides for two yaml configuration files so that everything is in one
> > > >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> > > >>>>>>> easier to maintain that dozens of little files all over the palce.
> > > >>>>>>
> > > >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> > > >>>>>> both cases here are personal preference and not some sort of objective
> > > >>>>>> and easily evaluated thing.
> > > >>>>>>
> > > >>>>>>>>>>> We need an 'all of the above' strategy here.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> > > >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, agreed.
> > > >>>>>>>>>
> > > >>>>>>>>>> I
> > > >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> > > >>>>>>>>>> pre-requisite of an OS being installed.
> > > >>>>>>>>>
> > > >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> > > >>>>>>>>> I hope eventually to have the installer run to completion and then
> > > >>>>>>>>> reboot to check all is well.
> > > >>>>>>>>
> > > >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > >>>>>>>> testing Linaro has going now that automates I believe it was current
> > > >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> > > >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> > > >>>>>>>> that at least some of the existing kernelci labs see the utility in
> > > >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> > > >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> > > >>>>>>>
> > > >>>>>>> OK.
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> > > >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> > > >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> > > >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> > > >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> > > >>>>>>>>>>
> > > >>>>>>>>>> That seems like a lot more work than documenting more what we have
> > > >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> > > >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> > > >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> > > >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> > > >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> > > >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> > > >>>>>>>>>> another thread.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> > > >>>>>>>>
> > > >>>>>>>> With it's own set of technical and legal challenges / obligations and
> > > >>>>>>>> difficulties depending on what you even mean by "self contained". And
> > > >>>>>>>> how often what's run where, and all sorts of other challenges too.
> > > >>>>>>>>
> > > >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> > > >>>>>>>> position that we need to document things more and worry less about
> > > >>>>>>>> prepackaged things. For example, making the documentation for the
> > > >>>>>>>> current net based OS boot means that for bringing up a new board the
> > > >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> > > >>>>>>>> functional OS image that has to also be messed with and is its own
> > > >>>>>>>> challenge.
> > > >>>>>>>
> > > >>>>>>> Yes
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> > > >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> > > >>>>>>>>>> distant third.
> > > >>>>>>>>>
> > > >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> > > >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> > > >>>>>>>>
> > > >>>>>>>> More and more public testing would be great. The notes I linked above
> > > >>>>>>>> explain one of the first problems there being that most companies will
> > > >>>>>>>> not or can not hook a lab to a public CI instance.
> > > >>>>>>>
> > > >>>>>>> Well corporate IT is what it is.
> > > >>>>>>>
> > > >>>>>>> That means that their boards will not be testing in CI, unless they do
> > > >>>>>>> it themselves, right?
> > > >>>>>>
> > > >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> > > >>>>>> tree at URL) and something that's been being done since the beginning
> > > >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> > > >>>>>>
> > > >>>>>>>> The next problem, as
> > > >>>>>>>> both of our personal labs show, is that just maintaining the physical
> > > >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> > > >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> > > >>>>>>>> that in to gitlab.
> > > >>>>>>>
> > > >>>>>>> OK
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> We should move away from relying on maintainers getting around to
> > > >>>>>>>>> testing patches months after they are sent, when they have time, but
> > > >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> > > >>>>>>>>> push this as well.
> > > >>>>>>>>
> > > >>>>>>>> I have been, and the results I've gotten are that companies are testing
> > > >>>>>>>> things internally but there's not any good way to publish results, and
> > > >>>>>>>> that's the kind of framework we're entirely missing.
> > > >>>>>>>
> > > >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> > > >>>>>>
> > > >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> > > >>>>>> going to happen. External collection is how it's handled for the linux
> > > >>>>>> kernel and that community has far more sway than we do. If we ride their
> > > >>>>>> coattails here so to speak, we can get results. If we push for something
> > > >>>>>> completely different we aren't likely to have success.
> > > >>>>>>
> > > >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> > > >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> > > >>>>>>>> that's not using labgrid to see how to configure things.
> > > >>>>>>>
> > > >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > >>>>>>> am happy to publish the config[1], but I still hold my view that all
> > > >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> > > >>>>>>> work with.
> > > >>>>>>
> > > >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> > > >>>>>> same path you took, and we can also support other lab management methods
> > > >>>>>> too, which is important to get as much testing as possible without
> > > >>>>>> needing to centralize everything.
> > > >>>>>
> > > >>>>> In summary, I suppose we just have different visions and ideas, which
> > > >>>>> should be a good thing.
> > > >>>>
> > > >>>> So long as the project can speak with one voice, yes. Which all circles
> > > >>>> back to why I do not think what you're doing with u-boot.org is at all
> > > >>>> helpful.
> > > >>>
> > > >>> I do need a relief valve for when my efforts are blocked.
> > > >>
> > > >> And I do not see how using the project domain is appropriate. Stop doing
> > > >> this. If you can't stop I'm going to reach my own limit here.
> > > >
> > > > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > > > need my own tree while my work is blocked from submission. I suggest
> > > > we discuss it on a call and try to figure out a compromise while we
> > > > are in this state.
> > >
> > > Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
> > >
> > > or u-boot-ci ?
> >
> > I don't have permission to do that. With my tree I am able to have CI
> > running in half the time, use my full lab (not just the subset Tom's
> > tree has), push patches and ideas that others have blocked, point
> > people to my work, etc. It is a much better environment for me to
> > continue (what I see as) necessary innovation.
> >
> > That said, if you wish to add a top-level project for me, then I would
> > push things to it, yes.
>
> To be clear, you can have
> https://source.denx.de/u-boot/contributors/sjg/ but you can't host your
> downstream fork on source.denx.de itself and I wish you wouldn't abuse
> the project domain for it either.

By calling my efforts a 'downstream fork', it seems to me you are
strengthening the current situation. I wish you would change your
terminology here.

Also, I have explained why I feel I must have my own tree for now. If
you are looking at making any changes to accomodate my concerns in the
short term, then perhaps we should refrain from discussing this topic
so often? It may just be wasting space on the mailing list if there is
no room for movement.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 25235 bytes --]

On Thu, Mar 06, 2025 at 09:11:28AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 6 Mar 2025 at 07:32, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Mar 06, 2025 at 07:16:08AM -0700, Simon Glass wrote:
> > > Hi Heiko,
> > >
> > > On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
> > > >
> > > > Hi Simon,
> > > >
> > > > On 27.02.25 20:26, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> > > > >>
> > > > >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > > > >>> Hi Tom,
> > > > >>>
> > > > >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>
> > > > >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > > >>>>> Hi Tom,
> > > > >>>>>
> > > > >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>
> > > > >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > >>>>>>> Hi Tom,
> > > > >>>>>>>
> > > > >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > >>>>>>>>> Hi Tom,
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > >>>>>>>>>>> Hi Tom,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > >>>>>>>>>>>>> Hi Tom,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > >>>>>>>>>>>>>>> Hi Tom,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > >>>>>>>>>>>>>>>>> installer starts up correctly.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > >>>>>>>>>>>>>>>>> ---
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Changes in v2:
> > > > >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> > > > >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> > > > >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> > > > >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> > > > >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> > > > >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> > > > >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> > > > >>>>>>>>>>>>>>>>>     variables:
> > > > >>>>>>>>>>>>>>>>>       ROLE: zybo
> > > > >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +qemu-x86_64:
> > > > >>>>>>>>>>>>>>>>> +  variables:
> > > > >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> > > > >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> > > > >>>>>>>>>>>>>>>> test.py QEMU stanza.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> > > > >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> > > > >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> > > > >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> > > > >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> > > > >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> > > > >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > >>>>>>>>>>>>>>>>> new file mode 100644
> > > > >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> > > > >>>>>>>>>>>>>>>>> --- /dev/null
> > > > >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> > > > >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> > > > >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> > > > >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> > > > >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +import pytest
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> > > > >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> > > > >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> > > > >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> > > > >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> > > > >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> > > > >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Go to end of line
> > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> > > > >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Send our noisy console
> > > > >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> > > > >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> > > > >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > >>>>>>>>>>>>>>>>> +
> > > > >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> > > > >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> > > > >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> > > > >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> > > > >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> > > > >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> > > > >>>>>>>>>>>>>>>> hardware too.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > >>>>>>>>>>>>>>> than what we have today?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> > > > >>>>>>>>>>>>>> test that we have today? No, it's not.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> > > > >>>>>>>>>>>> test, so yes, I guess you forgot.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> > > > >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> > > > >>>>>>>>>> test, like most tests that rely on the real platform.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> And do we need better documentation for test? Yes.
> > > > >>>>>>>>>
> > > > >>>>>>>>> +1
> > > > >>>>>>>>>
> > > > >>>>>>>>> I'll note that I did my bit!
> > > > >>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> > > > >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> > > > >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> > > > >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> > > > >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> > > > >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> > > > >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> > > > >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > >>>>>>>>>>>>> etc. are available.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> > > > >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> > > > >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> > > > >>>>>>>>>>>> example.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> > > > >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> > > > >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> > > > >>>>>>>>>> configuration infrastructure.
> > > > >>>>>>>>>
> > > > >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> > > > >>>>>>>>> and how can I access it?
> > > > >>>>>>>>
> > > > >>>>>>>> I'll point you at the notes for the first call we had recently:
> > > > >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> > > > >>>>>>>
> > > > >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> > > > >>>>>>
> > > > >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> > > > >>>>>> isn't feasible.
> > > > >>>>>>
> > > > >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> > > > >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> > > > >>>>>>>>> it's probably not a good idea.
> > > > >>>>>>>>
> > > > >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> > > > >>>>>>>> making it easier to contribute testing results from additional labs.
> > > > >>>>>>>
> > > > >>>>>>> I really don't think the test hooks are a good setup, though. It is
> > > > >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > >>>>>>> (Labman) to deal with all the confusion.
> > > > >>>>>>
> > > > >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> > > > >>>>>> management tooling and wrote your own.
> > > > >>>>>>
> > > > >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> > > > >>>>>>
> > > > >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> > > > >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> > > > >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> > > > >>>>>> other people with existing labgrid setups to look at what you've done.
> > > > >>>>>>
> > > > >>>>>>> provides for two yaml configuration files so that everything is in one
> > > > >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> > > > >>>>>>> easier to maintain that dozens of little files all over the palce.
> > > > >>>>>>
> > > > >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> > > > >>>>>> both cases here are personal preference and not some sort of objective
> > > > >>>>>> and easily evaluated thing.
> > > > >>>>>>
> > > > >>>>>>>>>>> We need an 'all of the above' strategy here.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> > > > >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Yes, agreed.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> I
> > > > >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> > > > >>>>>>>>>> pre-requisite of an OS being installed.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> > > > >>>>>>>>> I hope eventually to have the installer run to completion and then
> > > > >>>>>>>>> reboot to check all is well.
> > > > >>>>>>>>
> > > > >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > >>>>>>>> testing Linaro has going now that automates I believe it was current
> > > > >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> > > > >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> > > > >>>>>>>> that at least some of the existing kernelci labs see the utility in
> > > > >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> > > > >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > >>>>>>>
> > > > >>>>>>> OK.
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> > > > >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> > > > >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> > > > >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> > > > >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> That seems like a lot more work than documenting more what we have
> > > > >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> > > > >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> > > > >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> > > > >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> > > > >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> > > > >>>>>>>>>> another thread.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> > > > >>>>>>>>
> > > > >>>>>>>> With it's own set of technical and legal challenges / obligations and
> > > > >>>>>>>> difficulties depending on what you even mean by "self contained". And
> > > > >>>>>>>> how often what's run where, and all sorts of other challenges too.
> > > > >>>>>>>>
> > > > >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> > > > >>>>>>>> position that we need to document things more and worry less about
> > > > >>>>>>>> prepackaged things. For example, making the documentation for the
> > > > >>>>>>>> current net based OS boot means that for bringing up a new board the
> > > > >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> > > > >>>>>>>> functional OS image that has to also be messed with and is its own
> > > > >>>>>>>> challenge.
> > > > >>>>>>>
> > > > >>>>>>> Yes
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> > > > >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> > > > >>>>>>>>>> distant third.
> > > > >>>>>>>>>
> > > > >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> > > > >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> > > > >>>>>>>>
> > > > >>>>>>>> More and more public testing would be great. The notes I linked above
> > > > >>>>>>>> explain one of the first problems there being that most companies will
> > > > >>>>>>>> not or can not hook a lab to a public CI instance.
> > > > >>>>>>>
> > > > >>>>>>> Well corporate IT is what it is.
> > > > >>>>>>>
> > > > >>>>>>> That means that their boards will not be testing in CI, unless they do
> > > > >>>>>>> it themselves, right?
> > > > >>>>>>
> > > > >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > >>>>>> tree at URL) and something that's been being done since the beginning
> > > > >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> > > > >>>>>>
> > > > >>>>>>>> The next problem, as
> > > > >>>>>>>> both of our personal labs show, is that just maintaining the physical
> > > > >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> > > > >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> > > > >>>>>>>> that in to gitlab.
> > > > >>>>>>>
> > > > >>>>>>> OK
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> We should move away from relying on maintainers getting around to
> > > > >>>>>>>>> testing patches months after they are sent, when they have time, but
> > > > >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> > > > >>>>>>>>> push this as well.
> > > > >>>>>>>>
> > > > >>>>>>>> I have been, and the results I've gotten are that companies are testing
> > > > >>>>>>>> things internally but there's not any good way to publish results, and
> > > > >>>>>>>> that's the kind of framework we're entirely missing.
> > > > >>>>>>>
> > > > >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> > > > >>>>>>
> > > > >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> > > > >>>>>> going to happen. External collection is how it's handled for the linux
> > > > >>>>>> kernel and that community has far more sway than we do. If we ride their
> > > > >>>>>> coattails here so to speak, we can get results. If we push for something
> > > > >>>>>> completely different we aren't likely to have success.
> > > > >>>>>>
> > > > >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> > > > >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > >>>>>>>> that's not using labgrid to see how to configure things.
> > > > >>>>>>>
> > > > >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > >>>>>>> am happy to publish the config[1], but I still hold my view that all
> > > > >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > >>>>>>> work with.
> > > > >>>>>>
> > > > >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> > > > >>>>>> same path you took, and we can also support other lab management methods
> > > > >>>>>> too, which is important to get as much testing as possible without
> > > > >>>>>> needing to centralize everything.
> > > > >>>>>
> > > > >>>>> In summary, I suppose we just have different visions and ideas, which
> > > > >>>>> should be a good thing.
> > > > >>>>
> > > > >>>> So long as the project can speak with one voice, yes. Which all circles
> > > > >>>> back to why I do not think what you're doing with u-boot.org is at all
> > > > >>>> helpful.
> > > > >>>
> > > > >>> I do need a relief valve for when my efforts are blocked.
> > > > >>
> > > > >> And I do not see how using the project domain is appropriate. Stop doing
> > > > >> this. If you can't stop I'm going to reach my own limit here.
> > > > >
> > > > > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > > > > need my own tree while my work is blocked from submission. I suggest
> > > > > we discuss it on a call and try to figure out a compromise while we
> > > > > are in this state.
> > > >
> > > > Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
> > > >
> > > > or u-boot-ci ?
> > >
> > > I don't have permission to do that. With my tree I am able to have CI
> > > running in half the time, use my full lab (not just the subset Tom's
> > > tree has), push patches and ideas that others have blocked, point
> > > people to my work, etc. It is a much better environment for me to
> > > continue (what I see as) necessary innovation.
> > >
> > > That said, if you wish to add a top-level project for me, then I would
> > > push things to it, yes.
> >
> > To be clear, you can have
> > https://source.denx.de/u-boot/contributors/sjg/ but you can't host your
> > downstream fork on source.denx.de itself and I wish you wouldn't abuse
> > the project domain for it either.
> 
> By calling my efforts a 'downstream fork', it seems to me you are
> strengthening the current situation. I wish you would change your
> terminology here.
> 
> Also, I have explained why I feel I must have my own tree for now. If
> you are looking at making any changes to accomodate my concerns in the
> short term, then perhaps we should refrain from discussing this topic
> so often? It may just be wasting space on the mailing list if there is
> no room for movement.

We probably should keep discussion of what you see as your role in
U-Boot to another thread, yes. But I don't know what to call your
personal not mainline tree if not a fork.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Simon Glass]

Hi Tom,

On Thu, 6 Mar 2025 at 09:43, Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Mar 06, 2025 at 09:11:28AM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Thu, 6 Mar 2025 at 07:32, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Thu, Mar 06, 2025 at 07:16:08AM -0700, Simon Glass wrote:
> > > > Hi Heiko,
> > > >
> > > > On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
> > > > >
> > > > > Hi Simon,
> > > > >
> > > > > On 27.02.25 20:26, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>
> > > > > >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > > > > >>> Hi Tom,
> > > > > >>>
> > > > > >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>
> > > > > >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > > > >>>>> Hi Tom,
> > > > > >>>>>
> > > > > >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > > >>>>>>> Hi Tom,
> > > > > >>>>>>>
> > > > > >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > >>>>>>>>> Hi Tom,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > >>>>>>>>>>> Hi Tom,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > >>>>>>>>>>>>> Hi Tom,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > >>>>>>>>>>>>>>> Hi Tom,
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > >>>>>>>>>>>>>>>>> installer starts up correctly.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > >>>>>>>>>>>>>>>>> ---
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Changes in v2:
> > > > > >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> > > > > >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> > > > > >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> > > > > >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> > > > > >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> > > > > >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> > > > > >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> > > > > >>>>>>>>>>>>>>>>>     variables:
> > > > > >>>>>>>>>>>>>>>>>       ROLE: zybo
> > > > > >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +qemu-x86_64:
> > > > > >>>>>>>>>>>>>>>>> +  variables:
> > > > > >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> > > > > >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> > > > > >>>>>>>>>>>>>>>> test.py QEMU stanza.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> > > > > >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> > > > > >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> > > > > >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> > > > > >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> > > > > >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> > > > > >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > >>>>>>>>>>>>>>>>> new file mode 100644
> > > > > >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> > > > > >>>>>>>>>>>>>>>>> --- /dev/null
> > > > > >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> > > > > >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> > > > > >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> > > > > >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> > > > > >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +import pytest
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> > > > > >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> > > > > >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> > > > > >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> > > > > >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> > > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> > > > > >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> > > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> > > > > >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Go to end of line
> > > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> > > > > >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Send our noisy console
> > > > > >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> > > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> > > > > >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> > > > > >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > >>>>>>>>>>>>>>>>> +
> > > > > >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> > > > > >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> > > > > >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> > > > > >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> > > > > >>>>>>>>>>>>>>>> hardware too.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > >>>>>>>>>>>>>>> than what we have today?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > >>>>>>>>>>>>>> test that we have today? No, it's not.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> > > > > >>>>>>>>>>>> test, so yes, I guess you forgot.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> > > > > >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> > > > > >>>>>>>>>> test, like most tests that rely on the real platform.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> And do we need better documentation for test? Yes.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> +1
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I'll note that I did my bit!
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> > > > > >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> > > > > >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> > > > > >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> > > > > >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> > > > > >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> > > > > >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> > > > > >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > >>>>>>>>>>>>> etc. are available.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> > > > > >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> > > > > >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> > > > > >>>>>>>>>>>> example.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> > > > > >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> > > > > >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> > > > > >>>>>>>>>> configuration infrastructure.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> > > > > >>>>>>>>> and how can I access it?
> > > > > >>>>>>>>
> > > > > >>>>>>>> I'll point you at the notes for the first call we had recently:
> > > > > >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> > > > > >>>>>>>
> > > > > >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> > > > > >>>>>>
> > > > > >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> > > > > >>>>>> isn't feasible.
> > > > > >>>>>>
> > > > > >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> > > > > >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> > > > > >>>>>>>>> it's probably not a good idea.
> > > > > >>>>>>>>
> > > > > >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> > > > > >>>>>>>> making it easier to contribute testing results from additional labs.
> > > > > >>>>>>>
> > > > > >>>>>>> I really don't think the test hooks are a good setup, though. It is
> > > > > >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > > >>>>>>> (Labman) to deal with all the confusion.
> > > > > >>>>>>
> > > > > >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> > > > > >>>>>> management tooling and wrote your own.
> > > > > >>>>>>
> > > > > >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> > > > > >>>>>>
> > > > > >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> > > > > >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> > > > > >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> > > > > >>>>>> other people with existing labgrid setups to look at what you've done.
> > > > > >>>>>>
> > > > > >>>>>>> provides for two yaml configuration files so that everything is in one
> > > > > >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> > > > > >>>>>>> easier to maintain that dozens of little files all over the palce.
> > > > > >>>>>>
> > > > > >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> > > > > >>>>>> both cases here are personal preference and not some sort of objective
> > > > > >>>>>> and easily evaluated thing.
> > > > > >>>>>>
> > > > > >>>>>>>>>>> We need an 'all of the above' strategy here.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> > > > > >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Yes, agreed.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> I
> > > > > >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> > > > > >>>>>>>>>> pre-requisite of an OS being installed.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > >>>>>>>>> I hope eventually to have the installer run to completion and then
> > > > > >>>>>>>>> reboot to check all is well.
> > > > > >>>>>>>>
> > > > > >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > >>>>>>>> testing Linaro has going now that automates I believe it was current
> > > > > >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> > > > > >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > >>>>>>>> that at least some of the existing kernelci labs see the utility in
> > > > > >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> > > > > >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > > >>>>>>>
> > > > > >>>>>>> OK.
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> > > > > >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> > > > > >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> > > > > >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> That seems like a lot more work than documenting more what we have
> > > > > >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> > > > > >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> > > > > >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> > > > > >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> > > > > >>>>>>>>>> another thread.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> > > > > >>>>>>>>
> > > > > >>>>>>>> With it's own set of technical and legal challenges / obligations and
> > > > > >>>>>>>> difficulties depending on what you even mean by "self contained". And
> > > > > >>>>>>>> how often what's run where, and all sorts of other challenges too.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> > > > > >>>>>>>> position that we need to document things more and worry less about
> > > > > >>>>>>>> prepackaged things. For example, making the documentation for the
> > > > > >>>>>>>> current net based OS boot means that for bringing up a new board the
> > > > > >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> > > > > >>>>>>>> functional OS image that has to also be messed with and is its own
> > > > > >>>>>>>> challenge.
> > > > > >>>>>>>
> > > > > >>>>>>> Yes
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> > > > > >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> > > > > >>>>>>>>>> distant third.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> > > > > >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> > > > > >>>>>>>>
> > > > > >>>>>>>> More and more public testing would be great. The notes I linked above
> > > > > >>>>>>>> explain one of the first problems there being that most companies will
> > > > > >>>>>>>> not or can not hook a lab to a public CI instance.
> > > > > >>>>>>>
> > > > > >>>>>>> Well corporate IT is what it is.
> > > > > >>>>>>>
> > > > > >>>>>>> That means that their boards will not be testing in CI, unless they do
> > > > > >>>>>>> it themselves, right?
> > > > > >>>>>>
> > > > > >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > > >>>>>> tree at URL) and something that's been being done since the beginning
> > > > > >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> > > > > >>>>>>
> > > > > >>>>>>>> The next problem, as
> > > > > >>>>>>>> both of our personal labs show, is that just maintaining the physical
> > > > > >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> > > > > >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> > > > > >>>>>>>> that in to gitlab.
> > > > > >>>>>>>
> > > > > >>>>>>> OK
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>> We should move away from relying on maintainers getting around to
> > > > > >>>>>>>>> testing patches months after they are sent, when they have time, but
> > > > > >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> > > > > >>>>>>>>> push this as well.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I have been, and the results I've gotten are that companies are testing
> > > > > >>>>>>>> things internally but there's not any good way to publish results, and
> > > > > >>>>>>>> that's the kind of framework we're entirely missing.
> > > > > >>>>>>>
> > > > > >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> > > > > >>>>>>
> > > > > >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > > >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> > > > > >>>>>> going to happen. External collection is how it's handled for the linux
> > > > > >>>>>> kernel and that community has far more sway than we do. If we ride their
> > > > > >>>>>> coattails here so to speak, we can get results. If we push for something
> > > > > >>>>>> completely different we aren't likely to have success.
> > > > > >>>>>>
> > > > > >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> > > > > >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > >>>>>>>> that's not using labgrid to see how to configure things.
> > > > > >>>>>>>
> > > > > >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > > >>>>>>> am happy to publish the config[1], but I still hold my view that all
> > > > > >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > > >>>>>>> work with.
> > > > > >>>>>>
> > > > > >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> > > > > >>>>>> same path you took, and we can also support other lab management methods
> > > > > >>>>>> too, which is important to get as much testing as possible without
> > > > > >>>>>> needing to centralize everything.
> > > > > >>>>>
> > > > > >>>>> In summary, I suppose we just have different visions and ideas, which
> > > > > >>>>> should be a good thing.
> > > > > >>>>
> > > > > >>>> So long as the project can speak with one voice, yes. Which all circles
> > > > > >>>> back to why I do not think what you're doing with u-boot.org is at all
> > > > > >>>> helpful.
> > > > > >>>
> > > > > >>> I do need a relief valve for when my efforts are blocked.
> > > > > >>
> > > > > >> And I do not see how using the project domain is appropriate. Stop doing
> > > > > >> this. If you can't stop I'm going to reach my own limit here.
> > > > > >
> > > > > > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > > > > > need my own tree while my work is blocked from submission. I suggest
> > > > > > we discuss it on a call and try to figure out a compromise while we
> > > > > > are in this state.
> > > > >
> > > > > Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
> > > > >
> > > > > or u-boot-ci ?
> > > >
> > > > I don't have permission to do that. With my tree I am able to have CI
> > > > running in half the time, use my full lab (not just the subset Tom's
> > > > tree has), push patches and ideas that others have blocked, point
> > > > people to my work, etc. It is a much better environment for me to
> > > > continue (what I see as) necessary innovation.
> > > >
> > > > That said, if you wish to add a top-level project for me, then I would
> > > > push things to it, yes.
> > >
> > > To be clear, you can have
> > > https://source.denx.de/u-boot/contributors/sjg/ but you can't host your
> > > downstream fork on source.denx.de itself and I wish you wouldn't abuse
> > > the project domain for it either.
> >
> > By calling my efforts a 'downstream fork', it seems to me you are
> > strengthening the current situation. I wish you would change your
> > terminology here.
> >
> > Also, I have explained why I feel I must have my own tree for now. If
> > you are looking at making any changes to accomodate my concerns in the
> > short term, then perhaps we should refrain from discussing this topic
> > so often? It may just be wasting space on the mailing list if there is
> > no room for movement.
>
> We probably should keep discussion of what you see as your role in
> U-Boot to another thread, yes. But I don't know what to call your
> personal not mainline tree if not a fork.

OK good.

You could perhaps just call it 'Simon's tree'. After all, the extent
to which it becomes a fork is under your control, not mine.

Regards,
Simon

Re: [PATCH v2 28/28] test: Add a test for booting Ubuntu 24.04

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 27359 bytes --]

On Thu, Mar 06, 2025 at 04:31:14PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Thu, 6 Mar 2025 at 09:43, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Mar 06, 2025 at 09:11:28AM -0700, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Thu, 6 Mar 2025 at 07:32, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Thu, Mar 06, 2025 at 07:16:08AM -0700, Simon Glass wrote:
> > > > > Hi Heiko,
> > > > >
> > > > > On Thu, 27 Feb 2025 at 22:26, Heiko Schocher <hs@denx.de> wrote:
> > > > > >
> > > > > > Hi Simon,
> > > > > >
> > > > > > On 27.02.25 20:26, Simon Glass wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > On Thu, 27 Feb 2025 at 10:20, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>
> > > > > > >> On Thu, Feb 27, 2025 at 09:27:33AM -0700, Simon Glass wrote:
> > > > > > >>> Hi Tom,
> > > > > > >>>
> > > > > > >>> On Wed, 26 Feb 2025 at 07:35, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>
> > > > > > >>>> On Tue, Feb 25, 2025 at 07:56:03PM -0700, Simon Glass wrote:
> > > > > > >>>>> Hi Tom,
> > > > > > >>>>>
> > > > > > >>>>> On Tue, 25 Feb 2025 at 06:59, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Feb 24, 2025 at 10:54:13AM -0700, Simon Glass wrote:
> > > > > > >>>>>>> Hi Tom,
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Fri, 21 Feb 2025 at 09:06, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Fri, Feb 21, 2025 at 06:57:34AM -0700, Simon Glass wrote:
> > > > > > >>>>>>>>> Hi Tom,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Thu, 20 Feb 2025 at 07:53, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Thu, Feb 20, 2025 at 06:49:49AM -0700, Simon Glass wrote:
> > > > > > >>>>>>>>>>> Hi Tom,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Tue, 18 Feb 2025 at 17:55, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:01:40PM -0700, Simon Glass wrote:
> > > > > > >>>>>>>>>>>>> Hi Tom,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Tue, 18 Feb 2025 at 08:11, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Tue, Feb 18, 2025 at 05:09:23AM -0700, Simon Glass wrote:
> > > > > > >>>>>>>>>>>>>>> Hi Tom,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Mon, 17 Feb 2025 at 10:52, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Sun, Feb 16, 2025 at 01:44:13PM -0700, Simon Glass wrote:
> > > > > > >>>>>>>>>>>>>>>>> Now that U-Boot can boot this quickly, using kvm, add a test that the
> > > > > > >>>>>>>>>>>>>>>>> installer starts up correctly.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Use the qemu-x86_64 board in the SJG lab.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Signed-off-by: Simon Glass <sjg@chromium.org>
> > > > > > >>>>>>>>>>>>>>>>> ---
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Changes in v2:
> > > > > > >>>>>>>>>>>>>>>>> - Add more patches to support booting with kvm
> > > > > > >>>>>>>>>>>>>>>>> - Add new patch with a test for booting Ubuntu 24.04
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>   .gitlab-ci.yml               |  5 ++++
> > > > > > >>>>>>>>>>>>>>>>>   test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++
> > > > > > >>>>>>>>>>>>>>>>>   2 files changed, 58 insertions(+)
> > > > > > >>>>>>>>>>>>>>>>>   create mode 100644 test/py/tests/test_distro.py
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > > > > >>>>>>>>>>>>>>>>> index 8c49d5b0a79..ec799e97c10 100644
> > > > > > >>>>>>>>>>>>>>>>> --- a/.gitlab-ci.yml
> > > > > > >>>>>>>>>>>>>>>>> +++ b/.gitlab-ci.yml
> > > > > > >>>>>>>>>>>>>>>>> @@ -745,3 +745,8 @@ zybo:
> > > > > > >>>>>>>>>>>>>>>>>     variables:
> > > > > > >>>>>>>>>>>>>>>>>       ROLE: zybo
> > > > > > >>>>>>>>>>>>>>>>>     <<: *lab_dfn
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +qemu-x86_64:
> > > > > > >>>>>>>>>>>>>>>>> +  variables:
> > > > > > >>>>>>>>>>>>>>>>> +    ROLE: qemu-x86_64
> > > > > > >>>>>>>>>>>>>>>>> +  <<: *lab_dfn
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I'm not sure why this is in your lab stanza, rather than the normal
> > > > > > >>>>>>>>>>>>>>>> test.py QEMU stanza.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Are you wanting to add the Ubuntu image into CI? It is quite large.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> If we're going to be able to run it on N platforms, yes, we need to
> > > > > > >>>>>>>>>>>>>> think of a good way to cache the download. There's not a particular
> > > > > > >>>>>>>>>>>>>> reason we can't run the stock Ubuntu RISC-V image on the two sifive
> > > > > > >>>>>>>>>>>>>> targets and also qemu-riscv64, is there?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Yes, we can do that. It is pretty simple to set up in Labgrid and it
> > > > > > >>>>>>>>>>>>> doesn't require all the runners to download a much larger image, etc.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I don't quite understand why it's under "labgrid". These are generic CI
> > > > > > >>>>>>>>>>>> tests. Now maybe we need to, in both Gitlab and Azure, add some logic so
> > > > > > >>>>>>>>>>>> that certain longer or possibly destructive tests are only run on tagged
> > > > > > >>>>>>>>>>>> releases or as requested rather than every time, as it will take longer.
> > > > > > >>>>>>>>>>>> But pretty much every platform under the qemu target list should be able
> > > > > > >>>>>>>>>>>> to Just Boot an off the shelf OS distribution is my point.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Sure, and I'm not suggesting we shouldn't do that as well.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py
> > > > > > >>>>>>>>>>>>>>>>> new file mode 100644
> > > > > > >>>>>>>>>>>>>>>>> index 00000000000..51eec45cecc
> > > > > > >>>>>>>>>>>>>>>>> --- /dev/null
> > > > > > >>>>>>>>>>>>>>>>> +++ b/test/py/tests/test_distro.py
> > > > > > >>>>>>>>>>>>>>>>> @@ -0,0 +1,53 @@
> > > > > > >>>>>>>>>>>>>>>>> +# SPDX-License-Identifier: GPL-2.0+
> > > > > > >>>>>>>>>>>>>>>>> +# Copyright 2025 Canonical Ltd.
> > > > > > >>>>>>>>>>>>>>>>> +# Written by Simon Glass <simon.glass@canonical.com>
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +import pytest
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +DOWN = '\x1b\x5b\x42\x0d'
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +# Enable early console so that the test can see if something goes wrong
> > > > > > >>>>>>>>>>>>>>>>> +CONSOLE = 'earlycon=uart8250,io,0x3f8 console=uart8250,io,0x3f8'
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +@pytest.mark.boardspec('qemu-x86_64')
> > > > > > >>>>>>>>>>>>>>>>> +@pytest.mark.role('qemu-x86_64')
> > > > > > >>>>>>>>>>>>>>>>> +def test_distro(ubman):
> > > > > > >>>>>>>>>>>>>>>>> +    """Test that of-platdata can be generated and used in sandbox"""
> > > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('boot'):
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('boot', wait_for_prompt=False)
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Grub'):
> > > > > > >>>>>>>>>>>>>>>>> +        # Wait for grub to come up and offset a menu
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Try or Install Ubuntu'])
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Press 'e' to edit the command line
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.run_command('e', wait_for_prompt=False, send_nl=False)
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Wait until we see the editor appear
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['/casper/initrd'])
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Go down to the 'linux' line
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.send(DOWN * 3)
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Go to end of line
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('E')
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Backspace to remove 'quiet splash'
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.send('\b' * len('quiet splash'))
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Send our noisy console
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.send(CONSOLE)
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +        # Tell grub to boot
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.ctrl('X')
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Booting a command list'])
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Linux'):
> > > > > > >>>>>>>>>>>>>>>>> +        # Linux should start immediately
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Linux version'])
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +    with ubman.log.section('Ubuntu'):
> > > > > > >>>>>>>>>>>>>>>>> +        # Shortly later, we should see this banner
> > > > > > >>>>>>>>>>>>>>>>> +        ubman.p.expect(['Welcome to .*Ubuntu 24.04.1 LTS.*!'])
> > > > > > >>>>>>>>>>>>>>>>> +
> > > > > > >>>>>>>>>>>>>>>>> +    ubman.restart_uboot()
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> And this seems very inflexible. Please see
> > > > > > >>>>>>>>>>>>>>>> test/py/tests/test_net_boot.py for an example of how to have this be
> > > > > > >>>>>>>>>>>>>>>> configurable and work on arbitrary platforms. What I assume is tricky is
> > > > > > >>>>>>>>>>>>>>>> that the "role" part here is where you have a special disk image being
> > > > > > >>>>>>>>>>>>>>>> passed. That too could be dealt with in u-boot-test-hooks in a few ways,
> > > > > > >>>>>>>>>>>>>>>> and the images pre-fetched to the CI container. And if this was
> > > > > > >>>>>>>>>>>>>>>> configurable similar to the example I noted above, it could check real
> > > > > > >>>>>>>>>>>>>>>> hardware too.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> That wasn't the reaction I expected.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Yes, it is inflexible, but it is a starting point. Isn't it better
> > > > > > >>>>>>>>>>>>>>> than what we have today?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Is your inflexible boot an OS test better than the flexible boot an OS
> > > > > > >>>>>>>>>>>>>> test that we have today? No, it's not.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I didn't even know about it, or perhaps I forgot.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I believe I mentioned it every time you've said we should have an OS
> > > > > > >>>>>>>>>>>> test, so yes, I guess you forgot.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Well it was only added in May last year and it relies on board config
> > > > > > >>>>>>>>>>> which I don't have...although I see that you have now posted yours.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Yes, it was added not quite a year ago, and is documented within the
> > > > > > >>>>>>>>>> test, like most tests that rely on the real platform.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> And do we need better documentation for test? Yes.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> +1
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I'll note that I did my bit!
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Perhaps this relates to getting the labgrid config published and
> > > > > > >>>>>>>>>>>>> figuring out how to pass info from Labgrid to tests.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I would like to generalise this test to work on at least one real
> > > > > > >>>>>>>>>>>>>>> board, preferably one that doesn't use grub.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> OK. The test we have today does that, if you check for the "Welcome to
> > > > > > >>>>>>>>>>>>>> ..." string instead of the kernel has booted string. It also does
> > > > > > >>>>>>>>>>>>>> netboot rather than run default bootcmd. But that's an easy enough test
> > > > > > >>>>>>>>>>>>>> to write up. The only thing stopping me from doing that right now is I
> > > > > > >>>>>>>>>>>>>> need to find a board in the lab where we installed an OS to eMMC and not
> > > > > > >>>>>>>>>>>>>> SD card (some lab sd-mux issues).
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> OK. Labgrid has a 'features' thing which you can attach to targets, so
> > > > > > >>>>>>>>>>>>> I should be able to use that to indicate that Ubuntu, Debian, Armbian,
> > > > > > >>>>>>>>>>>>> etc. are available.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> OK, but that sounds like the opposite direction. These are generic tests
> > > > > > >>>>>>>>>>>> that can run in any / all of the labs, not just your labgrid
> > > > > > >>>>>>>>>>>> configuration. AMD has been contributing tests that run on hardware for
> > > > > > >>>>>>>>>>>> example.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> That's great, the more tests we have the better. But those tests can't
> > > > > > >>>>>>>>>>> and don't run in CI, whereas mine can and do.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> AFAICT they're running on AMD's CI. They run on my CI. They don't run on
> > > > > > >>>>>>>>>> *your* lab because you took things, intentionally, in a direction to
> > > > > > >>>>>>>>>> minimize using u-boot-test-hooks and our existing per-board
> > > > > > >>>>>>>>>> configuration infrastructure.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> When I look at CI all I see is my lab. Which CI are you referring to
> > > > > > >>>>>>>>> and how can I access it?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I'll point you at the notes for the first call we had recently:
> > > > > > >>>>>>>> https://lore.kernel.org/u-boot/20250128171923.GQ1233568@bill-the-cat/
> > > > > > >>>>>>>> and note that there are many labs doing testing on / with U-Boot.
> > > > > > >>>>>>>
> > > > > > >>>>>>> That's all good, but it isn't as good as having the lab in gitlab.
> > > > > > >>>>>>
> > > > > > >>>>>> Strongly disagree. Especially since having it in the mainline gitlab
> > > > > > >>>>>> isn't feasible.
> > > > > > >>>>>>
> > > > > > >>>>>>>>> Here I would like to make a case for moving to using Labgrid across
> > > > > > >>>>>>>>> the board, but unfortunately the project struggles to review PRs, so
> > > > > > >>>>>>>>> it's probably not a good idea.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> It would also be counter to the feedback from the U-Boot community about
> > > > > > >>>>>>>> making it easier to contribute testing results from additional labs.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I really don't think the test hooks are a good setup, though. It is
> > > > > > >>>>>>> OK-ish for small labs, but it is so fiddly to use that I wrote a tool
> > > > > > >>>>>>> (Labman) to deal with all the confusion.
> > > > > > >>>>>>
> > > > > > >>>>>> Yes, I don't know how hard you evaluated all of the then-current lab
> > > > > > >>>>>> management tooling and wrote your own.
> > > > > > >>>>>>
> > > > > > >>>>>>> Labgrid (which you suggested I use for my lab, if you recall),
> > > > > > >>>>>>
> > > > > > >>>>>> Yes, and I think you forgot the aim was to make it easy to show all of
> > > > > > >>>>>> the existing Labgrid based labs that do Linux kernel testing they could
> > > > > > >>>>>> easily add U-Boot to the mix. I've been trying to get feedback from
> > > > > > >>>>>> other people with existing labgrid setups to look at what you've done.
> > > > > > >>>>>>
> > > > > > >>>>>>> provides for two yaml configuration files so that everything is in one
> > > > > > >>>>>>> place. Apart from its primitive support for USB hubs, it is much
> > > > > > >>>>>>> easier to maintain that dozens of little files all over the palce.
> > > > > > >>>>>>
> > > > > > >>>>>> I mean, I looked at what you posted and strongly disagree, but I think
> > > > > > >>>>>> both cases here are personal preference and not some sort of objective
> > > > > > >>>>>> and easily evaluated thing.
> > > > > > >>>>>>
> > > > > > >>>>>>>>>>> We need an 'all of the above' strategy here.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Sure. But I still want to see things as reusable as possible. What you
> > > > > > >>>>>>>>>> have above is *extremely* board and OS specific and non-configurable.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Yes, agreed.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> I
> > > > > > >>>>>>>>>> also don't quite see why it's not a test of autoboot with the
> > > > > > >>>>>>>>>> pre-requisite of an OS being installed.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Ah OK, my test is just for the installer itself. Both are useful, but
> > > > > > >>>>>>>>> I hope eventually to have the installer run to completion and then
> > > > > > >>>>>>>>> reboot to check all is well.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> In the spirit of "yes, and.."'ing tests, sure. Ilias pointed me at some
> > > > > > >>>>>>>> testing Linaro has going now that automates I believe it was current
> > > > > > >>>>>>>> Yocto and current U-Boot (+ the pmb patches that've been posted) doing a
> > > > > > >>>>>>>> full install via network in CI. So yes, a Canonical lab might also find
> > > > > > >>>>>>>> it useful to end to end test installing Ubuntu. My own personal dream is
> > > > > > >>>>>>>> that at least some of the existing kernelci labs see the utility in
> > > > > > >>>>>>>> adding "current U-Boot" as one of the matrix variables they test and not
> > > > > > >>>>>>>> just "U-Boot as delivered by vendor" as a static part of the testing.
> > > > > > >>>>>>>
> > > > > > >>>>>>> OK.
> > > > > > >>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>>>> BTW, having thought about how test/py works a bit, instead of the
> > > > > > >>>>>>>>>>> env__net_tftp_bootable_file stuff, we should have code or data which
> > > > > > >>>>>>>>>>> sets up the required test files (on a suitable server) before running
> > > > > > >>>>>>>>>>> the test. That way, all the test code is in one Python file and we
> > > > > > >>>>>>>>>>> don't have to spend ages trying to divine what each test needs.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> That seems like a lot more work than documenting more what we have
> > > > > > >>>>>>>>>> today, and I'm not sure of the benefit. Given the contents of the pxe
> > > > > > >>>>>>>>>> test, yes, just having those files available to 'cp' in place would be
> > > > > > >>>>>>>>>> helpful. But that's not the case for booting a kernel (the FIT match
> > > > > > >>>>>>>>>> stuff doesn't work on the TI platforms atm). And if you look at the
> > > > > > >>>>>>>>>> config I posted it also includes bootstage configuration. It also won't
> > > > > > >>>>>>>>>> work well for the SPI tests, which I'm talking with Love about in
> > > > > > >>>>>>>>>> another thread.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Yes, perhaps, but having self-contained tests would be a win.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> With it's own set of technical and legal challenges / obligations and
> > > > > > >>>>>>>> difficulties depending on what you even mean by "self contained". And
> > > > > > >>>>>>>> how often what's run where, and all sorts of other challenges too.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Given the extreme depth that testing can go to, this is why I'm of the
> > > > > > >>>>>>>> position that we need to document things more and worry less about
> > > > > > >>>>>>>> prepackaged things. For example, making the documentation for the
> > > > > > >>>>>>>> current net based OS boot means that for bringing up a new board the
> > > > > > >>>>>>>> developer can just drop something in. Whereas if the tests expect a
> > > > > > >>>>>>>> functional OS image that has to also be messed with and is its own
> > > > > > >>>>>>>> challenge.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Yes
> > > > > > >>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>>> In other words, the majority of py/<host>/u_boot_boardenv_ content is
> > > > > > >>>>>>>>>> configuration details, specific to both the platform / SoC first, some
> > > > > > >>>>>>>>>> lab specific details second and drop-in existing 3rd party files a
> > > > > > >>>>>>>>>> distant third.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I think the u-boot-test-hooks was an amazing solution 9 years ago, but
> > > > > > >>>>>>>>> we have outgrown it. We want people to be able to connect their lab to
> > > > > > >>>>>>>>> CI (meaning gitlab), so testing is more automated.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> More and more public testing would be great. The notes I linked above
> > > > > > >>>>>>>> explain one of the first problems there being that most companies will
> > > > > > >>>>>>>> not or can not hook a lab to a public CI instance.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Well corporate IT is what it is.
> > > > > > >>>>>>>
> > > > > > >>>>>>> That means that their boards will not be testing in CI, unless they do
> > > > > > >>>>>>> it themselves, right?
> > > > > > >>>>>>
> > > > > > >>>>>> I'm not sure what you mean here. It's a solved problem for them (monitor
> > > > > > >>>>>> tree at URL) and something that's been being done since the beginning
> > > > > > >>>>>> even for U-Boot (it's how the original nvidia lab worked).
> > > > > > >>>>>>
> > > > > > >>>>>>>> The next problem, as
> > > > > > >>>>>>>> both of our personal labs show, is that just maintaining the physical
> > > > > > >>>>>>>> lab takes time and resources. I've added Heiko here because I've been
> > > > > > >>>>>>>> talking with him off-list about expanding tbot coverage and plumbing
> > > > > > >>>>>>>> that in to gitlab.
> > > > > > >>>>>>>
> > > > > > >>>>>>> OK
> > > > > > >>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> We should move away from relying on maintainers getting around to
> > > > > > >>>>>>>>> testing patches months after they are sent, when they have time, but
> > > > > > >>>>>>>>> they don't. Things need to be more automated and I'd encourage you to
> > > > > > >>>>>>>>> push this as well.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I have been, and the results I've gotten are that companies are testing
> > > > > > >>>>>>>> things internally but there's not any good way to publish results, and
> > > > > > >>>>>>>> that's the kind of framework we're entirely missing.
> > > > > > >>>>>>>
> > > > > > >>>>>>> If you like, but from my side, I like to see the results in gitlab.
> > > > > > >>>>>>
> > > > > > >>>>>> Depends on what you mean by gitlab. I assume you mean "triggered by a
> > > > > > >>>>>> push and visible in the main pipeline". Which isn't possible. It's not
> > > > > > >>>>>> going to happen. External collection is how it's handled for the linux
> > > > > > >>>>>> kernel and that community has far more sway than we do. If we ride their
> > > > > > >>>>>> coattails here so to speak, we can get results. If we push for something
> > > > > > >>>>>> completely different we aren't likely to have success.
> > > > > > >>>>>>
> > > > > > >>>>>>>> Which is another part of why I keep pushing against having U-Boot
> > > > > > >>>>>>>> configuration stuff inside of Labgrid as it makes it harder for any lab
> > > > > > >>>>>>>> that's not using labgrid to see how to configure things.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Well, as you requested, I looked at Labgrid and now my lab uses it. I
> > > > > > >>>>>>> am happy to publish the config[1], but I still hold my view that all
> > > > > > >>>>>>> the shell scripts in u-boot-test-hooks are limiting and painful to
> > > > > > >>>>>>> work with.
> > > > > > >>>>>>
> > > > > > >>>>>> Yes, and as I've shown, you can also use labgrid without going down the
> > > > > > >>>>>> same path you took, and we can also support other lab management methods
> > > > > > >>>>>> too, which is important to get as much testing as possible without
> > > > > > >>>>>> needing to centralize everything.
> > > > > > >>>>>
> > > > > > >>>>> In summary, I suppose we just have different visions and ideas, which
> > > > > > >>>>> should be a good thing.
> > > > > > >>>>
> > > > > > >>>> So long as the project can speak with one voice, yes. Which all circles
> > > > > > >>>> back to why I do not think what you're doing with u-boot.org is at all
> > > > > > >>>> helpful.
> > > > > > >>>
> > > > > > >>> I do need a relief valve for when my efforts are blocked.
> > > > > > >>
> > > > > > >> And I do not see how using the project domain is appropriate. Stop doing
> > > > > > >> this. If you can't stop I'm going to reach my own limit here.
> > > > > > >
> > > > > > > I'm happy to have source.u-boot.org or similar point to Denx, but I do
> > > > > > > need my own tree while my work is blocked from submission. I suggest
> > > > > > > we discuss it on a call and try to figure out a compromise while we
> > > > > > > are in this state.
> > > > > >
> > > > > > Isn;t it possible to add a repo at source.denx.de ? Like u-boot-sjg ?
> > > > > >
> > > > > > or u-boot-ci ?
> > > > >
> > > > > I don't have permission to do that. With my tree I am able to have CI
> > > > > running in half the time, use my full lab (not just the subset Tom's
> > > > > tree has), push patches and ideas that others have blocked, point
> > > > > people to my work, etc. It is a much better environment for me to
> > > > > continue (what I see as) necessary innovation.
> > > > >
> > > > > That said, if you wish to add a top-level project for me, then I would
> > > > > push things to it, yes.
> > > >
> > > > To be clear, you can have
> > > > https://source.denx.de/u-boot/contributors/sjg/ but you can't host your
> > > > downstream fork on source.denx.de itself and I wish you wouldn't abuse
> > > > the project domain for it either.
> > >
> > > By calling my efforts a 'downstream fork', it seems to me you are
> > > strengthening the current situation. I wish you would change your
> > > terminology here.
> > >
> > > Also, I have explained why I feel I must have my own tree for now. If
> > > you are looking at making any changes to accomodate my concerns in the
> > > short term, then perhaps we should refrain from discussing this topic
> > > so often? It may just be wasting space on the mailing list if there is
> > > no room for movement.
> >
> > We probably should keep discussion of what you see as your role in
> > U-Boot to another thread, yes. But I don't know what to call your
> > personal not mainline tree if not a fork.
> 
> OK good.
> 
> You could perhaps just call it 'Simon's tree'. After all, the extent
> to which it becomes a fork is under your control, not mine.

To borrow a phrase, it takes two to tango.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 00/28] x86: Improve operation under QEMU

[Paolo Bonzini]

On 2/16/25 21:43, Simon Glass wrote:
> U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
> is not perfect.
> 
> With both builds, executing the VESA ROM causes an intermittent hang, at
> least on some AMD CPUs.
> 
> With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
> is done in a way that works on real hardware but not with QEMU. This
> means that performance is 4-5x slower than it could be, at least on my
> CPU.

Do you have a reproducer for this?  It shouldn't happen... also, do you 
know if this is broken for both Intel and AMD CPUs?

Thanks,

Paolo

> We can work around the first problem by using Bochs, which is anyway a
> better choice than VESA for QEMU. The second can be addressed by using
> the same descriptor across the jump to long mode.
> 
> With an MTRR fix this allows booting into Ubuntu on qemu-x86_64
> 
> Changes in v2:
> - Redo commit message
> - Add new patch to rename the _D dirty flag
> - Add new patch to support CPU functions in long mode
> - Add new patch to tidy up address size in MTRR calculations
> - Add new patch with a 64-bit version of is_power_of_2()
> - Add new patch to set an MTRR for the RAM in QEMU
> - Add new patch with a helper to send characters
> - Add new patch to allow tests to be filtered by role
> - Add more patches to support booting with kvm
> - Add new patch with a test for booting Ubuntu 24.04
> 
> Simon Glass (28):
>    x86: Expand x86_64 early memory
>    x86: qemu: Switch to bochs display
>    x86: qemu: Enable dhrystone
>    x86: qemu: Avoid accessing BSS too early
>    x86: Drop mpspec from the SPL build
>    x86: Add some log categories
>    x86: Drop use of CONFIG_REALMODE_DEBUG
>    x86: Avoid clearing the VESA display
>    x86: Add 64-bit entries to the GDT
>    x86: Use defines for the cache flags
>    x86: spl: Drop duplicate CPU init
>    x86: Drop the message about features missing in 64-bit
>    x86: Include stdbool.h in interrupt header
>    x86: Tidy up the GDT size in start/16.S
>    x86: Disable paging before changing to long mode
>    x86: Use the same GDT when jumping to long mode
>    x86: Use a simple jump into long mode
>    x86: Rename the _D dirty flag
>    x86: Support CPU functions in long mode
>    x86: Tidy up address size in MTRR calculations
>    Add a 64-bit version of is_power_of_2()
>    x86: Support MTRRs of 4GB on 32-bit machines
>    x86: emulation: Set an MTRR for the RAM
>    scripts: Expand a few options
>    test/py: Add a helper to send characters
>    test/py: Allow tests to be filtered by role
>    RFC: test/py: Deal with timeouts
>    test: Add a test for booting Ubuntu 24.04
> 
>   .gitlab-ci.yml                   |  5 +++
>   arch/x86/cpu/cpu.c               | 24 +++++++++++++++
>   arch/x86/cpu/i386/call64.S       | 35 +++++++++------------
>   arch/x86/cpu/i386/cpu.c          | 41 +++++++++---------------
>   arch/x86/cpu/mtrr.c              | 12 +++++---
>   arch/x86/cpu/qemu/dram.c         | 15 +++++++++
>   arch/x86/cpu/qemu/qemu.c         | 20 ++++++++----
>   arch/x86/cpu/start.S             |  4 ++-
>   arch/x86/cpu/start16.S           |  3 +-
>   arch/x86/cpu/x86_64/cpu.c        |  5 ---
>   arch/x86/include/asm/cpu.h       | 35 ++++++++++++++++++---
>   arch/x86/include/asm/interrupt.h |  1 +
>   arch/x86/include/asm/processor.h |  5 ++-
>   arch/x86/lib/Makefile            |  2 ++
>   arch/x86/lib/bios.c              | 27 +++++++++-------
>   arch/x86/lib/bios_interrupts.c   |  8 ++---
>   arch/x86/lib/i8259.c             |  2 ++
>   arch/x86/lib/spl.c               |  4 +--
>   configs/qemu-x86_64_defconfig    | 10 +++---
>   configs/qemu-x86_defconfig       |  6 ++--
>   include/linux/log2.h             |  6 ++++
>   scripts/build-qemu.sh            |  4 +--
>   test/py/conftest.py              | 22 +++++++++++++
>   test/py/console_base.py          | 33 ++++++++++++--------
>   test/py/pytest.ini               |  1 +
>   test/py/tests/test_distro.py     | 53 ++++++++++++++++++++++++++++++++
>   test/py/tests/test_sleep.py      |  1 +
>   27 files changed, 273 insertions(+), 111 deletions(-)
>   create mode 100644 test/py/tests/test_distro.py
> 

Re: [PATCH v2 00/28] x86: Improve operation under QEMU

[Simon Glass]

Hi Paolo,

On Sun, 16 Feb 2025 at 14:14, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 2/16/25 21:43, Simon Glass wrote:
> > U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
> > is not perfect.
> >
> > With both builds, executing the VESA ROM causes an intermittent hang, at
> > least on some AMD CPUs.
> >
> > With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
> > is done in a way that works on real hardware but not with QEMU. This
> > means that performance is 4-5x slower than it could be, at least on my
> > CPU.
>
> Do you have a reproducer for this?  It shouldn't happen... also, do you
> know if this is broken for both Intel and AMD CPUs?

You can see this if you build the right commit (e52e307c7ca) from my
tree[1], i.e. after U-Boot changes to Bochs and before it changes the
64-bit entry mechanism:

04:57 $ pe
e52e307c7ca (HEAD) x86: qemu: Switch to bochs display
83f8b00df47 x86: Expand x86_64 early memory
71a6c3eb3d8 Revert "efi_memory: do not add U-Boot memory to the memory map"
d9d87e11e97 test: Make net tests depend on CONFIG_CMD_NET
3c67e62b109 test/py: Show info about module-loading
7059d976d22 test/py: Drop assigning ubman to cons
e361abfc638 test/py: Drop importing utils as util
460ecb6e5cd test/py: Drop u_boot_ prefix on test files
cc5600e734c test/py: Shorten u_boot_console
777c7a0c20e test: Update documentation
✔ ~/u [:e52e307c7ca|…6⚑ 1160]
04:57 $ ./tools/buildman/buildman --bo qemu-x86_64 -o /tmp/b/qemu-x86_64/ -w
Building current source for 1 boards (1 thread, 32 jobs per thread)
    1    0    0 /1              qemu-x86_64
Completed: 1 total built, 1 newly), duration 0:00:02, rate 0.50
✔ ~/u [:e52e307c7ca|…6⚑ 1160]
04:57 $ qemu-system-x86_64 -bios /tmp/b/qemu-x86_64/u-boot.rom
-enable-kvm    -nographic
Running qemu-system-x86_64  -display none -serial mon:stdio

U-Boot SPL 2025.01-rc3-00635-ge52e307c7ca8 (Feb 17 2025 - 04:57:59 -0700)
Trying to boot from SPI
Jumping to 64-bit U-Boot: Note many features are missing
(hangs here)
QEMU: Terminated

I am running this on an AMD system. But I just tried it on an Intel
laptop and it has the same issue.
>
> Thanks,
>
> Paolo
>
Regards,
Simon

> > We can work around the first problem by using Bochs, which is anyway a
> > better choice than VESA for QEMU. The second can be addressed by using
> > the same descriptor across the jump to long mode.
> >
> > With an MTRR fix this allows booting into Ubuntu on qemu-x86_64
> >
> > Changes in v2:
> > - Redo commit message
> > - Add new patch to rename the _D dirty flag
> > - Add new patch to support CPU functions in long mode
> > - Add new patch to tidy up address size in MTRR calculations
> > - Add new patch with a 64-bit version of is_power_of_2()
> > - Add new patch to set an MTRR for the RAM in QEMU
> > - Add new patch with a helper to send characters
> > - Add new patch to allow tests to be filtered by role
> > - Add more patches to support booting with kvm
> > - Add new patch with a test for booting Ubuntu 24.04
> >
> > Simon Glass (28):
> >    x86: Expand x86_64 early memory
> >    x86: qemu: Switch to bochs display
> >    x86: qemu: Enable dhrystone
> >    x86: qemu: Avoid accessing BSS too early
> >    x86: Drop mpspec from the SPL build
> >    x86: Add some log categories
> >    x86: Drop use of CONFIG_REALMODE_DEBUG
> >    x86: Avoid clearing the VESA display
> >    x86: Add 64-bit entries to the GDT
> >    x86: Use defines for the cache flags
> >    x86: spl: Drop duplicate CPU init
> >    x86: Drop the message about features missing in 64-bit
> >    x86: Include stdbool.h in interrupt header
> >    x86: Tidy up the GDT size in start/16.S
> >    x86: Disable paging before changing to long mode
> >    x86: Use the same GDT when jumping to long mode
> >    x86: Use a simple jump into long mode
> >    x86: Rename the _D dirty flag
> >    x86: Support CPU functions in long mode
> >    x86: Tidy up address size in MTRR calculations
> >    Add a 64-bit version of is_power_of_2()
> >    x86: Support MTRRs of 4GB on 32-bit machines
> >    x86: emulation: Set an MTRR for the RAM
> >    scripts: Expand a few options
> >    test/py: Add a helper to send characters
> >    test/py: Allow tests to be filtered by role
> >    RFC: test/py: Deal with timeouts
> >    test: Add a test for booting Ubuntu 24.04
> >
> >   .gitlab-ci.yml                   |  5 +++
> >   arch/x86/cpu/cpu.c               | 24 +++++++++++++++
> >   arch/x86/cpu/i386/call64.S       | 35 +++++++++------------
> >   arch/x86/cpu/i386/cpu.c          | 41 +++++++++---------------
> >   arch/x86/cpu/mtrr.c              | 12 +++++---
> >   arch/x86/cpu/qemu/dram.c         | 15 +++++++++
> >   arch/x86/cpu/qemu/qemu.c         | 20 ++++++++----
> >   arch/x86/cpu/start.S             |  4 ++-
> >   arch/x86/cpu/start16.S           |  3 +-
> >   arch/x86/cpu/x86_64/cpu.c        |  5 ---
> >   arch/x86/include/asm/cpu.h       | 35 ++++++++++++++++++---
> >   arch/x86/include/asm/interrupt.h |  1 +
> >   arch/x86/include/asm/processor.h |  5 ++-
> >   arch/x86/lib/Makefile            |  2 ++
> >   arch/x86/lib/bios.c              | 27 +++++++++-------
> >   arch/x86/lib/bios_interrupts.c   |  8 ++---
> >   arch/x86/lib/i8259.c             |  2 ++
> >   arch/x86/lib/spl.c               |  4 +--
> >   configs/qemu-x86_64_defconfig    | 10 +++---
> >   configs/qemu-x86_defconfig       |  6 ++--
> >   include/linux/log2.h             |  6 ++++
> >   scripts/build-qemu.sh            |  4 +--
> >   test/py/conftest.py              | 22 +++++++++++++
> >   test/py/console_base.py          | 33 ++++++++++++--------
> >   test/py/pytest.ini               |  1 +
> >   test/py/tests/test_distro.py     | 53 ++++++++++++++++++++++++++++++++
> >   test/py/tests/test_sleep.py      |  1 +
> >   27 files changed, 273 insertions(+), 111 deletions(-)
> >   create mode 100644 test/py/tests/test_distro.py
> >
>

[1] https://ci.u-boot.org/u-boot/u-boot/-/commits/qemu?ref_type=heads

Re: [PATCH v2 00/28] x86: Improve operation under QEMU

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Sun, Feb 16, 2025 at 01:43:45PM -0700, Simon Glass wrote:

> U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
> is not perfect.
> 
> With both builds, executing the VESA ROM causes an intermittent hang, at
> least on some AMD CPUs.
> 
> With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
> is done in a way that works on real hardware but not with QEMU. This
> means that performance is 4-5x slower than it could be, at least on my
> CPU.
> 
> We can work around the first problem by using Bochs, which is anyway a
> better choice than VESA for QEMU. The second can be addressed by using
> the same descriptor across the jump to long mode.
> 
> With an MTRR fix this allows booting into Ubuntu on qemu-x86_64

This seems needlessly not against mainline.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2 00/28] x86: Improve operation under QEMU

[Simon Glass]

Hi Tom,

On Sun, 16 Feb 2025 at 14:57, Tom Rini <trini@konsulko.com> wrote:
>
> On Sun, Feb 16, 2025 at 01:43:45PM -0700, Simon Glass wrote:
>
> > U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
> > is not perfect.
> >
> > With both builds, executing the VESA ROM causes an intermittent hang, at
> > least on some AMD CPUs.
> >
> > With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
> > is done in a way that works on real hardware but not with QEMU. This
> > means that performance is 4-5x slower than it could be, at least on my
> > CPU.
> >
> > We can work around the first problem by using Bochs, which is anyway a
> > better choice than VESA for QEMU. The second can be addressed by using
> > the same descriptor across the jump to long mode.
> >
> > With an MTRR fix this allows booting into Ubuntu on qemu-x86_64
>
> This seems needlessly not against mainline.

I just tried it and yes there is a very small delta. I can resend it
rebased to -next if you like. I'd really like to get some OS-booting
tests into CI.

Regards,
Simon

Re: [PATCH v2 00/28] x86: Improve operation under QEMU

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Mon, Feb 17, 2025 at 06:14:12AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Sun, 16 Feb 2025 at 14:57, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Sun, Feb 16, 2025 at 01:43:45PM -0700, Simon Glass wrote:
> >
> > > U-Boot can start and boot an OS in both qemu-x86 and qemu-x86_64 but it
> > > is not perfect.
> > >
> > > With both builds, executing the VESA ROM causes an intermittent hang, at
> > > least on some AMD CPUs.
> > >
> > > With qemu-x86_64 kvm cannot be used since the move to long mode (64-bit)
> > > is done in a way that works on real hardware but not with QEMU. This
> > > means that performance is 4-5x slower than it could be, at least on my
> > > CPU.
> > >
> > > We can work around the first problem by using Bochs, which is anyway a
> > > better choice than VESA for QEMU. The second can be addressed by using
> > > the same descriptor across the jump to long mode.
> > >
> > > With an MTRR fix this allows booting into Ubuntu on qemu-x86_64
> >
> > This seems needlessly not against mainline.
> 
> I just tried it and yes there is a very small delta. I can resend it
> rebased to -next if you like. I'd really like to get some OS-booting
> tests into CI.

Sure, and please start pushing scripts to u-boot-extras. Having these
scripts in u-boot itself explicitly makes it harder to use them for
debug as you now rely on them being within the tree with the changes you
want on whatever older commit you want. And since you want to move these
to Python too you that also means language updates need to be in there
as well. Thanks.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]