[PATCH v5 00/44] Boot modules for Hyperlaunch

[PATCH v5 00/44] Boot modules for Hyperlaunch

[Daniel P. Smith]

The Boot Modules for Hyperlaunch series is an effort to split out preliminary
changes necessary for the introduction of the Hyperlaunch domain builder
logic. These preliminary changes revolve around introducing the struct
boot_module and struct boot_domain structures. This includes converting the
dom0 construction path to use these structures. These abstractions lay the
groundwork to transform and extend the dom0 construction logic into a limited,
but general domain builder.

The splitting of Hyperlaunch into a set of series are twofold, to reduce the
effort in reviewing a much larger series, and to reduce the effort in handling
the knock-on effects to the construction logic from requested review changes.

Much thanks to AMD for supporting this work.

Documentation on Hyperlaunch:
https://wiki.xenproject.org/wiki/Hyperlaunch

Original Hyperlaunch v1 patch series:
https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00345.html

V/r,
Daniel P. Smith

Changes since v4:
- added requested inline code comments
- moved instance of struct boot_info to unit level and extern'ed
- array of struct boot_module moved into struct boot_info
- renamed function to multiboot_fill_bootinfo, now returns *struct boot_info
- multiboot_fill_bootinfo changed to take multiboot_info_t addr as param
- added missing guard that checked there were multiboot1/2 modules passed
- renmaed struct elements per the review
- fixed errant commit messages per the review
- corrected coding style per review
- attempted to repalce all open codings of page/addr translations touched 
- unified use of `bi` as var name for pointer ref to struct boot_info
- when appropriate, ensure variables where typed, eg size_t, paddr_t, etc.
- dropped all uses of "a = b = c"

Changes since v3:
- reduced scope to x86 only
- broke changes into a smaller chunks with a linear progression
- concerns about deconflicting with Arm deferred
- conversion from mb1 to boot modules no longer attempted at entry points
- the temporary conversion function is now the permenant means to convert
- incorporated suggestion from Andy Cooper for handling bootstrap_map

Changes since v2:
- combined v2 patches 7 and 8 for common review
- rebased the v2 series onto the current tip of staging (sorry)
- fixed the placement of the patch changelogs
- provided the changes description in the cover letter

Changes since v1:
- the v2 and v3 series implement functionality from v1 patches 2-4
    - v2 series objective is to enable efficient patch review in support
      of merging the functionality into the hypervisor. It implements a
      subset of the v1 series, incorporating changes from community
      feedback.
- the bootstrap map is made accessible early in the v2 series via both
  multiboot and boot module arguments until later in the series where
  multiboot use is retired. This allows for incremental conversion across
  several patches from multiboot to boot modules.
- the 32-bit x86 boot environment header is removed, and changes are
  made to allow the new common bootinfo headers to be used instead.
- Arm and RISC-V architecture bootinfo headers are added to ensure that
  builds on those architectures can complete correctly.
- The KConfig patch to set the maximum number of boot modules allowed
  is not included in this series, replaced with a static maximum define.

Andrew Cooper (1):
  x86/boot: split bootstrap_map_addr() out of bootstrap_map()

Christopher Clark (1):
  x86/boot: move x86 boot module counting into a new boot_info struct

Daniel P. Smith (42):
  x86/boot: move boot loader name to boot info
  x86/boot: move cmdline to boot info
  x86/boot: move mmap info to boot info
  x86/boot: introduce struct boot_module
  x86/boot: convert consider_modules to struct boot_module
  x86/boot: move headroom to boot modules
  x86/boot: convert setup.c mod refs to early_mod
  x86/boot: introduce boot module types
  x86/boot: introduce boot module flags
  x86/boot: add start and size fields to struct boot_module
  x86/boot: update struct boot_module on module relocation
  x86/boot: transition relocation calculations to struct boot_module
  x86/boot: introduce boot module interator
  x86/boot: introduce consumed flag for struct boot_module
  x86/boot: convert microcode loading to consume struct boot_info
  x86/boot: convert late microcode loading to struct boot_module
  x86/boot: use consumed boot module flag for microcode
  x86/boot: convert xsm policy loading to struct boot_module
  x86/boot: convert ramdisk locating to struct boot_module
  x86/boot: remove module_map usage from microcode loading
  x86/boot: remove module_map usage from xsm policy loading
  x86/boot: remove module_map usage by ramdisk loading
  x86/boot: convert create_dom0 to use boot info
  x86/boot: convert construct_dom0 to use struct boot_module
  x86/boot: relocate kextra into boot info
  x86/boot: add cmdline to struct boot_module
  x86/boot: convert dom0_construct_pv image param to struct boot_module
  x86/boot: convert dom0_construct_pv initrd param to struct boot_module
  x86/boot: convert dom0_construct_pvh to struct boot_module
  x86/boot: convert pvh_load_kernel to struct boot_module
  x86/boot: convert initial_images to struct boot_module
  x86/boot: drop the use of initial_images unit global
  x86/boot: remove usage of mod_end by discard_initial_images
  x86/boot: remove remaining early_mod references
  x86/boot: remove mod from struct boot_module
  x86/boot: introduce boot domain
  x86/boot: introduce domid field to struct boot_domain
  x86/boot: add cmdline to struct boot_domain
  x86/boot: add struct domain to struct boot_domain
  x86/boot: convert construct_dom0 to struct boot_domain
  x86/boot: convert dom0_construct_pv to struct boot_domain
  x86/boot: convert dom0_construct_pvh to struct boot_domain

 xen/arch/x86/cpu/microcode/core.c     |  78 +++----
 xen/arch/x86/dom0_build.c             |  21 +-
 xen/arch/x86/hvm/dom0_build.c         |  55 +++--
 xen/arch/x86/include/asm/bootdomain.h |  37 +++
 xen/arch/x86/include/asm/bootinfo.h   |  91 ++++++++
 xen/arch/x86/include/asm/dom0_build.h |  11 +-
 xen/arch/x86/include/asm/microcode.h  |  12 +-
 xen/arch/x86/include/asm/setup.h      |  11 +-
 xen/arch/x86/pv/dom0_build.c          |  54 ++---
 xen/arch/x86/setup.c                  | 321 ++++++++++++++++----------
 xen/include/xsm/xsm.h                 |  14 +-
 xen/xsm/xsm_core.c                    |  15 +-
 xen/xsm/xsm_policy.c                  |  18 +-
 13 files changed, 461 insertions(+), 277 deletions(-)
 create mode 100644 xen/arch/x86/include/asm/bootdomain.h
 create mode 100644 xen/arch/x86/include/asm/bootinfo.h

-- 
2.30.2

[PATCH v5 01/44] x86/boot: move x86 boot module counting into a new boot_info struct

[Daniel P. Smith]

From: Christopher Clark <christopher.w.clark@gmail.com>

An initial step towards a non-multiboot internal representation of boot
modules for common code, starting with x86 setup and converting the fields
that are accessed for the startup calculations.

Introduce a new header, <asm/bootinfo.h>, and populate it with a new boot_info
structure initially containing a count of the number of boot modules.

No functional change intended.

Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 29 +++++++++++++
 xen/arch/x86/include/asm/setup.h    |  2 +
 xen/arch/x86/setup.c                | 64 ++++++++++++++++++-----------
 3 files changed, 71 insertions(+), 24 deletions(-)
 create mode 100644 xen/arch/x86/include/asm/bootinfo.h

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
new file mode 100644
index 000000000000..a649500ee3a2
--- /dev/null
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
+ * Copyright (c) 2024 Apertus Solutions, LLC
+ * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
+ */
+
+#ifndef __XEN_X86_BOOTINFO_H__
+#define __XEN_X86_BOOTINFO_H__
+
+/*
+ * Xen internal representation of information provided by the
+ * bootloader/environment, or derived from the information.
+ */
+struct boot_info {
+    unsigned int nr_modules;
+};
+
+#endif /* __XEN_X86_BOOTINFO_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index d75589178b91..3d189521189d 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -32,6 +32,8 @@ int construct_dom0(
     const char *cmdline);
 void setup_io_bitmap(struct domain *d);
 
+extern struct boot_info xen_boot_info;
+
 unsigned long initial_images_nrpages(nodeid_t node);
 void discard_initial_images(void);
 void *bootstrap_map(const module_t *mod);
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index a6e77c9ed9fc..b75deb4fe4ee 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -32,6 +32,7 @@
 #include <compat/xen.h>
 #endif
 #include <xen/bitops.h>
+#include <asm/bootinfo.h>
 #include <asm/smp.h>
 #include <asm/processor.h>
 #include <asm/mpspec.h>
@@ -274,16 +275,28 @@ static int __init cf_check parse_acpi_param(const char *s)
 custom_param("acpi", parse_acpi_param);
 
 static const module_t *__initdata initial_images;
-static unsigned int __initdata nr_initial_images;
+
+struct boot_info __initdata xen_boot_info;
+
+static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
+{
+    struct boot_info *bi = &xen_boot_info;
+    const multiboot_info_t *mbi = __va(mbi_p);
+
+    bi->nr_modules = (mbi->flags & MBI_MODULES) ? mbi->mods_count : 0;
+
+    return bi;
+}
 
 unsigned long __init initial_images_nrpages(nodeid_t node)
 {
+    struct boot_info *bi = &xen_boot_info;
     unsigned long node_start = node_start_pfn(node);
     unsigned long node_end = node_end_pfn(node);
     unsigned long nr;
     unsigned int i;
 
-    for ( nr = i = 0; i < nr_initial_images; ++i )
+    for ( nr = i = 0; i < bi->nr_modules; ++i )
     {
         unsigned long start = initial_images[i].mod_start;
         unsigned long end = start + PFN_UP(initial_images[i].mod_end);
@@ -297,9 +310,10 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
 
 void __init discard_initial_images(void)
 {
+    struct boot_info *bi = &xen_boot_info;
     unsigned int i;
 
-    for ( i = 0; i < nr_initial_images; ++i )
+    for ( i = 0; i < bi->nr_modules; ++i )
     {
         uint64_t start = (uint64_t)initial_images[i].mod_start << PAGE_SHIFT;
 
@@ -307,7 +321,7 @@ void __init discard_initial_images(void)
                            start + PAGE_ALIGN(initial_images[i].mod_end));
     }
 
-    nr_initial_images = 0;
+    bi->nr_modules = 0;
     initial_images = NULL;
 }
 
@@ -969,6 +983,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     void *bsp_stack;
     struct cpu_info *info = get_cpu_info(), *bsp_info;
     unsigned int initrdidx, num_parked = 0;
+    struct boot_info *bi;
     multiboot_info_t *mbi;
     module_t *mod;
     unsigned long nr_pages, raw_max_page, modules_headroom, module_map[1];
@@ -1015,6 +1030,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         mod = __va(mbi->mods_addr);
     }
 
+    bi = multiboot_fill_boot_info(mbi_p);
+
     loader = (mbi->flags & MBI_LOADERNAME) ? __va(mbi->boot_loader_name)
                                            : "unknown";
 
@@ -1122,18 +1139,18 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
            bootsym(boot_edd_info_nr));
 
     /* Check that we have at least one Multiboot module. */
-    if ( !(mbi->flags & MBI_MODULES) || (mbi->mods_count == 0) )
+    if ( !bi->nr_modules )
         panic("dom0 kernel not specified. Check bootloader configuration\n");
 
     /* Check that we don't have a silly number of modules. */
-    if ( mbi->mods_count > sizeof(module_map) * 8 )
+    if ( bi->nr_modules > sizeof(module_map) * 8 )
     {
-        mbi->mods_count = sizeof(module_map) * 8;
-        printk("Excessive multiboot modules - using the first %u only\n",
-               mbi->mods_count);
+        bi->nr_modules = sizeof(module_map) * 8;
+        printk("Excessive boot modules - using the first %u only\n",
+               bi->nr_modules);
     }
 
-    bitmap_fill(module_map, mbi->mods_count);
+    bitmap_fill(module_map, bi->nr_modules);
     __clear_bit(0, module_map); /* Dom0 kernel is always first */
 
     if ( pvh_boot )
@@ -1306,9 +1323,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     kexec_reserve_area();
 
     initial_images = mod;
-    nr_initial_images = mbi->mods_count;
 
-    for ( i = 0; !efi_enabled(EFI_LOADER) && i < mbi->mods_count; i++ )
+    for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
     {
         if ( mod[i].mod_start & (PAGE_SIZE - 1) )
             panic("Bootloader didn't honor module alignment request\n");
@@ -1332,8 +1348,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
          * respective reserve_e820_ram() invocation below. No need to
          * query efi_boot_mem_unused() here, though.
          */
-        mod[mbi->mods_count].mod_start = virt_to_mfn(_stext);
-        mod[mbi->mods_count].mod_end = __2M_rwdata_end - _stext;
+        mod[bi->nr_modules].mod_start = virt_to_mfn(_stext);
+        mod[bi->nr_modules].mod_end = __2M_rwdata_end - _stext;
     }
 
     modules_headroom = bzimage_headroom(bootstrap_map(mod), mod->mod_end);
@@ -1393,7 +1409,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         {
             /* Don't overlap with modules. */
             end = consider_modules(s, e, reloc_size + mask,
-                                   mod, mbi->mods_count, -1);
+                                   mod, bi->nr_modules, -1);
             end &= ~mask;
         }
         else
@@ -1414,7 +1430,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         }
 
         /* Is the region suitable for relocating the multiboot modules? */
-        for ( j = mbi->mods_count - 1; j >= 0; j-- )
+        for ( j = bi->nr_modules - 1; j >= 0; j-- )
         {
             /*
              * 'headroom' is a guess for the decompressed size and
@@ -1429,7 +1445,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
             /* Don't overlap with other modules (or Xen itself). */
             end = consider_modules(s, e, size, mod,
-                                   mbi->mods_count + relocated, j);
+                                   bi->nr_modules + relocated, j);
 
             if ( highmem_start && end > highmem_start )
                 continue;
@@ -1456,7 +1472,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         {
             /* Don't overlap with modules (or Xen itself). */
             e = consider_modules(s, e, PAGE_ALIGN(kexec_crash_area.size), mod,
-                                 mbi->mods_count + relocated, -1);
+                                 bi->nr_modules + relocated, -1);
             if ( s >= e )
                 break;
             if ( e > kexec_crash_area_limit )
@@ -1471,7 +1487,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     if ( modules_headroom && !mod->reserved )
         panic("Not enough memory to relocate the dom0 kernel image\n");
-    for ( i = 0; i < mbi->mods_count; ++i )
+    for ( i = 0; i < bi->nr_modules; ++i )
     {
         uint64_t s = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
 
@@ -1551,7 +1567,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                     ASSERT(j);
                 }
                 map_e = boot_e820.map[j].addr + boot_e820.map[j].size;
-                for ( j = 0; j < mbi->mods_count; ++j )
+                for ( j = 0; j < bi->nr_modules; ++j )
                 {
                     uint64_t end = pfn_to_paddr(mod[j].mod_start) +
                                    mod[j].mod_end;
@@ -1626,7 +1642,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         }
     }
 
-    for ( i = 0; i < mbi->mods_count; ++i )
+    for ( i = 0; i < bi->nr_modules; ++i )
     {
         set_pdx_range(mod[i].mod_start,
                       mod[i].mod_start + PFN_UP(mod[i].mod_end));
@@ -2011,8 +2027,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
            cpu_has_nx ? XENLOG_INFO : XENLOG_WARNING "Warning: ",
            cpu_has_nx ? "" : "not ");
 
-    initrdidx = find_first_bit(module_map, mbi->mods_count);
-    if ( bitmap_weight(module_map, mbi->mods_count) > 1 )
+    initrdidx = find_first_bit(module_map, bi->nr_modules);
+    if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
         printk(XENLOG_WARNING
                "Multiple initrd candidates, picking module #%u\n",
                initrdidx);
@@ -2022,7 +2038,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
     dom0 = create_dom0(mod, modules_headroom,
-                       initrdidx < mbi->mods_count ? mod + initrdidx : NULL,
+                       initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
                        kextra, loader);
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
-- 
2.30.2

[PATCH v5 02/44] x86/boot: move boot loader name to boot info

[Daniel P. Smith]

Transition the incoming boot loader name to be held in struct boot_info.

No functional change intended.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  2 ++
 xen/arch/x86/setup.c                | 16 ++++++++--------
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index a649500ee3a2..98ba773c9bc5 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -14,6 +14,8 @@
  */
 struct boot_info {
     unsigned int nr_modules;
+
+    const char *loader;
 };
 
 #endif /* __XEN_X86_BOOTINFO_H__ */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index b75deb4fe4ee..aafc098ca268 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -285,6 +285,9 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
 
     bi->nr_modules = (mbi->flags & MBI_MODULES) ? mbi->mods_count : 0;
 
+    bi->loader = (mbi->flags & MBI_LOADERNAME) ?
+                            __va(mbi->boot_loader_name) : "unknown";
+
     return bi;
 }
 
@@ -978,7 +981,7 @@ static struct domain *__init create_dom0(const module_t *image,
 
 void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 {
-    const char *memmap_type = NULL, *loader, *cmdline = "";
+    const char *memmap_type = NULL, *cmdline = "";
     char *kextra;
     void *bsp_stack;
     struct cpu_info *info = get_cpu_info(), *bsp_info;
@@ -1032,12 +1035,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     bi = multiboot_fill_boot_info(mbi_p);
 
-    loader = (mbi->flags & MBI_LOADERNAME) ? __va(mbi->boot_loader_name)
-                                           : "unknown";
-
     /* Parse the command-line options. */
     if ( mbi->flags & MBI_CMDLINE )
-        cmdline = cmdline_cook(__va(mbi->cmdline), loader);
+        cmdline = cmdline_cook(__va(mbi->cmdline), bi->loader);
 
     if ( (kextra = strstr(cmdline, " -- ")) != NULL )
     {
@@ -1078,7 +1078,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     if ( pvh_boot )
         pvh_print_info();
 
-    printk("Bootloader: %s\n", loader);
+    printk("Bootloader: %s\n", bi->loader);
 
     printk("Command line: %s\n", cmdline);
 
@@ -1171,7 +1171,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         l3_bootmap[l3_table_offset(BOOTSTRAP_MAP_BASE)] =
             l3e_from_paddr(__pa(l2_bootmap), __PAGE_HYPERVISOR);
 
-        memmap_type = loader;
+        memmap_type = bi->loader;
     }
     else if ( efi_enabled(EFI_BOOT) )
         memmap_type = "EFI";
@@ -2039,7 +2039,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      */
     dom0 = create_dom0(mod, modules_headroom,
                        initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
-                       kextra, loader);
+                       kextra, bi->loader);
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
 
-- 
2.30.2

[PATCH v5 03/44] x86/boot: move cmdline to boot info

[Daniel P. Smith]

Transition Xen's command line to being held in struct boot_info.

No functional change intended.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  1 +
 xen/arch/x86/setup.c                | 20 ++++++++++++--------
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 98ba773c9bc5..327038465a44 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -16,6 +16,7 @@ struct boot_info {
     unsigned int nr_modules;
 
     const char *loader;
+    const char *cmdline;
 };
 
 #endif /* __XEN_X86_BOOTINFO_H__ */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index aafc098ca268..0921f296075f 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -274,6 +274,8 @@ static int __init cf_check parse_acpi_param(const char *s)
 }
 custom_param("acpi", parse_acpi_param);
 
+static const char *cmdline_cook(const char *p, const char *loader_name);
+
 static const module_t *__initdata initial_images;
 
 struct boot_info __initdata xen_boot_info;
@@ -288,6 +290,12 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     bi->loader = (mbi->flags & MBI_LOADERNAME) ?
                             __va(mbi->boot_loader_name) : "unknown";
 
+    /* Parse the command-line options. */
+    if ( mbi->flags & MBI_CMDLINE )
+        bi->cmdline = cmdline_cook(__va(mbi->cmdline), bi->loader);
+    else
+        bi->cmdline = "";
+
     return bi;
 }
 
@@ -981,7 +989,7 @@ static struct domain *__init create_dom0(const module_t *image,
 
 void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 {
-    const char *memmap_type = NULL, *cmdline = "";
+    const char *memmap_type = NULL;
     char *kextra;
     void *bsp_stack;
     struct cpu_info *info = get_cpu_info(), *bsp_info;
@@ -1035,11 +1043,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     bi = multiboot_fill_boot_info(mbi_p);
 
-    /* Parse the command-line options. */
-    if ( mbi->flags & MBI_CMDLINE )
-        cmdline = cmdline_cook(__va(mbi->cmdline), bi->loader);
-
-    if ( (kextra = strstr(cmdline, " -- ")) != NULL )
+    if ( (kextra = strstr(bi->cmdline, " -- ")) != NULL )
     {
         /*
          * Options after ' -- ' separator belong to dom0.
@@ -1050,7 +1054,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         kextra += 3;
         while ( kextra[1] == ' ' ) kextra++;
     }
-    cmdline_parse(cmdline);
+    cmdline_parse(bi->cmdline);
 
     /* Must be after command line argument parsing and before
      * allocing any xenheap structures wanted in lower memory. */
@@ -1080,7 +1084,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     printk("Bootloader: %s\n", bi->loader);
 
-    printk("Command line: %s\n", cmdline);
+    printk("Command line: %s\n", bi->cmdline);
 
     printk("Xen image load base address: %#lx\n", xen_phys_start);
     if ( hypervisor_name )
-- 
2.30.2

[PATCH v5 04/44] x86/boot: move mmap info to boot info

[Daniel P. Smith]

Transition the memory map info to be held in struct boot_info.

No functional change intended.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  5 +++++
 xen/arch/x86/setup.c                | 12 +++++++++---
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 327038465a44..87d311ac1399 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -8,6 +8,8 @@
 #ifndef __XEN_X86_BOOTINFO_H__
 #define __XEN_X86_BOOTINFO_H__
 
+#include <xen/types.h>
+
 /*
  * Xen internal representation of information provided by the
  * bootloader/environment, or derived from the information.
@@ -17,6 +19,9 @@ struct boot_info {
 
     const char *loader;
     const char *cmdline;
+
+    paddr_t memmap_addr;
+    size_t memmap_length;
 };
 
 #endif /* __XEN_X86_BOOTINFO_H__ */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 0921f296075f..f0482ca8cc55 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -296,6 +296,12 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     else
         bi->cmdline = "";
 
+    if ( mbi->flags & MBI_MEMMAP )
+    {
+        bi->memmap_addr = mbi->mmap_addr;
+        bi->memmap_length = mbi->mmap_length;
+    }
+
     return bi;
 }
 
@@ -1185,13 +1191,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     {
         memmap_type = "Xen-e820";
     }
-    else if ( mbi->flags & MBI_MEMMAP )
+    else if ( bi->memmap_addr )
     {
         memmap_type = "Multiboot-e820";
-        while ( bytes < mbi->mmap_length &&
+        while ( bytes < bi->memmap_length &&
                 e820_raw.nr_map < ARRAY_SIZE(e820_raw.map) )
         {
-            memory_map_t *map = __va(mbi->mmap_addr + bytes);
+            memory_map_t *map = __va(bi->memmap_addr + bytes);
 
             /*
              * This is a gross workaround for a BIOS bug. Some bootloaders do
-- 
2.30.2

[PATCH v5 05/44] x86/boot: introduce struct boot_module

[Daniel P. Smith]

This will introduce a new struct boot_module to provide a rich state
representation around modules provided by the boot loader. Support is for 64
boot modules, one held in reserve for Xen, and up to 63 can be provided by the
boot loader. The array of struct boot_modules will be accessible via a
reference held in struct boot_info.

A temporary `mod` parameter is included in struct boot_module to ease the
transition from using Multiboot v1 structures over to struct boot_module. Once
the transition is complete, the parameter will be dropped from the structure.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 14 ++++++++++++--
 xen/arch/x86/setup.c                |  9 +++++++++
 2 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 87d311ac1399..d19473d8941e 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -8,20 +8,30 @@
 #ifndef __XEN_X86_BOOTINFO_H__
 #define __XEN_X86_BOOTINFO_H__
 
+#include <xen/multiboot.h>
 #include <xen/types.h>
 
+/* Max number of boot modules a bootloader can provide in addition to Xen */
+#define MAX_NR_BOOTMODS 63
+
+struct boot_module {
+    /* Transitionary only */
+    module_t *mod;
+};
+
 /*
  * Xen internal representation of information provided by the
  * bootloader/environment, or derived from the information.
  */
 struct boot_info {
-    unsigned int nr_modules;
-
     const char *loader;
     const char *cmdline;
 
     paddr_t memmap_addr;
     size_t memmap_length;
+
+    unsigned int nr_modules;
+    struct boot_module mods[MAX_NR_BOOTMODS + 1];
 };
 
 #endif /* __XEN_X86_BOOTINFO_H__ */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index f0482ca8cc55..68eb4c848ae8 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -284,6 +284,8 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
 {
     struct boot_info *bi = &xen_boot_info;
     const multiboot_info_t *mbi = __va(mbi_p);
+    module_t *mods = __va(mbi->mods_addr);
+    unsigned int i;
 
     bi->nr_modules = (mbi->flags & MBI_MODULES) ? mbi->mods_count : 0;
 
@@ -302,6 +304,13 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
         bi->memmap_length = mbi->mmap_length;
     }
 
+    /*
+     * This will iterate over all modules to include an extra mb module, which
+     * should have been reserved to hold an entry for Xen.
+     */
+    for ( i = 0; i <= bi->nr_modules; i++ )
+        bi->mods[i].mod = &mods[i];
+
     return bi;
 }
 
-- 
2.30.2

[PATCH v5 06/44] x86/boot: convert consider_modules to struct boot_module

[Daniel P. Smith]

To start transitioning consider_modules() over to struct boot_module, begin
with taking the array of struct boot_modules but use the temporary struct
element mod.

No functional change intended.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 68eb4c848ae8..ba9f110d98c6 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -629,7 +629,7 @@ static void __init noinline move_xen(void)
 #undef BOOTSTRAP_MAP_LIMIT
 
 static uint64_t __init consider_modules(
-    uint64_t s, uint64_t e, uint32_t size, const module_t *mod,
+    uint64_t s, uint64_t e, uint32_t size, const struct boot_module mods[],
     unsigned int nr_mods, unsigned int this_mod)
 {
     unsigned int i;
@@ -639,20 +639,20 @@ static uint64_t __init consider_modules(
 
     for ( i = 0; i < nr_mods ; ++i )
     {
-        uint64_t start = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
-        uint64_t end = start + PAGE_ALIGN(mod[i].mod_end);
+        uint64_t start = (uint64_t)pfn_to_paddr(mods[i].mod->mod_start);
+        uint64_t end = start + PAGE_ALIGN(mods[i].mod->mod_end);
 
         if ( i == this_mod )
             continue;
 
         if ( s < end && start < e )
         {
-            end = consider_modules(end, e, size, mod + i + 1,
+            end = consider_modules(end, e, size, &mods[i + 1],
                                    nr_mods - i - 1, this_mod - i - 1);
             if ( end )
                 return end;
 
-            return consider_modules(s, start, size, mod + i + 1,
+            return consider_modules(s, start, size, &mods[i + 1],
                                     nr_mods - i - 1, this_mod - i - 1);
         }
     }
@@ -1428,7 +1428,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         {
             /* Don't overlap with modules. */
             end = consider_modules(s, e, reloc_size + mask,
-                                   mod, bi->nr_modules, -1);
+                                   bi->mods, bi->nr_modules, -1);
             end &= ~mask;
         }
         else
@@ -1463,7 +1463,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                 continue;
 
             /* Don't overlap with other modules (or Xen itself). */
-            end = consider_modules(s, e, size, mod,
+            end = consider_modules(s, e, size, bi->mods,
                                    bi->nr_modules + relocated, j);
 
             if ( highmem_start && end > highmem_start )
@@ -1490,7 +1490,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         while ( !kexec_crash_area.start )
         {
             /* Don't overlap with modules (or Xen itself). */
-            e = consider_modules(s, e, PAGE_ALIGN(kexec_crash_area.size), mod,
+            e = consider_modules(s, e, PAGE_ALIGN(kexec_crash_area.size), bi->mods,
                                  bi->nr_modules + relocated, -1);
             if ( s >= e )
                 break;
-- 
2.30.2

[PATCH v5 07/44] x86/boot: move headroom to boot modules

[Daniel P. Smith]

The purpose of struct boot_module is to encapsulate the state of boot module as
it is processed by Xen. Locating boot module state struct boot_module reduces
the number of global variables as well as the number of state variables that
must be passed around. It also lays the groundwork for hyperlaunch mult-domain
construction, where multiple instances of state variables like headroom will be
needed.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  5 +++++
 xen/arch/x86/setup.c                | 23 ++++++++++++++---------
 2 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index d19473d8941e..c7e6b4ebf0da 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -17,6 +17,11 @@
 struct boot_module {
     /* Transitionary only */
     module_t *mod;
+    /*
+     * A boot module may contain a compressed kernel that Xen will need space
+     * reserved, into which it will be decompressed.
+     */
+    unsigned long headroom;
 };
 
 /*
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index ba9f110d98c6..dd82ca3d43e2 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1012,7 +1012,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     struct boot_info *bi;
     multiboot_info_t *mbi;
     module_t *mod;
-    unsigned long nr_pages, raw_max_page, modules_headroom, module_map[1];
+    unsigned long nr_pages, raw_max_page, module_map[1];
     int i, j, e820_warn = 0, bytes = 0;
     unsigned long eb_start, eb_end;
     bool acpi_boot_table_init_done = false, relocated = false;
@@ -1371,7 +1371,10 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         mod[bi->nr_modules].mod_end = __2M_rwdata_end - _stext;
     }
 
-    modules_headroom = bzimage_headroom(bootstrap_map(mod), mod->mod_end);
+    bi->mods[0].headroom =
+        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
+                         bi->mods[0].mod->mod_end);
+
     bootstrap_map(NULL);
 
 #ifndef highmem_start
@@ -1456,8 +1459,10 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
              * decompressor overheads of mod[0] (the dom0 kernel).  When we
              * move mod[0], we incorporate this as extra space at the start.
              */
-            unsigned long headroom = j ? 0 : modules_headroom;
-            unsigned long size = PAGE_ALIGN(headroom + mod[j].mod_end);
+            struct boot_module *bm = &bi->mods[j];
+            unsigned long size;
+
+            size = PAGE_ALIGN(bm->headroom + mod[j].mod_end);
 
             if ( mod[j].reserved )
                 continue;
@@ -1470,14 +1475,14 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                 continue;
 
             if ( s < end &&
-                 (headroom ||
+                 (bm->headroom ||
                   ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) )
             {
-                move_memory(end - size + headroom,
+                move_memory(end - size + bm->headroom,
                             (uint64_t)mod[j].mod_start << PAGE_SHIFT,
                             mod[j].mod_end);
                 mod[j].mod_start = (end - size) >> PAGE_SHIFT;
-                mod[j].mod_end += headroom;
+                mod[j].mod_end += bm->headroom;
                 mod[j].reserved = 1;
             }
         }
@@ -1504,7 +1509,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 #endif
     }
 
-    if ( modules_headroom && !mod->reserved )
+    if ( bi->mods[0].headroom && !mod->reserved )
         panic("Not enough memory to relocate the dom0 kernel image\n");
     for ( i = 0; i < bi->nr_modules; ++i )
     {
@@ -2056,7 +2061,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
-    dom0 = create_dom0(mod, modules_headroom,
+    dom0 = create_dom0(mod, bi->mods[0].headroom,
                        initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
                        kextra, bi->loader);
     if ( !dom0 )
-- 
2.30.2

[PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Daniel P. Smith]

To allow a slow conversion of x86 over to struct boot_module, start with
replacing all references to struct mod to the early_mod element of struct
boot_module. These serves twofold, first to allow the incremental transition
from struct mod fields to struct boot_module fields. The second is to allow
the conversion of function definitions from taking struct mod parameters to
accepting struct boot_module as needed when a transitioned field will be
accessed.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 61 ++++++++++++++++++++++++--------------------
 1 file changed, 34 insertions(+), 27 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index dd82ca3d43e2..ba4bee6b93af 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1341,15 +1341,15 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
     kexec_reserve_area();
 
-    initial_images = mod;
+    initial_images = bi->mods[0].mod;
 
     for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
     {
-        if ( mod[i].mod_start & (PAGE_SIZE - 1) )
+        if ( bi->mods[i].mod->mod_start & (PAGE_SIZE - 1) )
             panic("Bootloader didn't honor module alignment request\n");
-        mod[i].mod_end -= mod[i].mod_start;
-        mod[i].mod_start >>= PAGE_SHIFT;
-        mod[i].reserved = 0;
+        bi->mods[i].mod->mod_end -= bi->mods[i].mod->mod_start;
+        bi->mods[i].mod->mod_start >>= PAGE_SHIFT;
+        bi->mods[i].mod->reserved = 0;
     }
 
     /*
@@ -1360,6 +1360,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     if ( xen_phys_start )
     {
+        unsigned int xen = bi->nr_modules;
         relocated = true;
 
         /*
@@ -1367,8 +1368,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
          * respective reserve_e820_ram() invocation below. No need to
          * query efi_boot_mem_unused() here, though.
          */
-        mod[bi->nr_modules].mod_start = virt_to_mfn(_stext);
-        mod[bi->nr_modules].mod_end = __2M_rwdata_end - _stext;
+        bi->mods[xen].mod->mod_start = virt_to_mfn(_stext);
+        bi->mods[xen].mod->mod_end = __2M_rwdata_end - _stext;
     }
 
     bi->mods[0].headroom =
@@ -1462,9 +1463,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
             struct boot_module *bm = &bi->mods[j];
             unsigned long size;
 
-            size = PAGE_ALIGN(bm->headroom + mod[j].mod_end);
+            size = PAGE_ALIGN(bm->headroom + bm->mod->mod_end);
 
-            if ( mod[j].reserved )
+            if ( bi->mods[j].mod->reserved )
                 continue;
 
             /* Don't overlap with other modules (or Xen itself). */
@@ -1476,14 +1477,14 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
             if ( s < end &&
                  (bm->headroom ||
-                  ((end - size) >> PAGE_SHIFT) > mod[j].mod_start) )
+                  ((end - size) >> PAGE_SHIFT) > bm->mod->mod_start) )
             {
                 move_memory(end - size + bm->headroom,
-                            (uint64_t)mod[j].mod_start << PAGE_SHIFT,
-                            mod[j].mod_end);
-                mod[j].mod_start = (end - size) >> PAGE_SHIFT;
-                mod[j].mod_end += bm->headroom;
-                mod[j].reserved = 1;
+                            (uint64_t)bm->mod->mod_start << PAGE_SHIFT,
+                            bm->mod->mod_end);
+                bm->mod->mod_start = (end - size) >> PAGE_SHIFT;
+                bm->mod->mod_end += bm->headroom;
+                bm->mod->reserved = 1;
             }
         }
 
@@ -1509,13 +1510,15 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 #endif
     }
 
-    if ( bi->mods[0].headroom && !mod->reserved )
+    if ( bi->mods[0].headroom && !bi->mods[0].mod->reserved )
         panic("Not enough memory to relocate the dom0 kernel image\n");
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        uint64_t s = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
+        uint64_t s = (uint64_t)bi->mods[i].mod->mod_start
+                        << PAGE_SHIFT;
 
-        reserve_e820_ram(&boot_e820, s, s + PAGE_ALIGN(mod[i].mod_end));
+        reserve_e820_ram(&boot_e820, s,
+                         s + PAGE_ALIGN(bi->mods[i].mod->mod_end));
     }
 
     if ( !xen_phys_start )
@@ -1593,8 +1596,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                 map_e = boot_e820.map[j].addr + boot_e820.map[j].size;
                 for ( j = 0; j < bi->nr_modules; ++j )
                 {
-                    uint64_t end = pfn_to_paddr(mod[j].mod_start) +
-                                   mod[j].mod_end;
+                    uint64_t end = pfn_to_paddr(
+                                   bi->mods[j].mod->mod_start) +
+                                   bi->mods[j].mod->mod_end;
 
                     if ( map_e < end )
                         map_e = end;
@@ -1668,11 +1672,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        set_pdx_range(mod[i].mod_start,
-                      mod[i].mod_start + PFN_UP(mod[i].mod_end));
-        map_pages_to_xen((unsigned long)mfn_to_virt(mod[i].mod_start),
-                         _mfn(mod[i].mod_start),
-                         PFN_UP(mod[i].mod_end), PAGE_HYPERVISOR);
+        set_pdx_range(bi->mods[i].mod->mod_start,
+                      bi->mods[i].mod->mod_start +
+                      PFN_UP(bi->mods[i].mod->mod_end));
+        map_pages_to_xen(
+            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),
+            _mfn(bi->mods[i].mod->mod_start),
+            PFN_UP(bi->mods[i].mod->mod_end), PAGE_HYPERVISOR);
     }
 
 #ifdef CONFIG_KEXEC
@@ -2061,8 +2067,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
-    dom0 = create_dom0(mod, bi->mods[0].headroom,
-                       initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
+    dom0 = create_dom0(bi->mods[0].mod, bi->mods[0].headroom,
+                       initrdidx < bi->nr_modules ?
+                            bi->mods[initrdidx].mod : NULL,
                        kextra, bi->loader);
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
-- 
2.30.2

[PATCH v5 09/44] x86/boot: introduce boot module types

[Daniel P. Smith]

This commit introduces module types of xen, kernel, and ramdisk to allow boot
module detect code to tag the purpose of a boot module. This reduces the need
for hard coded order assumptions and global variables to be used by consumers
of boot modules, such as domain construction.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 9 +++++++++
 xen/arch/x86/setup.c                | 6 ++++++
 2 files changed, 15 insertions(+)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index c7e6b4ebf0da..6941a8975ea6 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -14,6 +14,14 @@
 /* Max number of boot modules a bootloader can provide in addition to Xen */
 #define MAX_NR_BOOTMODS 63
 
+/* Boot module binary type / purpose */
+enum bootmod_type {
+    BOOTMOD_UNKNOWN,
+    BOOTMOD_XEN,
+    BOOTMOD_KERNEL,
+    BOOTMOD_RAMDISK,
+};
+
 struct boot_module {
     /* Transitionary only */
     module_t *mod;
@@ -22,6 +30,7 @@ struct boot_module {
      * reserved, into which it will be decompressed.
      */
     unsigned long headroom;
+    enum bootmod_type type;
 };
 
 /*
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index ba4bee6b93af..69c45f115523 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -311,6 +311,10 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     for ( i = 0; i <= bi->nr_modules; i++ )
         bi->mods[i].mod = &mods[i];
 
+    /* map the last mb module for xen entry */
+    bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
+    bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
+
     return bi;
 }
 
@@ -1171,6 +1175,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     bitmap_fill(module_map, bi->nr_modules);
     __clear_bit(0, module_map); /* Dom0 kernel is always first */
+    bi->mods[0].type = BOOTMOD_KERNEL;
 
     if ( pvh_boot )
     {
@@ -2058,6 +2063,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
            cpu_has_nx ? "" : "not ");
 
     initrdidx = find_first_bit(module_map, bi->nr_modules);
+    bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
     if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
         printk(XENLOG_WARNING
                "Multiple initrd candidates, picking module #%u\n",
-- 
2.30.2

[PATCH v5 10/44] x86/boot: introduce boot module flags

[Daniel P. Smith]

The existing startup code employs various ad-hoc state tracking about certain
boot module types by each area of the code. A boot module flags is added to
enable tracking these different states.  The first state to be transition by
this commit is module relocation.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 4 ++++
 xen/arch/x86/setup.c                | 8 ++++----
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 6941a8975ea6..021ff0d93643 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -31,6 +31,10 @@ struct boot_module {
      */
     unsigned long headroom;
     enum bootmod_type type;
+
+    uint32_t flags;
+#define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
+
 };
 
 /*
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 69c45f115523..161415a8e667 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1354,7 +1354,6 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
             panic("Bootloader didn't honor module alignment request\n");
         bi->mods[i].mod->mod_end -= bi->mods[i].mod->mod_start;
         bi->mods[i].mod->mod_start >>= PAGE_SHIFT;
-        bi->mods[i].mod->reserved = 0;
     }
 
     /*
@@ -1470,7 +1469,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
             size = PAGE_ALIGN(bm->headroom + bm->mod->mod_end);
 
-            if ( bi->mods[j].mod->reserved )
+            if ( bi->mods[j].flags & BOOTMOD_FLAG_X86_RELOCATED )
                 continue;
 
             /* Don't overlap with other modules (or Xen itself). */
@@ -1489,7 +1488,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                             bm->mod->mod_end);
                 bm->mod->mod_start = (end - size) >> PAGE_SHIFT;
                 bm->mod->mod_end += bm->headroom;
-                bm->mod->reserved = 1;
+                bm->flags |= BOOTMOD_FLAG_X86_RELOCATED;
             }
         }
 
@@ -1515,7 +1514,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 #endif
     }
 
-    if ( bi->mods[0].headroom && !bi->mods[0].mod->reserved )
+    if ( bi->mods[0].headroom &&
+         !(bi->mods[0].flags & BOOTMOD_FLAG_X86_RELOCATED) )
         panic("Not enough memory to relocate the dom0 kernel image\n");
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-- 
2.30.2

[PATCH v5 11/44] x86/boot: split bootstrap_map_addr() out of bootstrap_map()

[Daniel P. Smith]

From: Andrew Cooper <andrew.cooper3@citrix.com>

Using an interface based on addresses directly, not modules.

No functional change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/setup.h |  1 +
 xen/arch/x86/setup.c             | 19 +++++++++++++------
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index 3d189521189d..213584b05fb2 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -36,6 +36,7 @@ extern struct boot_info xen_boot_info;
 
 unsigned long initial_images_nrpages(nodeid_t node);
 void discard_initial_images(void);
+void *bootstrap_map_addr(paddr_t start, paddr_t end);
 void *bootstrap_map(const module_t *mod);
 
 int remove_xen_ranges(struct rangeset *r);
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 161415a8e667..1cc7fcba094b 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -437,24 +437,22 @@ static void __init normalise_cpu_order(void)
  * Ensure a given physical memory range is present in the bootstrap mappings.
  * Use superpage mappings to ensure that pagetable memory needn't be allocated.
  */
-void *__init bootstrap_map(const module_t *mod)
+void *__init bootstrap_map_addr(paddr_t start, paddr_t end)
 {
     static unsigned long __initdata map_cur = BOOTSTRAP_MAP_BASE;
-    uint64_t start, end, mask = (1L << L2_PAGETABLE_SHIFT) - 1;
+    uint64_t mask = (1L << L2_PAGETABLE_SHIFT) - 1;
     void *ret;
 
     if ( system_state != SYS_STATE_early_boot )
-        return mod ? mfn_to_virt(mod->mod_start) : NULL;
+        return end ? maddr_to_virt(start) : NULL;
 
-    if ( !mod )
+    if ( !end )
     {
         destroy_xen_mappings(BOOTSTRAP_MAP_BASE, BOOTSTRAP_MAP_LIMIT);
         map_cur = BOOTSTRAP_MAP_BASE;
         return NULL;
     }
 
-    start = (uint64_t)mod->mod_start << PAGE_SHIFT;
-    end = start + mod->mod_end;
     if ( start >= end )
         return NULL;
 
@@ -470,6 +468,15 @@ void *__init bootstrap_map(const module_t *mod)
     return ret;
 }
 
+void *__init bootstrap_map(const module_t *mod)
+{
+    if ( !mod )
+        return bootstrap_map_addr(0, 0);
+
+    return bootstrap_map_addr(pfn_to_paddr(mod->mod_start),
+                              pfn_to_paddr(mod->mod_start) + mod->mod_end);
+}
+
 static void __init move_memory(
     uint64_t dst, uint64_t src, unsigned int size)
 {
-- 
2.30.2

[PATCH v5 12/44] x86/boot: add start and size fields to struct boot_module

[Daniel P. Smith]

This commit introduces the start and size fields to struct boot_module and adds
a corresponding bootstrap mapping function, bootstrap_map_bm.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  2 ++
 xen/arch/x86/include/asm/setup.h    |  2 ++
 xen/arch/x86/setup.c                | 13 +++++++++++++
 3 files changed, 17 insertions(+)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 021ff0d93643..2ee0d5ad6d72 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -35,6 +35,8 @@ struct boot_module {
     uint32_t flags;
 #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
 
+    paddr_t start;
+    size_t size;
 };
 
 /*
diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index 213584b05fb2..bb7e73258a21 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -2,6 +2,7 @@
 #define __X86_SETUP_H_
 
 #include <xen/multiboot.h>
+#include <asm/bootinfo.h>
 #include <asm/numa.h>
 
 extern const char __2M_text_start[], __2M_text_end[];
@@ -38,6 +39,7 @@ unsigned long initial_images_nrpages(nodeid_t node);
 void discard_initial_images(void);
 void *bootstrap_map_addr(paddr_t start, paddr_t end);
 void *bootstrap_map(const module_t *mod);
+void *bootstrap_map_bm(const struct boot_module *bm);
 
 int remove_xen_ranges(struct rangeset *r);
 
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 1cc7fcba094b..093a4f5380d1 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -309,8 +309,13 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
      * should have been reserved to hold an entry for Xen.
      */
     for ( i = 0; i <= bi->nr_modules; i++ )
+    {
         bi->mods[i].mod = &mods[i];
 
+        bi->mods[i].start = (paddr_t)mods[i].mod_start;
+        bi->mods[i].size = mods[i].mod_end - mods[i].mod_start;
+    }
+
     /* map the last mb module for xen entry */
     bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
     bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
@@ -477,6 +482,14 @@ void *__init bootstrap_map(const module_t *mod)
                               pfn_to_paddr(mod->mod_start) + mod->mod_end);
 }
 
+void *__init bootstrap_map_bm(const struct boot_module *bm)
+{
+    if ( !bm )
+        return bootstrap_map_addr(0, 0);
+
+    return bootstrap_map_addr(bm->start, bm->start + bm->size);
+}
+
 static void __init move_memory(
     uint64_t dst, uint64_t src, unsigned int size)
 {
-- 
2.30.2

[PATCH v5 13/44] x86/boot: update struct boot_module on module relocation

[Daniel P. Smith]

When a boot module is relocated, ensure struct boot_module start and size
fields are updated along with early_mod.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 093a4f5380d1..f968758048ed 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1392,8 +1392,11 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
          * respective reserve_e820_ram() invocation below. No need to
          * query efi_boot_mem_unused() here, though.
          */
-        bi->mods[xen].mod->mod_start = virt_to_mfn(_stext);
-        bi->mods[xen].mod->mod_end = __2M_rwdata_end - _stext;
+        bi->mods[xen].start = virt_to_mfn(_stext);
+        bi->mods[xen].size = __2M_rwdata_end - _stext;
+
+        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
+        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
     }
 
     bi->mods[0].headroom =
-- 
2.30.2

[PATCH v5 14/44] x86/boot: transition relocation calculations to struct boot_module

[Daniel P. Smith]

Use struct boot_module fields, start and size, when calculating the relocation
address and size. It also ensures that early_mod references are kept in sync.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 36 +++++++++++++++++-------------------
 1 file changed, 17 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index f968758048ed..4f540c461b26 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1490,7 +1490,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
             struct boot_module *bm = &bi->mods[j];
             unsigned long size;
 
-            size = PAGE_ALIGN(bm->headroom + bm->mod->mod_end);
+            size = PAGE_ALIGN(bm->headroom + bm->size);
 
             if ( bi->mods[j].flags & BOOTMOD_FLAG_X86_RELOCATED )
                 continue;
@@ -1504,13 +1504,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
             if ( s < end &&
                  (bm->headroom ||
-                  ((end - size) >> PAGE_SHIFT) > bm->mod->mod_start) )
+                  paddr_to_pfn(end - size) > paddr_to_pfn(bm->start)) )
             {
-                move_memory(end - size + bm->headroom,
-                            (uint64_t)bm->mod->mod_start << PAGE_SHIFT,
-                            bm->mod->mod_end);
-                bm->mod->mod_start = (end - size) >> PAGE_SHIFT;
-                bm->mod->mod_end += bm->headroom;
+                move_memory(end - size + bm->headroom, bm->start, bm->size);
+                bm->start = (end - size);
+                bm->mod->mod_start = paddr_to_pfn(bm->start);
+                bm->size += bm->headroom;
+                bm->mod->mod_end = bm->size;
                 bm->flags |= BOOTMOD_FLAG_X86_RELOCATED;
             }
         }
@@ -1542,11 +1542,10 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         panic("Not enough memory to relocate the dom0 kernel image\n");
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        uint64_t s = (uint64_t)bi->mods[i].mod->mod_start
-                        << PAGE_SHIFT;
+        uint64_t s = (uint64_t)bi->mods[i].start;
 
         reserve_e820_ram(&boot_e820, s,
-                         s + PAGE_ALIGN(bi->mods[i].mod->mod_end));
+                         s + PAGE_ALIGN(bi->mods[i].size));
     }
 
     if ( !xen_phys_start )
@@ -1624,9 +1623,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
                 map_e = boot_e820.map[j].addr + boot_e820.map[j].size;
                 for ( j = 0; j < bi->nr_modules; ++j )
                 {
-                    uint64_t end = pfn_to_paddr(
-                                   bi->mods[j].mod->mod_start) +
-                                   bi->mods[j].mod->mod_end;
+                    uint64_t end = bi->mods[j].start +
+                                   bi->mods[j].size;
 
                     if ( map_e < end )
                         map_e = end;
@@ -1700,13 +1698,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        set_pdx_range(bi->mods[i].mod->mod_start,
-                      bi->mods[i].mod->mod_start +
-                      PFN_UP(bi->mods[i].mod->mod_end));
+        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
+                      paddr_to_pfn(bi->mods[i].mod->mod_start) +
+                      PFN_UP(bi->mods[i].size));
         map_pages_to_xen(
-            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),
-            _mfn(bi->mods[i].mod->mod_start),
-            PFN_UP(bi->mods[i].mod->mod_end), PAGE_HYPERVISOR);
+            (unsigned long)maddr_to_virt(bi->mods[i].start),
+            maddr_to_mfn(bi->mods[i].start),
+            PFN_UP(bi->mods[i].size), PAGE_HYPERVISOR);
     }
 
 #ifdef CONFIG_KEXEC
-- 
2.30.2

[PATCH v5 15/44] x86/boot: introduce boot module interator

[Daniel P. Smith]

Provide an iterator to go through boot module array searching based on type.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 2ee0d5ad6d72..c79678840d31 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -54,8 +54,24 @@ struct boot_info {
     struct boot_module mods[MAX_NR_BOOTMODS + 1];
 };
 
-#endif /* __XEN_X86_BOOTINFO_H__ */
+static inline int __init next_boot_module_index(
+    const struct boot_info *bi, enum bootmod_type t, int offset)
+{
+    int i;
+
+    for ( i = offset; i < bi->nr_modules; i++ )
+    {
+        if ( bi->mods[i].type == t )
+            return i;
+    }
+
+    return -1;
+}
 
+#define first_boot_module_index(bi, t)              \
+    next_boot_module_index(bi, t, 0)
+
+#endif /* __XEN_X86_BOOTINFO_H__ */
 /*
  * Local variables:
  * mode: C
-- 
2.30.2

[PATCH v5 16/44] x86/boot: introduce consumed flag for struct boot_module

[Daniel P. Smith]

Allow the tracking of when a boot module has been consumed by a handler in the
hypervisor independent of when it is claimed. The instances where the
hypervisor does nothing beyond claiming, the dom0 kernel, dom0 ramdisk, and a
placeholder for itself, are updated as being consumed at the time of being
claimed.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 1 +
 xen/arch/x86/setup.c                | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index c79678840d31..7833b065eff1 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -34,6 +34,7 @@ struct boot_module {
 
     uint32_t flags;
 #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
+#define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
 
     paddr_t start;
     size_t size;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 4f540c461b26..235b4e41f653 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -318,6 +318,7 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
 
     /* map the last mb module for xen entry */
     bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
+    bi->mods[bi->nr_modules].flags |= BOOTMOD_FLAG_X86_CONSUMED;
     bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
 
     return bi;
@@ -1196,6 +1197,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     bitmap_fill(module_map, bi->nr_modules);
     __clear_bit(0, module_map); /* Dom0 kernel is always first */
     bi->mods[0].type = BOOTMOD_KERNEL;
+    bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;
 
     if ( pvh_boot )
     {
@@ -2085,6 +2087,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     initrdidx = find_first_bit(module_map, bi->nr_modules);
     bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
+    bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
     if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
         printk(XENLOG_WARNING
                "Multiple initrd candidates, picking module #%u\n",
-- 
2.30.2

[PATCH v5 17/44] x86/boot: convert microcode loading to consume struct boot_info

[Daniel P. Smith]

Convert the microcode loading functions to take struct boot_info, and then
using struct boot_module to map and check for microcode. To keep the changes
focused, continue using the struct mod to hold the reference to the microcode
that is used by the late microcode logic.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/cpu/microcode/core.c    | 37 +++++++++++++---------------
 xen/arch/x86/include/asm/bootinfo.h  |  1 +
 xen/arch/x86/include/asm/microcode.h | 14 ++++++-----
 xen/arch/x86/setup.c                 |  4 +--
 4 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
index 8564e4d2c94c..22fea80bc97e 100644
--- a/xen/arch/x86/cpu/microcode/core.c
+++ b/xen/arch/x86/cpu/microcode/core.c
@@ -35,6 +35,7 @@
 #include <xen/watchdog.h>
 
 #include <asm/apic.h>
+#include <asm/bootinfo.h>
 #include <asm/cpu-policy.h>
 #include <asm/nmi.h>
 #include <asm/processor.h>
@@ -153,10 +154,8 @@ static int __init cf_check parse_ucode(const char *s)
 custom_param("ucode", parse_ucode);
 
 static void __init microcode_scan_module(
-    unsigned long *module_map,
-    const multiboot_info_t *mbi)
+    unsigned long *module_map, const struct boot_info *bi)
 {
-    module_t *mod = (module_t *)__va(mbi->mods_addr);
     uint64_t *_blob_start;
     unsigned long _blob_size;
     struct cpio_data cd;
@@ -178,16 +177,16 @@ static void __init microcode_scan_module(
     /*
      * Try all modules and see whichever could be the microcode blob.
      */
-    for ( i = 1 /* Ignore dom0 kernel */; i < mbi->mods_count; i++ )
+    for ( i = 1 /* Ignore dom0 kernel */; i < bi->nr_modules; i++ )
     {
         if ( !test_bit(i, module_map) )
             continue;
 
-        _blob_start = bootstrap_map(&mod[i]);
-        _blob_size = mod[i].mod_end;
+        _blob_start = bootstrap_map_bm(&bi->mods[i]);
+        _blob_size = bi->mods[i].size;
         if ( !_blob_start )
         {
-            printk("Could not map multiboot module #%d (size: %ld)\n",
+            printk("Could not map boot module #%d (size: %ld)\n",
                    i, _blob_size);
             continue;
         }
@@ -205,20 +204,18 @@ static void __init microcode_scan_module(
 }
 
 static void __init microcode_grab_module(
-    unsigned long *module_map,
-    const multiboot_info_t *mbi)
+    unsigned long *module_map, struct boot_info *bi)
 {
-    module_t *mod = (module_t *)__va(mbi->mods_addr);
-
     if ( ucode_mod_idx < 0 )
-        ucode_mod_idx += mbi->mods_count;
-    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= mbi->mods_count ||
+        ucode_mod_idx += bi->nr_modules;
+    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
          !__test_and_clear_bit(ucode_mod_idx, module_map) )
         goto scan;
-    ucode_mod = mod[ucode_mod_idx];
+    bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
+    ucode_mod = *bi->mods[ucode_mod_idx].mod;
 scan:
     if ( ucode_scan )
-        microcode_scan_module(module_map, mbi);
+        microcode_scan_module(module_map, bi);
 }
 
 static struct microcode_ops __ro_after_init ucode_ops;
@@ -822,8 +819,8 @@ static int __init early_update_cache(const void *data, size_t len)
     return rc;
 }
 
-int __init microcode_init_cache(unsigned long *module_map,
-                                const struct multiboot_info *mbi)
+int __init microcode_init_cache(
+    unsigned long *module_map, const struct boot_info *bi)
 {
     int rc = 0;
 
@@ -832,7 +829,7 @@ int __init microcode_init_cache(unsigned long *module_map,
 
     if ( ucode_scan )
         /* Need to rescan the modules because they might have been relocated */
-        microcode_scan_module(module_map, mbi);
+        microcode_scan_module(module_map, bi);
 
     if ( ucode_mod.mod_end )
         rc = early_update_cache(bootstrap_map(&ucode_mod),
@@ -879,7 +876,7 @@ static int __init early_microcode_update_cpu(void)
 }
 
 int __init early_microcode_init(unsigned long *module_map,
-                                const struct multiboot_info *mbi)
+                                struct boot_info *bi)
 {
     const struct cpuinfo_x86 *c = &boot_cpu_data;
     int rc = 0;
@@ -922,7 +919,7 @@ int __init early_microcode_init(unsigned long *module_map,
         return -ENODEV;
     }
 
-    microcode_grab_module(module_map, mbi);
+    microcode_grab_module(module_map, bi);
 
     if ( ucode_mod.mod_end || ucode_blob.size )
         rc = early_microcode_update_cpu();
diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 7833b065eff1..1ec29a423061 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -20,6 +20,7 @@ enum bootmod_type {
     BOOTMOD_XEN,
     BOOTMOD_KERNEL,
     BOOTMOD_RAMDISK,
+    BOOTMOD_MICROCODE,
 };
 
 struct boot_module {
diff --git a/xen/arch/x86/include/asm/microcode.h b/xen/arch/x86/include/asm/microcode.h
index 57c08205d475..495c8f7a7cc5 100644
--- a/xen/arch/x86/include/asm/microcode.h
+++ b/xen/arch/x86/include/asm/microcode.h
@@ -4,6 +4,8 @@
 #include <xen/types.h>
 #include <xen/percpu.h>
 
+#include <asm/bootinfo.h>
+
 #include <public/xen.h>
 
 struct multiboot_info;
@@ -22,12 +24,12 @@ struct cpu_signature {
 DECLARE_PER_CPU(struct cpu_signature, cpu_sig);
 
 void microcode_set_module(unsigned int idx);
-int microcode_update(XEN_GUEST_HANDLE(const_void) buf,
-                     unsigned long len, unsigned int flags);
-int early_microcode_init(unsigned long *module_map,
-                         const struct multiboot_info *mbi);
-int microcode_init_cache(unsigned long *module_map,
-                         const struct multiboot_info *mbi);
+int microcode_update(
+    XEN_GUEST_HANDLE(const_void) buf, unsigned long len, unsigned int flags);
+int early_microcode_init(
+    unsigned long *module_map, struct boot_info *bi);
+int microcode_init_cache(
+    unsigned long *module_map, const struct boot_info *bi);
 int microcode_update_one(void);
 
 #endif /* ASM_X86__MICROCODE_H */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 235b4e41f653..48c509b62a4c 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1382,7 +1382,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * TODO: load ucode earlier once multiboot modules become accessible
      * at an earlier stage.
      */
-    early_microcode_init(module_map, mbi);
+    early_microcode_init(module_map, bi);
 
     if ( xen_phys_start )
     {
@@ -1939,7 +1939,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     timer_init();
 
-    microcode_init_cache(module_map, mbi); /* Needs xmalloc() */
+    microcode_init_cache(module_map, bi); /* Needs xmalloc() */
 
     tsx_init(); /* Needs microcode.  May change HLE/RTM feature bits. */
 
-- 
2.30.2

[PATCH v5 18/44] x86/boot: convert late microcode loading to struct boot_module

[Daniel P. Smith]

Remove the use of struct mod to hold the reference for the microcode,
converting the code to work with a struct boot_module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/cpu/microcode/core.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)

diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
index 22fea80bc97e..7bcc17e0ab2f 100644
--- a/xen/arch/x86/cpu/microcode/core.c
+++ b/xen/arch/x86/cpu/microcode/core.c
@@ -28,7 +28,6 @@
 #include <xen/err.h>
 #include <xen/guest_access.h>
 #include <xen/init.h>
-#include <xen/multiboot.h>
 #include <xen/param.h>
 #include <xen/spinlock.h>
 #include <xen/stop_machine.h>
@@ -59,7 +58,7 @@
  */
 #define MICROCODE_UPDATE_TIMEOUT_US 1000000
 
-static module_t __initdata ucode_mod;
+static struct boot_module __initdata ucode_mod;
 static signed int __initdata ucode_mod_idx;
 static bool __initdata ucode_mod_forced;
 static unsigned int nr_cores;
@@ -98,7 +97,7 @@ struct patch_with_flags {
 
 static struct ucode_mod_blob __initdata ucode_blob;
 /*
- * By default we will NOT parse the multiboot modules to see if there is
+ * By default we will NOT parse the boot modules to see if there is
  * cpio image with the microcode images.
  */
 static bool __initdata ucode_scan;
@@ -199,7 +198,7 @@ static void __init microcode_scan_module(
             ucode_blob.data = cd.data;
             break;
         }
-        bootstrap_map(NULL);
+        bootstrap_map_bm(NULL);
     }
 }
 
@@ -212,7 +211,7 @@ static void __init microcode_grab_module(
          !__test_and_clear_bit(ucode_mod_idx, module_map) )
         goto scan;
     bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
-    ucode_mod = *bi->mods[ucode_mod_idx].mod;
+    ucode_mod = bi->mods[ucode_mod_idx];
 scan:
     if ( ucode_scan )
         microcode_scan_module(module_map, bi);
@@ -766,14 +765,14 @@ static int __init cf_check microcode_init(void)
      */
     if ( ucode_blob.size )
     {
-        bootstrap_map(NULL);
+        bootstrap_map_bm(NULL);
         ucode_blob.size = 0;
         ucode_blob.data = NULL;
     }
-    else if ( ucode_mod.mod_end )
+    else if ( ucode_mod.size )
     {
-        bootstrap_map(NULL);
-        ucode_mod.mod_end = 0;
+        bootstrap_map_bm(NULL);
+        ucode_mod.size = 0;
     }
 
     return 0;
@@ -831,9 +830,9 @@ int __init microcode_init_cache(
         /* Need to rescan the modules because they might have been relocated */
         microcode_scan_module(module_map, bi);
 
-    if ( ucode_mod.mod_end )
-        rc = early_update_cache(bootstrap_map(&ucode_mod),
-                                ucode_mod.mod_end);
+    if ( ucode_mod.size )
+        rc = early_update_cache(bootstrap_map_bm(&ucode_mod),
+                                ucode_mod.size);
     else if ( ucode_blob.size )
         rc = early_update_cache(ucode_blob.data, ucode_blob.size);
 
@@ -852,10 +851,10 @@ static int __init early_microcode_update_cpu(void)
         len = ucode_blob.size;
         data = ucode_blob.data;
     }
-    else if ( ucode_mod.mod_end )
+    else if ( ucode_mod.size )
     {
-        len = ucode_mod.mod_end;
-        data = bootstrap_map(&ucode_mod);
+        len = ucode_mod.size;
+        data = bootstrap_map_bm(&ucode_mod);
     }
 
     if ( !data )
@@ -921,7 +920,7 @@ int __init early_microcode_init(unsigned long *module_map,
 
     microcode_grab_module(module_map, bi);
 
-    if ( ucode_mod.mod_end || ucode_blob.size )
+    if ( ucode_mod.size || ucode_blob.size )
         rc = early_microcode_update_cpu();
 
     /*
-- 
2.30.2

[PATCH v5 19/44] x86/boot: use consumed boot module flag for microcode

[Daniel P. Smith]

To track if the microcode boot module was loaded, a copy of the boot module is
kept. The size element of this copy is set to zero as the indicator that the
microcode was loaded. A side effect is that the modules have to be rescanned to
find the boot module post-relocation, so the cache copy can be created.

Use the consumed boot module flag to track the loading of the microcode boot
module. This removes the need to manipulate the boot module size element, no
longer requiring the copy, thus allowing it to be replaced by a reference. As a
result it is no longer necessary to rescan the boot modules after relocation
has occurred.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/cpu/microcode/core.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
index 7bcc17e0ab2f..5b42aad2fdd0 100644
--- a/xen/arch/x86/cpu/microcode/core.c
+++ b/xen/arch/x86/cpu/microcode/core.c
@@ -58,7 +58,7 @@
  */
 #define MICROCODE_UPDATE_TIMEOUT_US 1000000
 
-static struct boot_module __initdata ucode_mod;
+static struct boot_module __initdata *ucode_mod;
 static signed int __initdata ucode_mod_idx;
 static bool __initdata ucode_mod_forced;
 static unsigned int nr_cores;
@@ -211,7 +211,7 @@ static void __init microcode_grab_module(
          !__test_and_clear_bit(ucode_mod_idx, module_map) )
         goto scan;
     bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
-    ucode_mod = bi->mods[ucode_mod_idx];
+    ucode_mod = &bi->mods[ucode_mod_idx];
 scan:
     if ( ucode_scan )
         microcode_scan_module(module_map, bi);
@@ -769,10 +769,10 @@ static int __init cf_check microcode_init(void)
         ucode_blob.size = 0;
         ucode_blob.data = NULL;
     }
-    else if ( ucode_mod.size )
+    else if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
     {
         bootstrap_map_bm(NULL);
-        ucode_mod.size = 0;
+        ucode_mod->flags |= BOOTMOD_FLAG_X86_CONSUMED;
     }
 
     return 0;
@@ -826,14 +826,14 @@ int __init microcode_init_cache(
     if ( !ucode_ops.apply_microcode )
         return -ENODEV;
 
-    if ( ucode_scan )
-        /* Need to rescan the modules because they might have been relocated */
+    /* Scan if microcode was not detected earlier */
+    if ( !ucode_mod )
         microcode_scan_module(module_map, bi);
 
-    if ( ucode_mod.size )
-        rc = early_update_cache(bootstrap_map_bm(&ucode_mod),
-                                ucode_mod.size);
-    else if ( ucode_blob.size )
+    if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
+        rc = early_update_cache(bootstrap_map_bm(ucode_mod),
+                                ucode_mod->size);
+    else if ( ucode_mod && ucode_blob.size )
         rc = early_update_cache(ucode_blob.data, ucode_blob.size);
 
     return rc;
@@ -851,10 +851,10 @@ static int __init early_microcode_update_cpu(void)
         len = ucode_blob.size;
         data = ucode_blob.data;
     }
-    else if ( ucode_mod.size )
+    else if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
     {
-        len = ucode_mod.size;
-        data = bootstrap_map_bm(&ucode_mod);
+        len = ucode_mod->size;
+        data = bootstrap_map_bm(ucode_mod);
     }
 
     if ( !data )
@@ -920,7 +920,7 @@ int __init early_microcode_init(unsigned long *module_map,
 
     microcode_grab_module(module_map, bi);
 
-    if ( ucode_mod.size || ucode_blob.size )
+    if ( ucode_mod || ucode_blob.size )
         rc = early_microcode_update_cpu();
 
     /*
-- 
2.30.2

[PATCH v5 20/44] x86/boot: convert xsm policy loading to struct boot_module

[Daniel P. Smith]

Iterate through the unclaimed struct boot_module to see if any are an XSM FLASK
policy. If one is located, mark it as an xsm policy.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  1 +
 xen/arch/x86/setup.c                |  2 +-
 xen/include/xsm/xsm.h               | 11 +++++++----
 xen/xsm/xsm_core.c                  | 13 +++++++++++--
 xen/xsm/xsm_policy.c                | 15 ++++++++-------
 5 files changed, 28 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 1ec29a423061..5cbd1cbbbccd 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -21,6 +21,7 @@ enum bootmod_type {
     BOOTMOD_KERNEL,
     BOOTMOD_RAMDISK,
     BOOTMOD_MICROCODE,
+    BOOTMOD_XSM_POLICY,
 };
 
 struct boot_module {
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 48c509b62a4c..e560fa798d71 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1865,7 +1865,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     mmio_ro_ranges = rangeset_new(NULL, "r/o mmio ranges",
                                   RANGESETF_prettyprint_hex);
 
-    xsm_multiboot_init(module_map, mbi);
+    xsm_multiboot_init(module_map, bi);
 
     /*
      * IOMMU-related ACPI table parsing may require some of the system domains
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index 627c0d2731af..9e511ef8878c 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -17,7 +17,10 @@
 
 #include <xen/alternative-call.h>
 #include <xen/sched.h>
-#include <xen/multiboot.h>
+
+#ifdef CONFIG_MULTIBOOT
+#include <asm/bootinfo.h>
+#endif
 
 /* policy magic number (defined by XSM_MAGIC) */
 typedef uint32_t xsm_magic_t;
@@ -779,9 +782,9 @@ static inline int xsm_argo_send(const struct domain *d, const struct domain *t)
 
 #ifdef CONFIG_MULTIBOOT
 int xsm_multiboot_init(
-    unsigned long *module_map, const multiboot_info_t *mbi);
+    unsigned long *module_map, const struct boot_info *bi);
 int xsm_multiboot_policy_init(
-    unsigned long *module_map, const multiboot_info_t *mbi,
+    unsigned long *module_map, const struct boot_info *bi,
     void **policy_buffer, size_t *policy_size);
 #endif
 
@@ -829,7 +832,7 @@ static const inline struct xsm_ops *silo_init(void)
 
 #ifdef CONFIG_MULTIBOOT
 static inline int xsm_multiboot_init (
-    unsigned long *module_map, const multiboot_info_t *mbi)
+    unsigned long *module_map, const struct boot_info *bi)
 {
     return 0;
 }
diff --git a/xen/xsm/xsm_core.c b/xen/xsm/xsm_core.c
index eaa028109bde..69d3800d4c39 100644
--- a/xen/xsm/xsm_core.c
+++ b/xen/xsm/xsm_core.c
@@ -21,6 +21,7 @@
 #ifdef CONFIG_XSM
 
 #ifdef CONFIG_MULTIBOOT
+#include <asm/bootinfo.h>
 #include <asm/setup.h>
 #endif
 
@@ -140,7 +141,7 @@ static int __init xsm_core_init(const void *policy_buffer, size_t policy_size)
 
 #ifdef CONFIG_MULTIBOOT
 int __init xsm_multiboot_init(
-    unsigned long *module_map, const multiboot_info_t *mbi)
+    unsigned long *module_map, struct boot_info *bi)
 {
     int ret = 0;
     void *policy_buffer = NULL;
@@ -150,7 +151,7 @@ int __init xsm_multiboot_init(
 
     if ( XSM_MAGIC )
     {
-        ret = xsm_multiboot_policy_init(module_map, mbi, &policy_buffer,
+        ret = xsm_multiboot_policy_init(module_map, bi, &policy_buffer,
                                         &policy_size);
         if ( ret )
         {
@@ -161,6 +162,14 @@ int __init xsm_multiboot_init(
     }
 
     ret = xsm_core_init(policy_buffer, policy_size);
+    if ( ret == 0 )
+    {
+        int idx = first_boot_module_index(bi, BOOTMOD_XSM_POLICY);
+
+        /* If the policy was loaded from a boot module, mark it consumed */
+        if ( idx >= 0 )
+            bi->mods[idx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
+    }
     bootstrap_map(NULL);
 
     return 0;
diff --git a/xen/xsm/xsm_policy.c b/xen/xsm/xsm_policy.c
index 8dafbc93810f..921bb254b9d1 100644
--- a/xen/xsm/xsm_policy.c
+++ b/xen/xsm/xsm_policy.c
@@ -21,6 +21,7 @@
 #include <xsm/xsm.h>
 #ifdef CONFIG_MULTIBOOT
 #include <xen/multiboot.h>
+#include <asm/bootinfo.h>
 #include <asm/setup.h>
 #endif
 #include <xen/bitops.h>
@@ -31,11 +32,10 @@
 
 #ifdef CONFIG_MULTIBOOT
 int __init xsm_multiboot_policy_init(
-    unsigned long *module_map, const multiboot_info_t *mbi,
+    unsigned long *module_map, struct boot_info *bi,
     void **policy_buffer, size_t *policy_size)
 {
     int i;
-    module_t *mod = (module_t *)__va(mbi->mods_addr);
     int rc = 0;
     u32 *_policy_start;
     unsigned long _policy_len;
@@ -44,13 +44,13 @@ int __init xsm_multiboot_policy_init(
      * Try all modules and see whichever could be the binary policy.
      * Adjust module_map for the module that is the binary policy.
      */
-    for ( i = mbi->mods_count-1; i >= 1; i-- )
+    for ( i = bi->nr_modules-1; i >= 1; i-- )
     {
-        if ( !test_bit(i, module_map) )
+        if ( bi->mods[i].type != BOOTMOD_UNKNOWN )
             continue;
 
-        _policy_start = bootstrap_map(mod + i);
-        _policy_len   = mod[i].mod_end;
+        _policy_start = bootstrap_map_bm(&bi->mods[i]);
+        _policy_len   = bi->mods[i].size;
 
         if ( (xsm_magic_t)(*_policy_start) == XSM_MAGIC )
         {
@@ -61,11 +61,12 @@ int __init xsm_multiboot_policy_init(
                    _policy_len,_policy_start);
 
             __clear_bit(i, module_map);
+            bi->mods[i].type = BOOTMOD_XSM_POLICY;
             break;
 
         }
 
-        bootstrap_map(NULL);
+        bootstrap_map_bm(NULL);
     }
 
     return rc;
-- 
2.30.2

[PATCH v5 21/44] x86/boot: convert ramdisk locating to struct boot_module

[Daniel P. Smith]

Locate the first unclaimed struct boot_module and mark it as ramdisk. If there
are any remaining unclaimed struct boot_module instances, report to the
console. In the change, the new boot module iterator is used to find the
initrd index, which returns a signed int. Switch initrdidx from unsigned to
signed.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e560fa798d71..e42afb0c30cb 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1033,7 +1033,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     char *kextra;
     void *bsp_stack;
     struct cpu_info *info = get_cpu_info(), *bsp_info;
-    unsigned int initrdidx, num_parked = 0;
+    unsigned int num_parked = 0;
     struct boot_info *bi;
     multiboot_info_t *mbi;
     module_t *mod;
@@ -1042,7 +1042,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     unsigned long eb_start, eb_end;
     bool acpi_boot_table_init_done = false, relocated = false;
     bool vm_init_done = false;
-    int ret;
+    int initrdidx, ret;
     struct ns16550_defaults ns16550 = {
         .data_bits = 8,
         .parity    = 'n',
@@ -2085,20 +2085,30 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
            cpu_has_nx ? XENLOG_INFO : XENLOG_WARNING "Warning: ",
            cpu_has_nx ? "" : "not ");
 
-    initrdidx = find_first_bit(module_map, bi->nr_modules);
-    bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
-    bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
-    if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
-        printk(XENLOG_WARNING
-               "Multiple initrd candidates, picking module #%u\n",
-               initrdidx);
+    /*
+     * At this point all capabilities that consume boot modules should have
+     * claimed their boot modules. Find the first unclaimed boot module and
+     * claim it as the initrd ramdisk. Do a second search to see if there are
+     * any remaining unclaimed boot modules, and report them as unusued initrd
+     * candidates.
+     */
+    initrdidx = first_boot_module_index(bi, BOOTMOD_UNKNOWN);
+    if ( initrdidx >= 0 )
+    {
+        bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
+        bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
+        if ( first_boot_module_index(bi, BOOTMOD_UNKNOWN) >= 0 )
+            printk(XENLOG_WARNING
+                   "Multiple initrd candidates, picking module #%u\n",
+                   initrdidx);
+    }
 
     /*
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
     dom0 = create_dom0(bi->mods[0].mod, bi->mods[0].headroom,
-                       initrdidx < bi->nr_modules ?
+                       (initrdidx >= 0 && initrdidx < bi->nr_modules) ?
                             bi->mods[initrdidx].mod : NULL,
                        kextra, bi->loader);
     if ( !dom0 )
-- 
2.30.2

[PATCH v5 22/44] x86/boot: remove module_map usage from microcode loading

[Daniel P. Smith]

With all consumers of module_map converted, remove usage of it
by the microcode loading logic.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/cpu/microcode/core.c    | 22 +++++++++-------------
 xen/arch/x86/include/asm/microcode.h |  6 ++----
 xen/arch/x86/setup.c                 |  4 ++--
 3 files changed, 13 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
index 5b42aad2fdd0..f7ce2247a33e 100644
--- a/xen/arch/x86/cpu/microcode/core.c
+++ b/xen/arch/x86/cpu/microcode/core.c
@@ -152,8 +152,7 @@ static int __init cf_check parse_ucode(const char *s)
 }
 custom_param("ucode", parse_ucode);
 
-static void __init microcode_scan_module(
-    unsigned long *module_map, const struct boot_info *bi)
+static void __init microcode_scan_module(const struct boot_info *bi)
 {
     uint64_t *_blob_start;
     unsigned long _blob_size;
@@ -178,7 +177,7 @@ static void __init microcode_scan_module(
      */
     for ( i = 1 /* Ignore dom0 kernel */; i < bi->nr_modules; i++ )
     {
-        if ( !test_bit(i, module_map) )
+        if ( bi->mods[i].type != BOOTMOD_UNKNOWN )
             continue;
 
         _blob_start = bootstrap_map_bm(&bi->mods[i]);
@@ -202,19 +201,18 @@ static void __init microcode_scan_module(
     }
 }
 
-static void __init microcode_grab_module(
-    unsigned long *module_map, struct boot_info *bi)
+static void __init microcode_grab_module(struct boot_info *bi)
 {
     if ( ucode_mod_idx < 0 )
         ucode_mod_idx += bi->nr_modules;
     if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
-         !__test_and_clear_bit(ucode_mod_idx, module_map) )
+         (bi->mods[ucode_mod_idx].type != BOOTMOD_UNKNOWN) )
         goto scan;
     bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
     ucode_mod = &bi->mods[ucode_mod_idx];
 scan:
     if ( ucode_scan )
-        microcode_scan_module(module_map, bi);
+        microcode_scan_module(bi);
 }
 
 static struct microcode_ops __ro_after_init ucode_ops;
@@ -818,8 +816,7 @@ static int __init early_update_cache(const void *data, size_t len)
     return rc;
 }
 
-int __init microcode_init_cache(
-    unsigned long *module_map, const struct boot_info *bi)
+int __init microcode_init_cache(const struct boot_info *bi)
 {
     int rc = 0;
 
@@ -828,7 +825,7 @@ int __init microcode_init_cache(
 
     /* Scan if microcode was not detected earlier */
     if ( !ucode_mod )
-        microcode_scan_module(module_map, bi);
+        microcode_scan_module(bi);
 
     if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
         rc = early_update_cache(bootstrap_map_bm(ucode_mod),
@@ -874,8 +871,7 @@ static int __init early_microcode_update_cpu(void)
     return microcode_update_cpu(patch, 0);
 }
 
-int __init early_microcode_init(unsigned long *module_map,
-                                struct boot_info *bi)
+int __init early_microcode_init(struct boot_info *bi)
 {
     const struct cpuinfo_x86 *c = &boot_cpu_data;
     int rc = 0;
@@ -918,7 +914,7 @@ int __init early_microcode_init(unsigned long *module_map,
         return -ENODEV;
     }
 
-    microcode_grab_module(module_map, bi);
+    microcode_grab_module(bi);
 
     if ( ucode_mod || ucode_blob.size )
         rc = early_microcode_update_cpu();
diff --git a/xen/arch/x86/include/asm/microcode.h b/xen/arch/x86/include/asm/microcode.h
index 495c8f7a7cc5..84e0eb487244 100644
--- a/xen/arch/x86/include/asm/microcode.h
+++ b/xen/arch/x86/include/asm/microcode.h
@@ -26,10 +26,8 @@ DECLARE_PER_CPU(struct cpu_signature, cpu_sig);
 void microcode_set_module(unsigned int idx);
 int microcode_update(
     XEN_GUEST_HANDLE(const_void) buf, unsigned long len, unsigned int flags);
-int early_microcode_init(
-    unsigned long *module_map, struct boot_info *bi);
-int microcode_init_cache(
-    unsigned long *module_map, const struct boot_info *bi);
+int early_microcode_init(struct boot_info *bi);
+int microcode_init_cache(const struct boot_info *bi);
 int microcode_update_one(void);
 
 #endif /* ASM_X86__MICROCODE_H */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e42afb0c30cb..90acd4180441 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1382,7 +1382,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * TODO: load ucode earlier once multiboot modules become accessible
      * at an earlier stage.
      */
-    early_microcode_init(module_map, bi);
+    early_microcode_init(bi);
 
     if ( xen_phys_start )
     {
@@ -1939,7 +1939,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     timer_init();
 
-    microcode_init_cache(module_map, bi); /* Needs xmalloc() */
+    microcode_init_cache(bi); /* Needs xmalloc() */
 
     tsx_init(); /* Needs microcode.  May change HLE/RTM feature bits. */
 
-- 
2.30.2

[PATCH v5 23/44] x86/boot: remove module_map usage from xsm policy loading

[Daniel P. Smith]

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c  | 2 +-
 xen/include/xsm/xsm.h | 9 +++------
 xen/xsm/xsm_core.c    | 6 ++----
 xen/xsm/xsm_policy.c  | 5 +----
 4 files changed, 7 insertions(+), 15 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 90acd4180441..b0946216ea3f 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1865,7 +1865,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     mmio_ro_ranges = rangeset_new(NULL, "r/o mmio ranges",
                                   RANGESETF_prettyprint_hex);
 
-    xsm_multiboot_init(module_map, bi);
+    xsm_multiboot_init(bi);
 
     /*
      * IOMMU-related ACPI table parsing may require some of the system domains
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index 9e511ef8878c..791936e5285b 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -781,11 +781,9 @@ static inline int xsm_argo_send(const struct domain *d, const struct domain *t)
 #endif /* XSM_NO_WRAPPERS */
 
 #ifdef CONFIG_MULTIBOOT
-int xsm_multiboot_init(
-    unsigned long *module_map, const struct boot_info *bi);
+int xsm_multiboot_init(struct boot_info *bi);
 int xsm_multiboot_policy_init(
-    unsigned long *module_map, const struct boot_info *bi,
-    void **policy_buffer, size_t *policy_size);
+    struct boot_info *bi, void **policy_buffer, size_t *policy_size);
 #endif
 
 #ifdef CONFIG_HAS_DEVICE_TREE
@@ -831,8 +829,7 @@ static const inline struct xsm_ops *silo_init(void)
 #include <xsm/dummy.h>
 
 #ifdef CONFIG_MULTIBOOT
-static inline int xsm_multiboot_init (
-    unsigned long *module_map, const struct boot_info *bi)
+static inline int xsm_multiboot_init(struct boot_info *bi)
 {
     return 0;
 }
diff --git a/xen/xsm/xsm_core.c b/xen/xsm/xsm_core.c
index 69d3800d4c39..0e74b96a4cc7 100644
--- a/xen/xsm/xsm_core.c
+++ b/xen/xsm/xsm_core.c
@@ -140,8 +140,7 @@ static int __init xsm_core_init(const void *policy_buffer, size_t policy_size)
 }
 
 #ifdef CONFIG_MULTIBOOT
-int __init xsm_multiboot_init(
-    unsigned long *module_map, struct boot_info *bi)
+int __init xsm_multiboot_init(struct boot_info *bi)
 {
     int ret = 0;
     void *policy_buffer = NULL;
@@ -151,8 +150,7 @@ int __init xsm_multiboot_init(
 
     if ( XSM_MAGIC )
     {
-        ret = xsm_multiboot_policy_init(module_map, bi, &policy_buffer,
-                                        &policy_size);
+        ret = xsm_multiboot_policy_init(bi, &policy_buffer, &policy_size);
         if ( ret )
         {
             bootstrap_map(NULL);
diff --git a/xen/xsm/xsm_policy.c b/xen/xsm/xsm_policy.c
index 921bb254b9d1..a22367a62e93 100644
--- a/xen/xsm/xsm_policy.c
+++ b/xen/xsm/xsm_policy.c
@@ -32,8 +32,7 @@
 
 #ifdef CONFIG_MULTIBOOT
 int __init xsm_multiboot_policy_init(
-    unsigned long *module_map, struct boot_info *bi,
-    void **policy_buffer, size_t *policy_size)
+    struct boot_info *bi, void **policy_buffer, size_t *policy_size)
 {
     int i;
     int rc = 0;
@@ -42,7 +41,6 @@ int __init xsm_multiboot_policy_init(
 
     /*
      * Try all modules and see whichever could be the binary policy.
-     * Adjust module_map for the module that is the binary policy.
      */
     for ( i = bi->nr_modules-1; i >= 1; i-- )
     {
@@ -60,7 +58,6 @@ int __init xsm_multiboot_policy_init(
             printk("Policy len %#lx, start at %p.\n",
                    _policy_len,_policy_start);
 
-            __clear_bit(i, module_map);
             bi->mods[i].type = BOOTMOD_XSM_POLICY;
             break;
 
-- 
2.30.2

[PATCH v5 24/44] x86/boot: remove module_map usage by ramdisk loading

[Daniel P. Smith]

The ramdisk loading is the last user of module_map, remove
its usage and any remaining remnants of module_map.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index b0946216ea3f..0d2ee19998aa 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1037,7 +1037,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     struct boot_info *bi;
     multiboot_info_t *mbi;
     module_t *mod;
-    unsigned long nr_pages, raw_max_page, module_map[1];
+    unsigned long nr_pages, raw_max_page;
     int i, j, e820_warn = 0, bytes = 0;
     unsigned long eb_start, eb_end;
     bool acpi_boot_table_init_done = false, relocated = false;
@@ -1187,15 +1187,14 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         panic("dom0 kernel not specified. Check bootloader configuration\n");
 
     /* Check that we don't have a silly number of modules. */
-    if ( bi->nr_modules > sizeof(module_map) * 8 )
+    if ( bi->nr_modules > MAX_NR_BOOTMODS + 1 )
     {
-        bi->nr_modules = sizeof(module_map) * 8;
-        printk("Excessive boot modules - using the first %u only\n",
+        bi->nr_modules = MAX_NR_BOOTMODS + 1;
+        printk("Excessive multiboot modules - using the first %u only\n",
                bi->nr_modules);
     }
 
-    bitmap_fill(module_map, bi->nr_modules);
-    __clear_bit(0, module_map); /* Dom0 kernel is always first */
+    /* Dom0 kernel is always first */
     bi->mods[0].type = BOOTMOD_KERNEL;
     bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;
 
-- 
2.30.2

[PATCH v5 25/44] x86/boot: convert create_dom0 to use boot info

[Daniel P. Smith]

This commit changes create_dom0 to no longer take the individual components and
take struct boot_info instead. Internally, it is changed to locate the kernel
and ramdisk details from struct boot_info.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 0d2ee19998aa..c2bcddc50990 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -946,10 +946,8 @@ static unsigned int __init copy_bios_e820(struct e820entry *map, unsigned int li
     return n;
 }
 
-static struct domain *__init create_dom0(const module_t *image,
-                                         unsigned long headroom,
-                                         module_t *initrd, const char *kextra,
-                                         const char *loader)
+static struct domain *__init create_dom0(const struct boot_info *bi,
+                                         const char *kextra)
 {
     static char __initdata cmdline[MAX_GUEST_CMDLINE];
 
@@ -964,9 +962,21 @@ static struct domain *__init create_dom0(const module_t *image,
             .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
         },
     };
+    int headroom, mod_idx = first_boot_module_index(bi, BOOTMOD_RAMDISK);
+    module_t *image, *initrd;
     struct domain *d;
     domid_t domid;
 
+    /* Map boot_module to mb1 module for dom0 */
+    image = bi->mods[0].mod;
+    headroom = bi->mods[0].headroom;
+
+    /* Map boot_module to mb1 module for initrd */
+    if ( mod_idx < 0 )
+        initrd = NULL;
+    else
+        initrd = bi->mods[mod_idx].mod;
+
     if ( opt_dom0_pvh )
     {
         dom0_cfg.flags |= (XEN_DOMCTL_CDF_hvm |
@@ -995,7 +1005,7 @@ static struct domain *__init create_dom0(const module_t *image,
     if ( image->string || kextra )
     {
         if ( image->string )
-            safe_strcpy(cmdline, cmdline_cook(__va(image->string), loader));
+            safe_strcpy(cmdline, cmdline_cook(__va(image->string), bi->loader));
 
         if ( kextra )
             /* kextra always includes exactly one leading space. */
@@ -2106,10 +2116,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
-    dom0 = create_dom0(bi->mods[0].mod, bi->mods[0].headroom,
-                       (initrdidx >= 0 && initrdidx < bi->nr_modules) ?
-                            bi->mods[initrdidx].mod : NULL,
-                       kextra, bi->loader);
+    dom0 = create_dom0(bi, kextra);
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
 
-- 
2.30.2

[PATCH v5 26/44] x86/boot: convert construct_dom0 to use struct boot_module

[Daniel P. Smith]

The construct_dom0 function is converted to consume struct boot_module
instances for the kernel and ramdisk. With this change, it is no longer
necessary for the internal use of struct mod by create_dom0, so they are
changed to struct boot_module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c        | 12 +++++++-----
 xen/arch/x86/include/asm/setup.h |  6 ++----
 xen/arch/x86/setup.c             | 20 ++++++++++----------
 3 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 8d56705a0861..1eff3192f72d 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -13,6 +13,7 @@
 #include <xen/softirq.h>
 
 #include <asm/amd.h>
+#include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
 #include <asm/guest.h>
 #include <asm/hpet.h>
@@ -596,9 +597,8 @@ int __init dom0_setup_permissions(struct domain *d)
     return rc;
 }
 
-int __init construct_dom0(struct domain *d, const module_t *image,
-                          unsigned long image_headroom, module_t *initrd,
-                          const char *cmdline)
+int __init construct_dom0(struct domain *d, const struct boot_module *image,
+                          struct boot_module *initrd, const char *cmdline)
 {
     int rc;
 
@@ -610,9 +610,11 @@ int __init construct_dom0(struct domain *d, const module_t *image,
     process_pending_softirqs();
 
     if ( is_hvm_domain(d) )
-        rc = dom0_construct_pvh(d, image, image_headroom, initrd, cmdline);
+        rc = dom0_construct_pvh(d, image->mod, image->headroom, initrd->mod,
+                                cmdline);
     else if ( is_pv_domain(d) )
-        rc = dom0_construct_pv(d, image, image_headroom, initrd, cmdline);
+        rc = dom0_construct_pv(d, image->mod, image->headroom, initrd->mod,
+                               cmdline);
     else
         panic("Cannot construct Dom0. No guest interface available\n");
 
diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index bb7e73258a21..04bf0e62a9d7 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -27,10 +27,8 @@ void subarch_init_memory(void);
 void init_IRQ(void);
 
 int construct_dom0(
-    struct domain *d,
-    const module_t *image, unsigned long image_headroom,
-    module_t *initrd,
-    const char *cmdline);
+    struct domain *d, const struct boot_module *image,
+    struct boot_module *initrd, const char *cmdline);
 void setup_io_bitmap(struct domain *d);
 
 extern struct boot_info xen_boot_info;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index c2bcddc50990..135bd6591c1c 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -946,7 +946,7 @@ static unsigned int __init copy_bios_e820(struct e820entry *map, unsigned int li
     return n;
 }
 
-static struct domain *__init create_dom0(const struct boot_info *bi,
+static struct domain *__init create_dom0(struct boot_info *bi,
                                          const char *kextra)
 {
     static char __initdata cmdline[MAX_GUEST_CMDLINE];
@@ -962,20 +962,19 @@ static struct domain *__init create_dom0(const struct boot_info *bi,
             .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
         },
     };
-    int headroom, mod_idx = first_boot_module_index(bi, BOOTMOD_RAMDISK);
-    module_t *image, *initrd;
+    int mod_idx = first_boot_module_index(bi, BOOTMOD_RAMDISK);
+    struct boot_module *image, *initrd;
     struct domain *d;
     domid_t domid;
 
     /* Map boot_module to mb1 module for dom0 */
-    image = bi->mods[0].mod;
-    headroom = bi->mods[0].headroom;
+    image = &bi->mods[0];
 
     /* Map boot_module to mb1 module for initrd */
     if ( mod_idx < 0 )
         initrd = NULL;
     else
-        initrd = bi->mods[mod_idx].mod;
+        initrd = &bi->mods[mod_idx];
 
     if ( opt_dom0_pvh )
     {
@@ -1002,10 +1001,11 @@ static struct domain *__init create_dom0(const struct boot_info *bi,
         panic("Error creating d%uv0\n", domid);
 
     /* Grab the DOM0 command line. */
-    if ( image->string || kextra )
+    if ( image->mod->string || kextra )
     {
-        if ( image->string )
-            safe_strcpy(cmdline, cmdline_cook(__va(image->string), bi->loader));
+        if ( image->mod->string )
+            safe_strcpy(cmdline, cmdline_cook(__va(image->mod->string),
+                        bi->loader));
 
         if ( kextra )
             /* kextra always includes exactly one leading space. */
@@ -1028,7 +1028,7 @@ static struct domain *__init create_dom0(const struct boot_info *bi,
         }
     }
 
-    if ( construct_dom0(d, image, headroom, initrd, cmdline) != 0 )
+    if ( construct_dom0(d, image, initrd, cmdline) != 0 )
         panic("Could not construct domain 0\n");
 
     return d;
-- 
2.30.2

[PATCH v5 27/44] x86/boot: relocate kextra into boot info

[Daniel P. Smith]

Move kextra into struct boot_info, thus no longer needed to be passed as a
parameter to create_dom0.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h |  1 +
 xen/arch/x86/setup.c                | 12 ++++++------
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 5cbd1cbbbccd..9ed260629012 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -49,6 +49,7 @@ struct boot_module {
 struct boot_info {
     const char *loader;
     const char *cmdline;
+    const char *kextra;
 
     paddr_t memmap_addr;
     size_t memmap_length;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 135bd6591c1c..aed0837902c4 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -946,8 +946,7 @@ static unsigned int __init copy_bios_e820(struct e820entry *map, unsigned int li
     return n;
 }
 
-static struct domain *__init create_dom0(struct boot_info *bi,
-                                         const char *kextra)
+static struct domain *__init create_dom0(struct boot_info *bi)
 {
     static char __initdata cmdline[MAX_GUEST_CMDLINE];
 
@@ -1001,15 +1000,15 @@ static struct domain *__init create_dom0(struct boot_info *bi,
         panic("Error creating d%uv0\n", domid);
 
     /* Grab the DOM0 command line. */
-    if ( image->mod->string || kextra )
+    if ( image->mod->string || bi->kextra )
     {
         if ( image->mod->string )
             safe_strcpy(cmdline, cmdline_cook(__va(image->mod->string),
                         bi->loader));
 
-        if ( kextra )
+        if ( bi->kextra )
             /* kextra always includes exactly one leading space. */
-            safe_strcat(cmdline, kextra);
+            safe_strcat(cmdline, bi->kextra);
 
         /* Append any extra parameters. */
         if ( skip_ioapic_setup && !strstr(cmdline, "noapic") )
@@ -1103,6 +1102,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
         *kextra = '\0';
         kextra += 3;
         while ( kextra[1] == ' ' ) kextra++;
+        bi->kextra = kextra;
     }
     cmdline_parse(bi->cmdline);
 
@@ -2116,7 +2116,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
      * We're going to setup domain0 using the module(s) that we stashed safely
      * above our heap. The second module, if present, is an initrd ramdisk.
      */
-    dom0 = create_dom0(bi, kextra);
+    dom0 = create_dom0(bi);
     if ( !dom0 )
         panic("Could not set up DOM0 guest OS\n");
 
-- 
2.30.2

[PATCH v5 28/44] x86/boot: add cmdline to struct boot_module

[Daniel P. Smith]

Add a char pointer field, cmdline, to struct boot_module to hold the address
pointed to by the string field of struct mod. This removes the need to use the
early_mod field to get to the dom0 kernel command line.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 2 ++
 xen/arch/x86/setup.c                | 9 ++++++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 9ed260629012..3b6bfbe88770 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -38,6 +38,8 @@ struct boot_module {
 #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
 #define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
 
+    const char *cmdline;
+
     paddr_t start;
     size_t size;
 };
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index aed0837902c4..d5916e85f68e 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -312,6 +312,8 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     {
         bi->mods[i].mod = &mods[i];
 
+        bi->mods[i].cmdline = (char *)(paddr_t)mods[i].string;
+
         bi->mods[i].start = (paddr_t)mods[i].mod_start;
         bi->mods[i].size = mods[i].mod_end - mods[i].mod_start;
     }
@@ -1000,10 +1002,11 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         panic("Error creating d%uv0\n", domid);
 
     /* Grab the DOM0 command line. */
-    if ( image->mod->string || bi->kextra )
+    if ( image->cmdline || bi->kextra )
     {
-        if ( image->mod->string )
-            safe_strcpy(cmdline, cmdline_cook(__va(image->mod->string),
+        if ( image->cmdline )
+            safe_strcpy(cmdline,
+                        cmdline_cook(__va((unsigned long)image->cmdline),
                         bi->loader));
 
         if ( bi->kextra )
-- 
2.30.2

[PATCH v5 29/44] x86/boot: convert dom0_construct_pv image param to struct boot_module

[Daniel P. Smith]

This changes the type for the image parameter of dom0_construct_pv to be struct
boot_module. Removing the usage of early_mod field for kernel module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             |  3 +--
 xen/arch/x86/include/asm/dom0_build.h |  7 +++----
 xen/arch/x86/pv/dom0_build.c          | 14 ++++++--------
 3 files changed, 10 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 1eff3192f72d..54db578dd29f 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -613,8 +613,7 @@ int __init construct_dom0(struct domain *d, const struct boot_module *image,
         rc = dom0_construct_pvh(d, image->mod, image->headroom, initrd->mod,
                                 cmdline);
     else if ( is_pv_domain(d) )
-        rc = dom0_construct_pv(d, image->mod, image->headroom, initrd->mod,
-                               cmdline);
+        rc = dom0_construct_pv(d, image, initrd->mod, cmdline);
     else
         panic("Cannot construct Dom0. No guest interface available\n");
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 107c1ff98367..a1f36f7d360d 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -13,10 +13,9 @@ unsigned long dom0_compute_nr_pages(struct domain *d,
                                     unsigned long initrd_len);
 int dom0_setup_permissions(struct domain *d);
 
-int dom0_construct_pv(struct domain *d, const module_t *image,
-                      unsigned long image_headroom,
-                      module_t *initrd,
-                      const char *cmdline);
+int dom0_construct_pv(
+    struct domain *d, const struct boot_module *image,
+    module_t *initrd, const char *cmdline);
 
 int dom0_construct_pvh(struct domain *d, const module_t *image,
                        unsigned long image_headroom,
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 262edb6bf2f0..7b6afe64d799 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -355,8 +355,7 @@ static struct page_info * __init alloc_chunk(struct domain *d,
 }
 
 static int __init dom0_construct(struct domain *d,
-                                 const module_t *image,
-                                 unsigned long image_headroom,
+                                 const struct boot_module *image,
                                  module_t *initrd,
                                  const char *cmdline)
 {
@@ -374,9 +373,9 @@ static int __init dom0_construct(struct domain *d,
     unsigned int flush_flags = 0;
     start_info_t *si;
     struct vcpu *v = d->vcpu[0];
-    void *image_base = bootstrap_map(image);
-    unsigned long image_len = image->mod_end;
-    void *image_start = image_base + image_headroom;
+    void *image_base = bootstrap_map_bm(image);
+    unsigned long image_len = image->size;
+    void *image_start = image_base + image->headroom;
     unsigned long initrd_len = initrd ? initrd->mod_end : 0;
     l4_pgentry_t *l4tab = NULL, *l4start = NULL;
     l3_pgentry_t *l3tab = NULL, *l3start = NULL;
@@ -1052,8 +1051,7 @@ out:
 }
 
 int __init dom0_construct_pv(struct domain *d,
-                             const module_t *image,
-                             unsigned long image_headroom,
+                             const struct boot_module *image,
                              module_t *initrd,
                              const char *cmdline)
 {
@@ -1072,7 +1070,7 @@ int __init dom0_construct_pv(struct domain *d,
         write_cr4(read_cr4() & ~X86_CR4_SMAP);
     }
 
-    rc = dom0_construct(d, image, image_headroom, initrd, cmdline);
+    rc = dom0_construct(d, image, initrd, cmdline);
 
     if ( boot_cpu_has(X86_FEATURE_XEN_SMAP) )
     {
-- 
2.30.2

[PATCH v5 30/44] x86/boot: convert dom0_construct_pv initrd param to struct boot_module

[Daniel P. Smith]

This changes the type for the initrd parameter of dom0_construct_pv to be struct
boot_module. This conversion requires several adjustments throughout dom0_construct_pv
to account for the type change. Removes the usage of early_mod field for ramdisk module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             |  2 +-
 xen/arch/x86/include/asm/dom0_build.h |  2 +-
 xen/arch/x86/pv/dom0_build.c          | 29 +++++++++++++++------------
 3 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 54db578dd29f..1d05ee53a6d8 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -613,7 +613,7 @@ int __init construct_dom0(struct domain *d, const struct boot_module *image,
         rc = dom0_construct_pvh(d, image->mod, image->headroom, initrd->mod,
                                 cmdline);
     else if ( is_pv_domain(d) )
-        rc = dom0_construct_pv(d, image, initrd->mod, cmdline);
+        rc = dom0_construct_pv(d, image, initrd, cmdline);
     else
         panic("Cannot construct Dom0. No guest interface available\n");
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index a1f36f7d360d..2ce5ea3851af 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -15,7 +15,7 @@ int dom0_setup_permissions(struct domain *d);
 
 int dom0_construct_pv(
     struct domain *d, const struct boot_module *image,
-    module_t *initrd, const char *cmdline);
+    struct boot_module *initrd, const char *cmdline);
 
 int dom0_construct_pvh(struct domain *d, const module_t *image,
                        unsigned long image_headroom,
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 7b6afe64d799..16b8c1e40998 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -356,7 +356,7 @@ static struct page_info * __init alloc_chunk(struct domain *d,
 
 static int __init dom0_construct(struct domain *d,
                                  const struct boot_module *image,
-                                 module_t *initrd,
+                                 struct boot_module *initrd,
                                  const char *cmdline)
 {
     int i, rc, order, machine;
@@ -367,7 +367,8 @@ static int __init dom0_construct(struct domain *d,
     unsigned long nr_pt_pages;
     unsigned long alloc_spfn;
     unsigned long alloc_epfn;
-    unsigned long initrd_pfn = -1, initrd_mfn = 0;
+    unsigned long initrd_pfn = -1;
+    mfn_t  initrd_mfn = { 0 };
     unsigned long count;
     struct page_info *page = NULL;
     unsigned int flush_flags = 0;
@@ -376,7 +377,7 @@ static int __init dom0_construct(struct domain *d,
     void *image_base = bootstrap_map_bm(image);
     unsigned long image_len = image->size;
     void *image_start = image_base + image->headroom;
-    unsigned long initrd_len = initrd ? initrd->mod_end : 0;
+    unsigned long initrd_len = initrd ? initrd->size : 0;
     l4_pgentry_t *l4tab = NULL, *l4start = NULL;
     l3_pgentry_t *l3tab = NULL, *l3start = NULL;
     l2_pgentry_t *l2tab = NULL, *l2start = NULL;
@@ -612,7 +613,8 @@ static int __init dom0_construct(struct domain *d,
         initrd_pfn = vinitrd_start ?
                      (vinitrd_start - v_start) >> PAGE_SHIFT :
                      domain_tot_pages(d);
-        initrd_mfn = mfn = initrd->mod_start;
+        initrd_mfn = maddr_to_mfn(initrd->start);
+        mfn = mfn_x(initrd_mfn);
         count = PFN_UP(initrd_len);
         if ( d->arch.physaddr_bitsize &&
              ((mfn + count - 1) >> (d->arch.physaddr_bitsize - PAGE_SHIFT)) )
@@ -627,12 +629,13 @@ static int __init dom0_construct(struct domain *d,
                     free_domheap_pages(page, order);
                     page += 1UL << order;
                 }
-            memcpy(page_to_virt(page), mfn_to_virt(initrd->mod_start),
+            memcpy(page_to_virt(page), maddr_to_virt(initrd->start),
                    initrd_len);
-            mpt_alloc = (paddr_t)initrd->mod_start << PAGE_SHIFT;
+            mpt_alloc = initrd->start;
             init_domheap_pages(mpt_alloc,
                                mpt_alloc + PAGE_ALIGN(initrd_len));
-            initrd->mod_start = initrd_mfn = mfn_x(page_to_mfn(page));
+            initrd_mfn = page_to_mfn(page);
+            initrd->start = mfn_to_maddr(initrd_mfn);
         }
         else
         {
@@ -640,9 +643,9 @@ static int __init dom0_construct(struct domain *d,
                 if ( assign_pages(mfn_to_page(_mfn(mfn++)), 1, d, 0) )
                     BUG();
         }
-        initrd->mod_end = 0;
+        initrd->size = 0;
 
-        iommu_memory_setup(d, "initrd", mfn_to_page(_mfn(initrd_mfn)),
+        iommu_memory_setup(d, "initrd", mfn_to_page(initrd_mfn),
                            PFN_UP(initrd_len), &flush_flags);
     }
 
@@ -654,7 +657,7 @@ static int __init dom0_construct(struct domain *d,
                nr_pages - domain_tot_pages(d));
     if ( initrd )
     {
-        mpt_alloc = (paddr_t)initrd->mod_start << PAGE_SHIFT;
+        mpt_alloc = initrd->start;
         printk("\n Init. ramdisk: %"PRIpaddr"->%"PRIpaddr,
                mpt_alloc, mpt_alloc + initrd_len);
     }
@@ -763,7 +766,7 @@ static int __init dom0_construct(struct domain *d,
         if ( count < initrd_pfn || count >= initrd_pfn + PFN_UP(initrd_len) )
             mfn = pfn++;
         else
-            mfn = initrd_mfn++;
+            mfn = mfn_x(initrd_mfn) + 1;
         *l1tab = l1e_from_pfn(mfn, compat ? COMPAT_L1_PROT : L1_PROT);
         l1tab++;
 
@@ -882,7 +885,7 @@ static int __init dom0_construct(struct domain *d,
         if ( pfn >= initrd_pfn )
         {
             if ( pfn < initrd_pfn + PFN_UP(initrd_len) )
-                mfn = initrd->mod_start + (pfn - initrd_pfn);
+                mfn = paddr_to_pfn(initrd->start) + (pfn - initrd_pfn);
             else
                 mfn -= PFN_UP(initrd_len);
         }
@@ -1052,7 +1055,7 @@ out:
 
 int __init dom0_construct_pv(struct domain *d,
                              const struct boot_module *image,
-                             module_t *initrd,
+                             struct boot_module *initrd,
                              const char *cmdline)
 {
     int rc;
-- 
2.30.2

[PATCH v5 31/44] x86/boot: convert dom0_construct_pvh to struct boot_module

[Daniel P. Smith]

This changes both the kernel and ramdisk parameters over to struct
boot_module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             |  3 +--
 xen/arch/x86/hvm/dom0_build.c         | 11 +++++------
 xen/arch/x86/include/asm/dom0_build.h |  7 +++----
 3 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 1d05ee53a6d8..71b2e3afc1a1 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -610,8 +610,7 @@ int __init construct_dom0(struct domain *d, const struct boot_module *image,
     process_pending_softirqs();
 
     if ( is_hvm_domain(d) )
-        rc = dom0_construct_pvh(d, image->mod, image->headroom, initrd->mod,
-                                cmdline);
+        rc = dom0_construct_pvh(d, image, initrd, cmdline);
     else if ( is_pv_domain(d) )
         rc = dom0_construct_pv(d, image, initrd, cmdline);
     else
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 3dd913bdb029..beaa1d492077 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -1300,10 +1300,9 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
     }
 }
 
-int __init dom0_construct_pvh(struct domain *d, const module_t *image,
-                              unsigned long image_headroom,
-                              module_t *initrd,
-                              const char *cmdline)
+int __init dom0_construct_pvh(
+    struct domain *d, const struct boot_module *image,
+    struct boot_module *initrd, const char *cmdline)
 {
     paddr_t entry, start_info;
     int rc;
@@ -1347,8 +1346,8 @@ int __init dom0_construct_pvh(struct domain *d, const module_t *image,
         return rc;
     }
 
-    rc = pvh_load_kernel(d, image, image_headroom, initrd, bootstrap_map(image),
-                         cmdline, &entry, &start_info);
+    rc = pvh_load_kernel(d, image->mod, image->headroom, initrd->mod,
+                         bootstrap_map_bm(image), cmdline, &entry, &start_info);
     if ( rc )
     {
         printk("Failed to load Dom0 kernel\n");
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 2ce5ea3851af..8f7b37f3d308 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -17,10 +17,9 @@ int dom0_construct_pv(
     struct domain *d, const struct boot_module *image,
     struct boot_module *initrd, const char *cmdline);
 
-int dom0_construct_pvh(struct domain *d, const module_t *image,
-                       unsigned long image_headroom,
-                       module_t *initrd,
-                       const char *cmdline);
+int dom0_construct_pvh(
+    struct domain *d, const struct boot_module *image,
+    struct boot_module *initrd, const char *cmdline);
 
 unsigned long dom0_paging_pages(const struct domain *d,
                                 unsigned long nr_pages);
-- 
2.30.2

[PATCH v5 32/44] x86/boot: convert pvh_load_kernel to struct boot_module

[Daniel P. Smith]

This changes both the kernel and ramdisk parameters over to struct boot_module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index beaa1d492077..a3fd5e762dc4 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -642,15 +642,14 @@ static bool __init check_and_adjust_load_address(
     return true;
 }
 
-static int __init pvh_load_kernel(struct domain *d, const module_t *image,
-                                  unsigned long image_headroom,
-                                  module_t *initrd, void *image_base,
-                                  const char *cmdline, paddr_t *entry,
-                                  paddr_t *start_info_addr)
+static int __init pvh_load_kernel(
+    struct domain *d, const struct boot_module *image,
+    struct boot_module *initrd, void *image_base,
+    const char *cmdline, paddr_t *entry, paddr_t *start_info_addr)
 {
-    void *image_start = image_base + image_headroom;
-    unsigned long image_len = image->mod_end;
-    unsigned long initrd_len = initrd ? initrd->mod_end : 0;
+    void *image_start = image_base + image->headroom;
+    unsigned long image_len = image->size;
+    unsigned long initrd_len = initrd ? initrd->size : 0;
     struct elf_binary elf;
     struct elf_dom_parms parms;
     paddr_t last_addr;
@@ -725,7 +724,7 @@ static int __init pvh_load_kernel(struct domain *d, const module_t *image,
 
     if ( initrd != NULL )
     {
-        rc = hvm_copy_to_guest_phys(last_addr, mfn_to_virt(initrd->mod_start),
+        rc = hvm_copy_to_guest_phys(last_addr, maddr_to_virt(initrd->start),
                                     initrd_len, v);
         if ( rc )
         {
@@ -736,9 +735,9 @@ static int __init pvh_load_kernel(struct domain *d, const module_t *image,
         mod.paddr = last_addr;
         mod.size = initrd_len;
         last_addr += ROUNDUP(initrd_len, elf_64bit(&elf) ? 8 : 4);
-        if ( initrd->string )
+        if ( initrd->cmdline )
         {
-            char *str = __va(initrd->string);
+            char *str = __va((unsigned long)initrd->cmdline);
             size_t len = strlen(str) + 1;
 
             rc = hvm_copy_to_guest_phys(last_addr, str, len, v);
@@ -1346,8 +1345,8 @@ int __init dom0_construct_pvh(
         return rc;
     }
 
-    rc = pvh_load_kernel(d, image->mod, image->headroom, initrd->mod,
-                         bootstrap_map_bm(image), cmdline, &entry, &start_info);
+    rc = pvh_load_kernel(d, image, initrd, bootstrap_map_bm(image), cmdline,
+                         &entry, &start_info);
     if ( rc )
     {
         printk("Failed to load Dom0 kernel\n");
-- 
2.30.2

[PATCH v5 33/44] x86/boot: convert initial_images to struct boot_module

[Daniel P. Smith]

The variable initial_images is used for tracking the boot modules passed in by
the boot loader. Convert to a struct boot_module and adjust the code that uses
it accordingly.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index d5916e85f68e..30a139074833 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -276,7 +276,7 @@ custom_param("acpi", parse_acpi_param);
 
 static const char *cmdline_cook(const char *p, const char *loader_name);
 
-static const module_t *__initdata initial_images;
+static const struct boot_module *__initdata initial_images;
 
 struct boot_info __initdata xen_boot_info;
 
@@ -336,8 +336,9 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
 
     for ( nr = i = 0; i < bi->nr_modules; ++i )
     {
-        unsigned long start = initial_images[i].mod_start;
-        unsigned long end = start + PFN_UP(initial_images[i].mod_end);
+        unsigned long start = initial_images[i].mod->mod_start;
+        unsigned long end = start +
+                            PFN_UP(initial_images[i].mod->mod_end);
 
         if ( end > node_start && node_end > start )
             nr += min(node_end, end) - max(node_start, start);
@@ -353,10 +354,12 @@ void __init discard_initial_images(void)
 
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        uint64_t start = (uint64_t)initial_images[i].mod_start << PAGE_SHIFT;
+        uint64_t start =
+            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
 
         init_domheap_pages(start,
-                           start + PAGE_ALIGN(initial_images[i].mod_end));
+                           start +
+                           PAGE_ALIGN(initial_images[i].mod->mod_end));
     }
 
     bi->nr_modules = 0;
@@ -1380,7 +1383,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
     kexec_reserve_area();
 
-    initial_images = bi->mods[0].mod;
+    initial_images = bi->mods;
 
     for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
     {
-- 
2.30.2

[PATCH v5 34/44] x86/boot: drop the use of initial_images unit global

[Daniel P. Smith]

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 30a139074833..b3b6e6f38622 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -276,8 +276,6 @@ custom_param("acpi", parse_acpi_param);
 
 static const char *cmdline_cook(const char *p, const char *loader_name);
 
-static const struct boot_module *__initdata initial_images;
-
 struct boot_info __initdata xen_boot_info;
 
 static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
@@ -336,9 +334,9 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
 
     for ( nr = i = 0; i < bi->nr_modules; ++i )
     {
-        unsigned long start = initial_images[i].mod->mod_start;
+        unsigned long start = bi->mods[i].mod->mod_start;
         unsigned long end = start +
-                            PFN_UP(initial_images[i].mod->mod_end);
+                            PFN_UP(bi->mods[i].mod->mod_end);
 
         if ( end > node_start && node_end > start )
             nr += min(node_end, end) - max(node_start, start);
@@ -355,15 +353,14 @@ void __init discard_initial_images(void)
     for ( i = 0; i < bi->nr_modules; ++i )
     {
         uint64_t start =
-            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
+            (uint64_t)bi->mods[i].mod->mod_start << PAGE_SHIFT;
 
         init_domheap_pages(start,
                            start +
-                           PAGE_ALIGN(initial_images[i].mod->mod_end));
+                           PAGE_ALIGN(bi->mods[i].mod->mod_end));
     }
 
     bi->nr_modules = 0;
-    initial_images = NULL;
 }
 
 static void __init init_idle_domain(void)
@@ -1383,8 +1380,6 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
     kexec_reserve_area();
 
-    initial_images = bi->mods;
-
     for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
     {
         if ( bi->mods[i].mod->mod_start & (PAGE_SIZE - 1) )
-- 
2.30.2

[PATCH v5 35/44] x86/boot: remove usage of mod_end by discard_initial_images

[Daniel P. Smith]

This eliminates usage of early_mod by discard_initial_images

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index b3b6e6f38622..e9e3da3204f1 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -352,12 +352,10 @@ void __init discard_initial_images(void)
 
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        uint64_t start =
-            (uint64_t)bi->mods[i].mod->mod_start << PAGE_SHIFT;
+        uint64_t start = bi->mods[i].start;
 
         init_domheap_pages(start,
-                           start +
-                           PAGE_ALIGN(bi->mods[i].mod->mod_end));
+                           start + PAGE_ALIGN(bi->mods[i].size));
     }
 
     bi->nr_modules = 0;
-- 
2.30.2

[PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Daniel P. Smith]

Any direct usages of struct mod have been transitioned, remove the remaining
references to early_mod fields.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 31 +++++++++++--------------------
 1 file changed, 11 insertions(+), 20 deletions(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e9e3da3204f1..0ffe8d3ff8dd 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -334,9 +334,8 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
 
     for ( nr = i = 0; i < bi->nr_modules; ++i )
     {
-        unsigned long start = bi->mods[i].mod->mod_start;
-        unsigned long end = start +
-                            PFN_UP(bi->mods[i].mod->mod_end);
+        unsigned long start = bi->mods[i].start;
+        unsigned long end = start + PFN_UP(bi->mods[i].size);
 
         if ( end > node_start && node_end > start )
             nr += min(node_end, end) - max(node_start, start);
@@ -664,8 +663,8 @@ static uint64_t __init consider_modules(
 
     for ( i = 0; i < nr_mods ; ++i )
     {
-        uint64_t start = (uint64_t)pfn_to_paddr(mods[i].mod->mod_start);
-        uint64_t end = start + PAGE_ALIGN(mods[i].mod->mod_end);
+        uint64_t start = (uint64_t)mods[i].start;
+        uint64_t end = start + PAGE_ALIGN(mods[i].size);
 
         if ( i == this_mod )
             continue;
@@ -1380,10 +1379,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
     {
-        if ( bi->mods[i].mod->mod_start & (PAGE_SIZE - 1) )
+        if ( bi->mods[i].start & (PAGE_SIZE - 1) )
             panic("Bootloader didn't honor module alignment request\n");
-        bi->mods[i].mod->mod_end -= bi->mods[i].mod->mod_start;
-        bi->mods[i].mod->mod_start >>= PAGE_SHIFT;
     }
 
     /*
@@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
          */
         bi->mods[xen].start = virt_to_mfn(_stext);
         bi->mods[xen].size = __2M_rwdata_end - _stext;
-
-        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
-        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
     }
 
-    bi->mods[0].headroom =
-        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
-                         bi->mods[0].mod->mod_end);
-
-    bootstrap_map(NULL);
+    bi->mods[0].headroom = bzimage_headroom(
+                        bootstrap_map_bm(&bi->mods[0]),
+                        bi->mods[0].size);
+    bootstrap_map_bm(NULL);
 
 #ifndef highmem_start
     /* Don't allow split below 4Gb. */
@@ -1518,9 +1511,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
             {
                 move_memory(end - size + bm->headroom, bm->start, bm->size);
                 bm->start = (end - size);
-                bm->mod->mod_start = paddr_to_pfn(bm->start);
                 bm->size += bm->headroom;
-                bm->mod->mod_end = bm->size;
                 bm->flags |= BOOTMOD_FLAG_X86_RELOCATED;
             }
         }
@@ -1708,8 +1699,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
 
     for ( i = 0; i < bi->nr_modules; ++i )
     {
-        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
-                      paddr_to_pfn(bi->mods[i].mod->mod_start) +
+        set_pdx_range(paddr_to_pfn(bi->mods[i].start),
+                      paddr_to_pfn(bi->mods[i].start) +
                       PFN_UP(bi->mods[i].size));
         map_pages_to_xen(
             (unsigned long)maddr_to_virt(bi->mods[i].start),
-- 
2.30.2

[PATCH v5 37/44] x86/boot: remove mod from struct boot_module

[Daniel P. Smith]

With all references to mod field removed, remove the mod field from struct
boot_module.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootinfo.h | 3 ---
 xen/arch/x86/setup.c                | 3 ---
 2 files changed, 6 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 3b6bfbe88770..4cb722e8ba0a 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -8,7 +8,6 @@
 #ifndef __XEN_X86_BOOTINFO_H__
 #define __XEN_X86_BOOTINFO_H__
 
-#include <xen/multiboot.h>
 #include <xen/types.h>
 
 /* Max number of boot modules a bootloader can provide in addition to Xen */
@@ -25,8 +24,6 @@ enum bootmod_type {
 };
 
 struct boot_module {
-    /* Transitionary only */
-    module_t *mod;
     /*
      * A boot module may contain a compressed kernel that Xen will need space
      * reserved, into which it will be decompressed.
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 0ffe8d3ff8dd..3604c8fbe40a 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -308,8 +308,6 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
      */
     for ( i = 0; i <= bi->nr_modules; i++ )
     {
-        bi->mods[i].mod = &mods[i];
-
         bi->mods[i].cmdline = (char *)(paddr_t)mods[i].string;
 
         bi->mods[i].start = (paddr_t)mods[i].mod_start;
@@ -319,7 +317,6 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     /* map the last mb module for xen entry */
     bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
     bi->mods[bi->nr_modules].flags |= BOOTMOD_FLAG_X86_CONSUMED;
-    bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
 
     return bi;
 }
-- 
2.30.2

[PATCH v5 38/44] x86/boot: introduce boot domain

[Daniel P. Smith]

To begin moving toward allowing the hypervisor to construct more than one
domain at boot, a container is needed for a domain's build information.

Introduce a new header, <xen/asm/bootdomain.h>, that contains the initial
struct boot_domain that encapsulate the build information for a domain.

No functional change intended.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootdomain.h | 28 +++++++++++++++++++++++++++
 xen/arch/x86/include/asm/bootinfo.h   |  5 +++++
 xen/arch/x86/setup.c                  | 24 ++++++++---------------
 3 files changed, 41 insertions(+), 16 deletions(-)
 create mode 100644 xen/arch/x86/include/asm/bootdomain.h

diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
new file mode 100644
index 000000000000..4285223ac5ab
--- /dev/null
+++ b/xen/arch/x86/include/asm/bootdomain.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (c) 2024 Apertus Solutions, LLC
+ * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
+ * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
+ */
+
+#ifndef __XEN_X86_BOOTDOMAIN_H__
+#define __XEN_X86_BOOTDOMAIN_H__
+
+struct boot_module;
+
+struct boot_domain {
+    struct boot_module *kernel;
+    struct boot_module *ramdisk;
+};
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
index 4cb722e8ba0a..7b4da6cad1ed 100644
--- a/xen/arch/x86/include/asm/bootinfo.h
+++ b/xen/arch/x86/include/asm/bootinfo.h
@@ -9,10 +9,14 @@
 #define __XEN_X86_BOOTINFO_H__
 
 #include <xen/types.h>
+#include <asm/bootdomain.h>
 
 /* Max number of boot modules a bootloader can provide in addition to Xen */
 #define MAX_NR_BOOTMODS 63
 
+/* Max number of boot domains that Xen can construct */
+#define MAX_NR_BOOTDOMS 1
+
 /* Boot module binary type / purpose */
 enum bootmod_type {
     BOOTMOD_UNKNOWN,
@@ -55,6 +59,7 @@ struct boot_info {
 
     unsigned int nr_modules;
     struct boot_module mods[MAX_NR_BOOTMODS + 1];
+    struct boot_domain domains[MAX_NR_BOOTDOMS];
 };
 
 static inline int __init next_boot_module_index(
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 3604c8fbe40a..ad4a1f473f6d 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -957,20 +957,10 @@ static struct domain *__init create_dom0(struct boot_info *bi)
             .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
         },
     };
-    int mod_idx = first_boot_module_index(bi, BOOTMOD_RAMDISK);
-    struct boot_module *image, *initrd;
+    struct boot_domain *bd = &bi->domains[0];
     struct domain *d;
     domid_t domid;
 
-    /* Map boot_module to mb1 module for dom0 */
-    image = &bi->mods[0];
-
-    /* Map boot_module to mb1 module for initrd */
-    if ( mod_idx < 0 )
-        initrd = NULL;
-    else
-        initrd = &bi->mods[mod_idx];
-
     if ( opt_dom0_pvh )
     {
         dom0_cfg.flags |= (XEN_DOMCTL_CDF_hvm |
@@ -996,11 +986,11 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         panic("Error creating d%uv0\n", domid);
 
     /* Grab the DOM0 command line. */
-    if ( image->cmdline || bi->kextra )
+    if ( bd->kernel->cmdline || bi->kextra )
     {
-        if ( image->cmdline )
-            safe_strcpy(cmdline,
-                        cmdline_cook(__va((unsigned long)image->cmdline),
+        if ( bd->kernel->cmdline )
+            safe_strcpy(cmdline, cmdline_cook(
+                        __va((unsigned long)bd->kernel->cmdline),
                         bi->loader));
 
         if ( bi->kextra )
@@ -1024,7 +1014,7 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         }
     }
 
-    if ( construct_dom0(d, image, initrd, cmdline) != 0 )
+    if ( construct_dom0(d, bd->kernel, bd->ramdisk, cmdline) != 0 )
         panic("Could not construct domain 0\n");
 
     return d;
@@ -1204,6 +1194,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     /* Dom0 kernel is always first */
     bi->mods[0].type = BOOTMOD_KERNEL;
     bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;
+    bi->domains[0].kernel = &bi->mods[0];
 
     if ( pvh_boot )
     {
@@ -2093,6 +2084,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
     {
         bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
         bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
+        bi->domains[0].ramdisk = &bi->mods[initrdidx];
         if ( first_boot_module_index(bi, BOOTMOD_UNKNOWN) >= 0 )
             printk(XENLOG_WARNING
                    "Multiple initrd candidates, picking module #%u\n",
-- 
2.30.2

[PATCH v5 39/44] x86/boot: introduce domid field to struct boot_domain

[Daniel P. Smith]

Add a domid field to struct boot_domain to hold the assigned domain id for the
domain. During initialization, ensure all instances of struct boot_domain have
the invalid domid to ensure that the domid must be set either by convention or
configuration.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootdomain.h |  2 ++
 xen/arch/x86/setup.c                  | 12 +++++++-----
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
index 4285223ac5ab..d6264d554dba 100644
--- a/xen/arch/x86/include/asm/bootdomain.h
+++ b/xen/arch/x86/include/asm/bootdomain.h
@@ -11,6 +11,8 @@
 struct boot_module;
 
 struct boot_domain {
+    domid_t domid;
+
     struct boot_module *kernel;
     struct boot_module *ramdisk;
 };
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index ad4a1f473f6d..a1204b2bd594 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -318,6 +318,9 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
     bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
     bi->mods[bi->nr_modules].flags |= BOOTMOD_FLAG_X86_CONSUMED;
 
+    for ( i = 0; i < MAX_NR_BOOTDOMS; i++ )
+        bi->domains[i].domid = DOMID_INVALID;
+
     return bi;
 }
 
@@ -959,7 +962,6 @@ static struct domain *__init create_dom0(struct boot_info *bi)
     };
     struct boot_domain *bd = &bi->domains[0];
     struct domain *d;
-    domid_t domid;
 
     if ( opt_dom0_pvh )
     {
@@ -975,15 +977,15 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
 
     /* Create initial domain.  Not d0 for pvshim. */
-    domid = get_initial_domain_id();
-    d = domain_create(domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
+    bd->domid = get_initial_domain_id();
+    d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
     if ( IS_ERR(d) )
-        panic("Error creating d%u: %ld\n", domid, PTR_ERR(d));
+        panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
 
     init_dom0_cpuid_policy(d);
 
     if ( alloc_dom0_vcpu0(d) == NULL )
-        panic("Error creating d%uv0\n", domid);
+        panic("Error creating d%uv0\n", bd->domid);
 
     /* Grab the DOM0 command line. */
     if ( bd->kernel->cmdline || bi->kextra )
-- 
2.30.2

[PATCH v5 40/44] x86/boot: add cmdline to struct boot_domain

[Daniel P. Smith]

Add a container for the "cooked" command line for a domain.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootdomain.h |  4 ++++
 xen/arch/x86/setup.c                  | 18 ++++++++----------
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
index d6264d554dba..00f7d9267965 100644
--- a/xen/arch/x86/include/asm/bootdomain.h
+++ b/xen/arch/x86/include/asm/bootdomain.h
@@ -8,9 +8,13 @@
 #ifndef __XEN_X86_BOOTDOMAIN_H__
 #define __XEN_X86_BOOTDOMAIN_H__
 
+#include <public/xen.h>
+
 struct boot_module;
 
 struct boot_domain {
+    char cmdline[MAX_GUEST_CMDLINE];
+
     domid_t domid;
 
     struct boot_module *kernel;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index a1204b2bd594..f250638edf09 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -947,8 +947,6 @@ static unsigned int __init copy_bios_e820(struct e820entry *map, unsigned int li
 
 static struct domain *__init create_dom0(struct boot_info *bi)
 {
-    static char __initdata cmdline[MAX_GUEST_CMDLINE];
-
     struct xen_domctl_createdomain dom0_cfg = {
         .flags = IS_ENABLED(CONFIG_TBOOT) ? XEN_DOMCTL_CDF_s3_integrity : 0,
         .max_evtchn_port = -1,
@@ -991,17 +989,17 @@ static struct domain *__init create_dom0(struct boot_info *bi)
     if ( bd->kernel->cmdline || bi->kextra )
     {
         if ( bd->kernel->cmdline )
-            safe_strcpy(cmdline, cmdline_cook(
+            safe_strcpy(bd->cmdline, cmdline_cook(
                         __va((unsigned long)bd->kernel->cmdline),
                         bi->loader));
 
         if ( bi->kextra )
             /* kextra always includes exactly one leading space. */
-            safe_strcat(cmdline, bi->kextra);
+            safe_strcat(bd->cmdline, bi->kextra);
 
         /* Append any extra parameters. */
-        if ( skip_ioapic_setup && !strstr(cmdline, "noapic") )
-            safe_strcat(cmdline, " noapic");
+        if ( skip_ioapic_setup && !strstr(bd->cmdline, "noapic") )
+            safe_strcat(bd->cmdline, " noapic");
 
         if ( (strlen(acpi_param) == 0) && acpi_disabled )
         {
@@ -1009,14 +1007,14 @@ static struct domain *__init create_dom0(struct boot_info *bi)
             safe_strcpy(acpi_param, "off");
         }
 
-        if ( (strlen(acpi_param) != 0) && !strstr(cmdline, "acpi=") )
+        if ( (strlen(acpi_param) != 0) && !strstr(bd->cmdline, "acpi=") )
         {
-            safe_strcat(cmdline, " acpi=");
-            safe_strcat(cmdline, acpi_param);
+            safe_strcat(bd->cmdline, " acpi=");
+            safe_strcat(bd->cmdline, acpi_param);
         }
     }
 
-    if ( construct_dom0(d, bd->kernel, bd->ramdisk, cmdline) != 0 )
+    if ( construct_dom0(d, bd->kernel, bd->ramdisk, bd->cmdline) != 0 )
         panic("Could not construct domain 0\n");
 
     return d;
-- 
2.30.2

[PATCH v5 41/44] x86/boot: add struct domain to struct boot_domain

[Daniel P. Smith]

Store a reference to the created domain in struct boot_domain.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/include/asm/bootdomain.h |  3 +++
 xen/arch/x86/setup.c                  | 15 +++++++--------
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
index 00f7d9267965..2322c459e36a 100644
--- a/xen/arch/x86/include/asm/bootdomain.h
+++ b/xen/arch/x86/include/asm/bootdomain.h
@@ -10,6 +10,7 @@
 
 #include <public/xen.h>
 
+struct domain;
 struct boot_module;
 
 struct boot_domain {
@@ -19,6 +20,8 @@ struct boot_domain {
 
     struct boot_module *kernel;
     struct boot_module *ramdisk;
+
+    struct domain *d;
 };
 
 #endif
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index f250638edf09..e6a231bd2d42 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -959,7 +959,6 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         },
     };
     struct boot_domain *bd = &bi->domains[0];
-    struct domain *d;
 
     if ( opt_dom0_pvh )
     {
@@ -976,13 +975,13 @@ static struct domain *__init create_dom0(struct boot_info *bi)
 
     /* Create initial domain.  Not d0 for pvshim. */
     bd->domid = get_initial_domain_id();
-    d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
-    if ( IS_ERR(d) )
-        panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
+    bd->d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
+    if ( IS_ERR(bd->d) )
+        panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(bd->d));
 
-    init_dom0_cpuid_policy(d);
+    init_dom0_cpuid_policy(bd->d);
 
-    if ( alloc_dom0_vcpu0(d) == NULL )
+    if ( alloc_dom0_vcpu0(bd->d) == NULL )
         panic("Error creating d%uv0\n", bd->domid);
 
     /* Grab the DOM0 command line. */
@@ -1014,10 +1013,10 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         }
     }
 
-    if ( construct_dom0(d, bd->kernel, bd->ramdisk, bd->cmdline) != 0 )
+    if ( construct_dom0(bd->d, bd->kernel, bd->ramdisk, bd->cmdline) != 0 )
         panic("Could not construct domain 0\n");
 
-    return d;
+    return bd->d;
 }
 
 /* How much of the directmap is prebuilt at compile time. */
-- 
2.30.2

[PATCH v5 42/44] x86/boot: convert construct_dom0 to struct boot_domain

[Daniel P. Smith]

A struct boot_domain now encapsulates the domain reference, kernel, ramdisk,
and command line for the domain being constructed. As a result of this
encapsulation, construct_dom0 can now take a single struct boot_domain instead
of these four parameters.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c        | 19 +++++++++----------
 xen/arch/x86/include/asm/setup.h |  4 +---
 xen/arch/x86/setup.c             |  2 +-
 3 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 71b2e3afc1a1..e552f2e9abef 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -597,22 +597,21 @@ int __init dom0_setup_permissions(struct domain *d)
     return rc;
 }
 
-int __init construct_dom0(struct domain *d, const struct boot_module *image,
-                          struct boot_module *initrd, const char *cmdline)
+int __init construct_dom0(struct boot_domain *bd)
 {
     int rc;
 
     /* Sanity! */
-    BUG_ON(!pv_shim && d->domain_id != 0);
-    BUG_ON(d->vcpu[0] == NULL);
-    BUG_ON(d->vcpu[0]->is_initialised);
+    BUG_ON(!pv_shim && bd->d->domain_id != 0);
+    BUG_ON(bd->d->vcpu[0] == NULL);
+    BUG_ON(bd->d->vcpu[0]->is_initialised);
 
     process_pending_softirqs();
 
-    if ( is_hvm_domain(d) )
-        rc = dom0_construct_pvh(d, image, initrd, cmdline);
-    else if ( is_pv_domain(d) )
-        rc = dom0_construct_pv(d, image, initrd, cmdline);
+    if ( is_hvm_domain(bd->d) )
+        rc = dom0_construct_pvh(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
+    else if ( is_pv_domain(bd->d) )
+        rc = dom0_construct_pv(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
     else
         panic("Cannot construct Dom0. No guest interface available\n");
 
@@ -620,7 +619,7 @@ int __init construct_dom0(struct domain *d, const struct boot_module *image,
         return rc;
 
     /* Sanity! */
-    BUG_ON(!d->vcpu[0]->is_initialised);
+    BUG_ON(!bd->d->vcpu[0]->is_initialised);
 
     return 0;
 }
diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index 04bf0e62a9d7..6fdb392432ad 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -26,9 +26,7 @@ void subarch_init_memory(void);
 
 void init_IRQ(void);
 
-int construct_dom0(
-    struct domain *d, const struct boot_module *image,
-    struct boot_module *initrd, const char *cmdline);
+int construct_dom0(struct boot_domain *d);
 void setup_io_bitmap(struct domain *d);
 
 extern struct boot_info xen_boot_info;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index e6a231bd2d42..6ea825b64541 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1013,7 +1013,7 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         }
     }
 
-    if ( construct_dom0(bd->d, bd->kernel, bd->ramdisk, bd->cmdline) != 0 )
+    if ( construct_dom0(bd) != 0 )
         panic("Could not construct domain 0\n");
 
     return bd->d;
-- 
2.30.2

[PATCH v5 43/44] x86/boot: convert dom0_construct_pv to struct boot_domain

[Daniel P. Smith]

With construct_dom0 consuming struct boot_domain, continue passing the
structure down to dom0_construct_pv.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             |  2 +-
 xen/arch/x86/include/asm/dom0_build.h |  5 ++--
 xen/arch/x86/pv/dom0_build.c          | 43 ++++++++++++---------------
 3 files changed, 22 insertions(+), 28 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index e552f2e9abef..8beb33032940 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -611,7 +611,7 @@ int __init construct_dom0(struct boot_domain *bd)
     if ( is_hvm_domain(bd->d) )
         rc = dom0_construct_pvh(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
     else if ( is_pv_domain(bd->d) )
-        rc = dom0_construct_pv(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
+        rc = dom0_construct_pv(bd);
     else
         panic("Cannot construct Dom0. No guest interface available\n");
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 8f7b37f3d308..60e9cb21f14d 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -4,6 +4,7 @@
 #include <xen/libelf.h>
 #include <xen/sched.h>
 
+#include <asm/bootinfo.h>
 #include <asm/setup.h>
 
 extern unsigned int dom0_memflags;
@@ -13,9 +14,7 @@ unsigned long dom0_compute_nr_pages(struct domain *d,
                                     unsigned long initrd_len);
 int dom0_setup_permissions(struct domain *d);
 
-int dom0_construct_pv(
-    struct domain *d, const struct boot_module *image,
-    struct boot_module *initrd, const char *cmdline);
+int dom0_construct_pv(struct boot_domain *bd);
 
 int dom0_construct_pvh(
     struct domain *d, const struct boot_module *image,
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 16b8c1e40998..34974aa7cd56 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -354,10 +354,7 @@ static struct page_info * __init alloc_chunk(struct domain *d,
     return page;
 }
 
-static int __init dom0_construct(struct domain *d,
-                                 const struct boot_module *image,
-                                 struct boot_module *initrd,
-                                 const char *cmdline)
+static int __init dom0_construct(struct boot_domain *bd)
 {
     int i, rc, order, machine;
     bool compatible, compat;
@@ -373,11 +370,12 @@ static int __init dom0_construct(struct domain *d,
     struct page_info *page = NULL;
     unsigned int flush_flags = 0;
     start_info_t *si;
-    struct vcpu *v = d->vcpu[0];
-    void *image_base = bootstrap_map_bm(image);
-    unsigned long image_len = image->size;
-    void *image_start = image_base + image->headroom;
-    unsigned long initrd_len = initrd ? initrd->size : 0;
+    struct domain *d = bd->d;
+    struct vcpu *v = bd->d->vcpu[0];
+    void *image_base = bootstrap_map_bm(bd->kernel);
+    unsigned long image_len = bd->kernel->size;
+    void *image_start = image_base + bd->kernel->headroom;
+    unsigned long initrd_len = bd->ramdisk ? bd->ramdisk->size : 0;
     l4_pgentry_t *l4tab = NULL, *l4start = NULL;
     l3_pgentry_t *l3tab = NULL, *l3start = NULL;
     l2_pgentry_t *l2tab = NULL, *l2start = NULL;
@@ -613,7 +611,7 @@ static int __init dom0_construct(struct domain *d,
         initrd_pfn = vinitrd_start ?
                      (vinitrd_start - v_start) >> PAGE_SHIFT :
                      domain_tot_pages(d);
-        initrd_mfn = maddr_to_mfn(initrd->start);
+        initrd_mfn = maddr_to_mfn(bd->ramdisk->start);
         mfn = mfn_x(initrd_mfn);
         count = PFN_UP(initrd_len);
         if ( d->arch.physaddr_bitsize &&
@@ -629,13 +627,13 @@ static int __init dom0_construct(struct domain *d,
                     free_domheap_pages(page, order);
                     page += 1UL << order;
                 }
-            memcpy(page_to_virt(page), maddr_to_virt(initrd->start),
+            memcpy(page_to_virt(page), maddr_to_virt(bd->ramdisk->start),
                    initrd_len);
-            mpt_alloc = initrd->start;
+            mpt_alloc = bd->ramdisk->start;
             init_domheap_pages(mpt_alloc,
                                mpt_alloc + PAGE_ALIGN(initrd_len));
             initrd_mfn = page_to_mfn(page);
-            initrd->start = mfn_to_maddr(initrd_mfn);
+            bd->ramdisk->start = mfn_to_maddr(initrd_mfn);
         }
         else
         {
@@ -643,7 +641,7 @@ static int __init dom0_construct(struct domain *d,
                 if ( assign_pages(mfn_to_page(_mfn(mfn++)), 1, d, 0) )
                     BUG();
         }
-        initrd->size = 0;
+        bd->ramdisk->size = 0;
 
         iommu_memory_setup(d, "initrd", mfn_to_page(initrd_mfn),
                            PFN_UP(initrd_len), &flush_flags);
@@ -655,9 +653,9 @@ static int __init dom0_construct(struct domain *d,
     if ( domain_tot_pages(d) < nr_pages )
         printk(" (%lu pages to be allocated)",
                nr_pages - domain_tot_pages(d));
-    if ( initrd )
+    if ( bd->ramdisk )
     {
-        mpt_alloc = initrd->start;
+        mpt_alloc = bd->ramdisk->start;
         printk("\n Init. ramdisk: %"PRIpaddr"->%"PRIpaddr,
                mpt_alloc, mpt_alloc + initrd_len);
     }
@@ -885,7 +883,7 @@ static int __init dom0_construct(struct domain *d,
         if ( pfn >= initrd_pfn )
         {
             if ( pfn < initrd_pfn + PFN_UP(initrd_len) )
-                mfn = paddr_to_pfn(initrd->start) + (pfn - initrd_pfn);
+                mfn = paddr_to_pfn(bd->ramdisk->start) + (pfn - initrd_pfn);
             else
                 mfn -= PFN_UP(initrd_len);
         }
@@ -955,8 +953,8 @@ static int __init dom0_construct(struct domain *d,
     }
 
     memset(si->cmd_line, 0, sizeof(si->cmd_line));
-    if ( cmdline != NULL )
-        strlcpy((char *)si->cmd_line, cmdline, sizeof(si->cmd_line));
+    if ( bd->cmdline[0] != '\0' )
+        strlcpy((char *)si->cmd_line, bd->cmdline, sizeof(si->cmd_line));
 
 #ifdef CONFIG_VIDEO
     if ( !pv_shim && fill_console_start_info((void *)(si + 1)) )
@@ -1053,10 +1051,7 @@ out:
     return rc;
 }
 
-int __init dom0_construct_pv(struct domain *d,
-                             const struct boot_module *image,
-                             struct boot_module *initrd,
-                             const char *cmdline)
+int __init dom0_construct_pv(struct boot_domain *bd)
 {
     int rc;
 
@@ -1073,7 +1068,7 @@ int __init dom0_construct_pv(struct domain *d,
         write_cr4(read_cr4() & ~X86_CR4_SMAP);
     }
 
-    rc = dom0_construct(d, image, initrd, cmdline);
+    rc = dom0_construct(bd);
 
     if ( boot_cpu_has(X86_FEATURE_XEN_SMAP) )
     {
-- 
2.30.2

[PATCH v5 44/44] x86/boot: convert dom0_construct_pvh to struct boot_domain

[Daniel P. Smith]

With construct_dom0 consuming struct boot_domain, continue passing the
structure down to dom0_construct_pvh.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             |  2 +-
 xen/arch/x86/hvm/dom0_build.c         | 31 +++++++++++++--------------
 xen/arch/x86/include/asm/dom0_build.h |  4 +---
 3 files changed, 17 insertions(+), 20 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 8beb33032940..6a21fd46d5a3 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -609,7 +609,7 @@ int __init construct_dom0(struct boot_domain *bd)
     process_pending_softirqs();
 
     if ( is_hvm_domain(bd->d) )
-        rc = dom0_construct_pvh(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
+        rc = dom0_construct_pvh(bd);
     else if ( is_pv_domain(bd->d) )
         rc = dom0_construct_pv(bd);
     else
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index a3fd5e762dc4..755e257cdf30 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -1299,25 +1299,23 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
     }
 }
 
-int __init dom0_construct_pvh(
-    struct domain *d, const struct boot_module *image,
-    struct boot_module *initrd, const char *cmdline)
+int __init dom0_construct_pvh(const struct boot_domain *bd)
 {
     paddr_t entry, start_info;
     int rc;
 
-    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", d->domain_id);
+    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", bd->domid);
 
-    if ( is_hardware_domain(d) )
+    if ( is_hardware_domain(bd->d) )
     {
         /*
          * Setup permissions early so that calls to add MMIO regions to the
          * p2m as part of vPCI setup don't fail due to permission checks.
          */
-        rc = dom0_setup_permissions(d);
+        rc = dom0_setup_permissions(bd->d);
         if ( rc )
         {
-            printk("%pd unable to setup permissions: %d\n", d, rc);
+            printk("%pd unable to setup permissions: %d\n", bd->d, rc);
             return rc;
         }
     }
@@ -1327,25 +1325,26 @@ int __init dom0_construct_pvh(
      * initialization so the iommu code can fetch the MMCFG regions used by the
      * domain.
      */
-    pvh_setup_mmcfg(d);
+    pvh_setup_mmcfg(bd->d);
 
     /*
      * Craft dom0 physical memory map and set the paging allocation. This must
      * be done before the iommu initializion, since iommu initialization code
      * will likely add mappings required by devices to the p2m (ie: RMRRs).
      */
-    pvh_init_p2m(d);
+    pvh_init_p2m(bd->d);
 
-    iommu_hwdom_init(d);
+    iommu_hwdom_init(bd->d);
 
-    rc = pvh_populate_p2m(d);
+    rc = pvh_populate_p2m(bd->d);
     if ( rc )
     {
         printk("Failed to setup Dom0 physical memory map\n");
         return rc;
     }
 
-    rc = pvh_load_kernel(d, image, initrd, bootstrap_map_bm(image), cmdline,
+    rc = pvh_load_kernel(bd->d, bd->kernel, bd->ramdisk,
+                         bootstrap_map_bm(bd->kernel), bd->cmdline,
                          &entry, &start_info);
     if ( rc )
     {
@@ -1353,14 +1352,14 @@ int __init dom0_construct_pvh(
         return rc;
     }
 
-    rc = pvh_setup_cpus(d, entry, start_info);
+    rc = pvh_setup_cpus(bd->d, entry, start_info);
     if ( rc )
     {
         printk("Failed to setup Dom0 CPUs: %d\n", rc);
         return rc;
     }
 
-    rc = pvh_setup_acpi(d, start_info);
+    rc = pvh_setup_acpi(bd->d, start_info);
     if ( rc )
     {
         printk("Failed to setup Dom0 ACPI tables: %d\n", rc);
@@ -1369,8 +1368,8 @@ int __init dom0_construct_pvh(
 
     if ( opt_dom0_verbose )
     {
-        printk("Dom%u memory map:\n", d->domain_id);
-        print_e820_memory_map(d->arch.e820, d->arch.nr_e820);
+        printk("Dom%u memory map:\n", bd->domid);
+        print_e820_memory_map(bd->d->arch.e820, bd->d->arch.nr_e820);
     }
 
     return 0;
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 60e9cb21f14d..adbe90bfd034 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -16,9 +16,7 @@ int dom0_setup_permissions(struct domain *d);
 
 int dom0_construct_pv(struct boot_domain *bd);
 
-int dom0_construct_pvh(
-    struct domain *d, const struct boot_module *image,
-    struct boot_module *initrd, const char *cmdline);
+int dom0_construct_pvh(const struct boot_domain *bd);
 
 unsigned long dom0_paging_pages(const struct domain *d,
                                 unsigned long nr_pages);
-- 
2.30.2

Re: [PATCH v5 01/44] x86/boot: move x86 boot module counting into a new boot_info struct

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> From: Christopher Clark <christopher.w.clark@gmail.com>
> 
> An initial step towards a non-multiboot internal representation of boot
> modules for common code, starting with x86 setup and converting the fields
> that are accessed for the startup calculations.
> 
> Introduce a new header, <asm/bootinfo.h>, and populate it with a new boot_info
> structure initially containing a count of the number of boot modules.
> 
> No functional change intended.
> 
> Signed-off-by: Christopher Clark <christopher.w.clark@gmail.com>
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h | 29 +++++++++++++
>   xen/arch/x86/include/asm/setup.h    |  2 +
>   xen/arch/x86/setup.c                | 64 ++++++++++++++++++-----------
>   3 files changed, 71 insertions(+), 24 deletions(-)
>   create mode 100644 xen/arch/x86/include/asm/bootinfo.h
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> new file mode 100644
> index 000000000000..a649500ee3a2
> --- /dev/null
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -0,0 +1,29 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +/*
> + * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
> + * Copyright (c) 2024 Apertus Solutions, LLC
> + * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
> + */
> +
> +#ifndef __XEN_X86_BOOTINFO_H__
> +#define __XEN_X86_BOOTINFO_H__

I haven't been following closely, but I think if we follow Frediano's 
naming scheme, it would be:
ASM__X86__BOOTINFO_H

With that (or whatever it should be),
Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

> +
> +/*
> + * Xen internal representation of information provided by the
> + * bootloader/environment, or derived from the information.
> + */

I guess fine for now.  Should probably be expanded when it starts 
containing Hyperlaunch domain configs.

> +struct boot_info {
> +    unsigned int nr_modules;
> +};
> +
> +#endif /* __XEN_X86_BOOTINFO_H__ */

Regards,
Jason

Re: [PATCH v5 02/44] x86/boot: move boot loader name to boot info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Transition the incoming boot loader name to be held in struct boot_info.
> 
> No functional change intended.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 03/44] x86/boot: move cmdline to boot info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Transition Xen's command line to being held in struct boot_info.
> 
> No functional change intended.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---

>   #endif /* __XEN_X86_BOOTINFO_H__ */
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index aafc098ca268..0921f296075f 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -274,6 +274,8 @@ static int __init cf_check parse_acpi_param(const char *s)
>   }
>   custom_param("acpi", parse_acpi_param);
>   
> +static const char *cmdline_cook(const char *p, const char *loader_name);

Is there a reason not to move cmdline_cook() (and loader_is_grub2()) 
earlier to avoid this forward declaration?

Everything else looks okay.

Regards,
Jason

Re: [PATCH v5 04/44] x86/boot: move mmap info to boot info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Transition the memory map info to be held in struct boot_info.
> 
> No functional change intended.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 05/44] x86/boot: introduce struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This will introduce a new struct boot_module to provide a rich state
> representation around modules provided by the boot loader. Support is for 64
> boot modules, one held in reserve for Xen, and up to 63 can be provided by the
> boot loader. The array of struct boot_modules will be accessible via a
> reference held in struct boot_info.
> 
> A temporary `mod` parameter is included in struct boot_module to ease the
> transition from using Multiboot v1 structures over to struct boot_module. Once
> the transition is complete, the parameter will be dropped from the structure.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h | 14 ++++++++++++--
>   xen/arch/x86/setup.c                |  9 +++++++++
>   2 files changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> index 87d311ac1399..d19473d8941e 100644
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -8,20 +8,30 @@
>   #ifndef __XEN_X86_BOOTINFO_H__
>   #define __XEN_X86_BOOTINFO_H__
>   
> +#include <xen/multiboot.h>
>   #include <xen/types.h>
>   
> +/* Max number of boot modules a bootloader can provide in addition to Xen */
> +#define MAX_NR_BOOTMODS 63
> +
> +struct boot_module {
> +    /* Transitionary only */
> +    module_t *mod;
> +};
> +
>   /*
>    * Xen internal representation of information provided by the
>    * bootloader/environment, or derived from the information.
>    */
>   struct boot_info {
> -    unsigned int nr_modules;
> -

(You should probably re-work the other patches to insert ahead of this 
and avoid the movement.)

>       const char *loader;
>       const char *cmdline;
>   
>       paddr_t memmap_addr;
>       size_t memmap_length;
> +
> +    unsigned int nr_modules;
> +    struct boot_module mods[MAX_NR_BOOTMODS + 1];
>   };

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 06/44] x86/boot: convert consider_modules to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> To start transitioning consider_modules() over to struct boot_module, begin
> with taking the array of struct boot_modules but use the temporary struct
> element mod.
> 
> No functional change intended.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@gmail.com>

Re: [PATCH v5 07/44] x86/boot: move headroom to boot modules

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The purpose of struct boot_module is to encapsulate the state of boot module as
> it is processed by Xen. Locating boot module state struct boot_module reduces
> the number of global variables as well as the number of state variables that
> must be passed around. It also lays the groundwork for hyperlaunch mult-domain
> construction, where multiple instances of state variables like headroom will be
> needed.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h |  5 +++++
>   xen/arch/x86/setup.c                | 23 ++++++++++++++---------
>   2 files changed, 19 insertions(+), 9 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> index d19473d8941e..c7e6b4ebf0da 100644
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -17,6 +17,11 @@
>   struct boot_module {
>       /* Transitionary only */
>       module_t *mod;
> +    /*
> +     * A boot module may contain a compressed kernel that Xen will need space
> +     * reserved, into which it will be decompressed.

Maybe "Extra space, before the module data, for compressed kernel 
modules to be decompressed into."

And some ascii art could help:

[ headroom ][ compressed data ]
  <decompression>
[ decompressed data ]

(Not sure how to create a down arrow...)

> +     */
> +    unsigned long headroom;
>   };
>   
>   /*
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index ba9f110d98c6..dd82ca3d43e2 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1012,7 +1012,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>       struct boot_info *bi;
>       multiboot_info_t *mbi;
>       module_t *mod;
> -    unsigned long nr_pages, raw_max_page, modules_headroom, module_map[1];
> +    unsigned long nr_pages, raw_max_page, module_map[1];
>       int i, j, e820_warn = 0, bytes = 0;
>       unsigned long eb_start, eb_end;
>       bool acpi_boot_table_init_done = false, relocated = false;
> @@ -1371,7 +1371,10 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>           mod[bi->nr_modules].mod_end = __2M_rwdata_end - _stext;
>       }
>   
> -    modules_headroom = bzimage_headroom(bootstrap_map(mod), mod->mod_end);
> +    bi->mods[0].headroom =
> +        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
> +                         bi->mods[0].mod->mod_end);
> +
>       bootstrap_map(NULL);
>   
>   #ifndef highmem_start
> @@ -1456,8 +1459,10 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>                * decompressor overheads of mod[0] (the dom0 kernel).  When we
>                * move mod[0], we incorporate this as extra space at the start.
>                */
> -            unsigned long headroom = j ? 0 : modules_headroom;
> -            unsigned long size = PAGE_ALIGN(headroom + mod[j].mod_end);
> +            struct boot_module *bm = &bi->mods[j];
> +            unsigned long size;
> +
> +            size = PAGE_ALIGN(bm->headroom + mod[j].mod_end);

Just do
             unsigned long size = PAGE_ALIGN(bm->headroom + mod[j].mod_end);
?

>   
>               if ( mod[j].reserved )
>                   continue;

The rest looks good.

Regards,
Jason

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> To allow a slow conversion of x86 over to struct boot_module, start with
> replacing all references to struct mod to the early_mod element of struct
> boot_module. These serves twofold, first to allow the incremental transition
> from struct mod fields to struct boot_module fields. The second is to allow
> the conversion of function definitions from taking struct mod parameters to
> accepting struct boot_module as needed when a transitioned field will be
> accessed.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 61 ++++++++++++++++++++++++--------------------
>   1 file changed, 34 insertions(+), 27 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index dd82ca3d43e2..ba4bee6b93af 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1341,15 +1341,15 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>       set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
>       kexec_reserve_area();
>   
> -    initial_images = mod;
> +    initial_images = bi->mods[0].mod;

Isn't this wrong?
mod is the array of module_t * of *all* modules, but bi->mods[0].mod is 
a single module_t *?

>   
>       for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
>       {
> -        if ( mod[i].mod_start & (PAGE_SIZE - 1) )
> +        if ( bi->mods[i].mod->mod_start & (PAGE_SIZE - 1) )
>               panic("Bootloader didn't honor module alignment request\n");
> -        mod[i].mod_end -= mod[i].mod_start;
> -        mod[i].mod_start >>= PAGE_SHIFT;
> -        mod[i].reserved = 0;
> +        bi->mods[i].mod->mod_end -= bi->mods[i].mod->mod_start;
> +        bi->mods[i].mod->mod_start >>= PAGE_SHIFT;
> +        bi->mods[i].mod->reserved = 0;
>       }
>   
>       /*

> @@ -1509,13 +1510,15 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   #endif
>       }
>   
> -    if ( bi->mods[0].headroom && !mod->reserved )
> +    if ( bi->mods[0].headroom && !bi->mods[0].mod->reserved )
>           panic("Not enough memory to relocate the dom0 kernel image\n");
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
> -        uint64_t s = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
> +        uint64_t s = (uint64_t)bi->mods[i].mod->mod_start
> +                        << PAGE_SHIFT;

pfn_to_paddr() ?

>   
> -        reserve_e820_ram(&boot_e820, s, s + PAGE_ALIGN(mod[i].mod_end));
> +        reserve_e820_ram(&boot_e820, s,
> +                         s + PAGE_ALIGN(bi->mods[i].mod->mod_end));
>       }
>   
>       if ( !xen_phys_start )
> @@ -1593,8 +1596,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>                   map_e = boot_e820.map[j].addr + boot_e820.map[j].size;
>                   for ( j = 0; j < bi->nr_modules; ++j )
>                   {
> -                    uint64_t end = pfn_to_paddr(mod[j].mod_start) +
> -                                   mod[j].mod_end;
> +                    uint64_t end = pfn_to_paddr(
> +                                   bi->mods[j].mod->mod_start) +
> +                                   bi->mods[j].mod->mod_end;

I think you want a different indent.  I think
     uint64_t end = pfn_to_paddr(bi->mods[j].mod->mod_start)

will all fit on one line (indented all the way).  (Thunderbird makes it 
difficult me to send indented.)

>   
>                       if ( map_e < end )
>                           map_e = end;
> @@ -1668,11 +1672,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
> -        set_pdx_range(mod[i].mod_start,
> -                      mod[i].mod_start + PFN_UP(mod[i].mod_end));
> -        map_pages_to_xen((unsigned long)mfn_to_virt(mod[i].mod_start),
> -                         _mfn(mod[i].mod_start),
> -                         PFN_UP(mod[i].mod_end), PAGE_HYPERVISOR);
> +        set_pdx_range(bi->mods[i].mod->mod_start,
> +                      bi->mods[i].mod->mod_start +
> +                      PFN_UP(bi->mods[i].mod->mod_end));
> +        map_pages_to_xen(
> +            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),

map_pages_to_xen((unsigned long)maddr_to_virt(bi->mods[i].start),

All fits on one line.

> +            _mfn(bi->mods[i].mod->mod_start),
> +            PFN_UP(bi->mods[i].mod->mod_end), PAGE_HYPERVISOR);
>       }
>   
>   #ifdef CONFIG_KEXEC

Regards,
Jason

Re: [PATCH v5 09/44] x86/boot: introduce boot module types

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This commit introduces module types of xen, kernel, and ramdisk to allow boot
> module detect code to tag the purpose of a boot module. This reduces the need
> for hard coded order assumptions and global variables to be used by consumers
> of boot modules, such as domain construction.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---

> @@ -2058,6 +2063,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>              cpu_has_nx ? "" : "not ");
>   
>       initrdidx = find_first_bit(module_map, bi->nr_modules);
> +    bi->mods[initrdidx].type = BOOTMOD_RAMDISK;

This is incorrect if an initrd isn't present.

>       if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
>           printk(XENLOG_WARNING
>                  "Multiple initrd candidates, picking module #%u\n",

Re: [PATCH v5 10/44] x86/boot: introduce boot module flags

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The existing startup code employs various ad-hoc state tracking about certain
> boot module types by each area of the code. A boot module flags is added to
> enable tracking these different states.  The first state to be transition by
> this commit is module relocation.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h | 4 ++++
>   xen/arch/x86/setup.c                | 8 ++++----
>   2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> index 6941a8975ea6..021ff0d93643 100644
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -31,6 +31,10 @@ struct boot_module {
>        */
>       unsigned long headroom;
>       enum bootmod_type type;
> +
> +    uint32_t flags;
> +#define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
> +

Stray newline.  Otherwise:
Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

>   };
>   
>   /*

Re: [PATCH v5 11/44] x86/boot: split bootstrap_map_addr() out of bootstrap_map()

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> From: Andrew Cooper <andrew.cooper3@citrix.com>
> 
> Using an interface based on addresses directly, not modules.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 12/44] x86/boot: add start and size fields to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This commit introduces the start and size fields to struct boot_module and adds
> a corresponding bootstrap mapping function, bootstrap_map_bm.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 13/44] x86/boot: update struct boot_module on module relocation

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> When a boot module is relocated, ensure struct boot_module start and size
> fields are updated along with early_mod.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index 093a4f5380d1..f968758048ed 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1392,8 +1392,11 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>            * respective reserve_e820_ram() invocation below. No need to
>            * query efi_boot_mem_unused() here, though.
>            */
> -        bi->mods[xen].mod->mod_start = virt_to_mfn(_stext);
> -        bi->mods[xen].mod->mod_end = __2M_rwdata_end - _stext;
> +        bi->mods[xen].start = virt_to_mfn(_stext);
> +        bi->mods[xen].size = __2M_rwdata_end - _stext;

The last patch did:
bi->mods[i].start = (paddr_t)mods[i].mod_start;

and start is a paddr_t.

Is virt_to_mfn() wrong?

> +
> +        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
> +        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>       }
>   
>       bi->mods[0].headroom =

Re: [PATCH v5 14/44] x86/boot: transition relocation calculations to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Use struct boot_module fields, start and size, when calculating the relocation
> address and size. It also ensures that early_mod references are kept in sync.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 36 +++++++++++++++++-------------------
>   1 file changed, 17 insertions(+), 19 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index f968758048ed..4f540c461b26 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1490,7 +1490,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>               struct boot_module *bm = &bi->mods[j];
>               unsigned long size;
>   
> -            size = PAGE_ALIGN(bm->headroom + bm->mod->mod_end);
> +            size = PAGE_ALIGN(bm->headroom + bm->size);

Is there a mismatch from mod_end in PFNs to bm->size in bytes?  Or is 
mod_start in pfns and mod_end in bytes?

>   
>               if ( bi->mods[j].flags & BOOTMOD_FLAG_X86_RELOCATED )
>                   continue;
> @@ -1504,13 +1504,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   
>               if ( s < end &&
>                    (bm->headroom ||
> -                  ((end - size) >> PAGE_SHIFT) > bm->mod->mod_start) )
> +                  paddr_to_pfn(end - size) > paddr_to_pfn(bm->start)) )

Drop the paddr_to_pfn if both sides are now in bytes?

>               {
> -                move_memory(end - size + bm->headroom,
> -                            (uint64_t)bm->mod->mod_start << PAGE_SHIFT,
> -                            bm->mod->mod_end);
> -                bm->mod->mod_start = (end - size) >> PAGE_SHIFT;
> -                bm->mod->mod_end += bm->headroom;
> +                move_memory(end - size + bm->headroom, bm->start, bm->size);
> +                bm->start = (end - size);
> +                bm->mod->mod_start = paddr_to_pfn(bm->start);
> +                bm->size += bm->headroom;
> +                bm->mod->mod_end = bm->size;
>                   bm->flags |= BOOTMOD_FLAG_X86_RELOCATED;
>               }
>           }

> @@ -1700,13 +1698,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
> -        set_pdx_range(bi->mods[i].mod->mod_start,
> -                      bi->mods[i].mod->mod_start +
> -                      PFN_UP(bi->mods[i].mod->mod_end));
> +        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
> +                      paddr_to_pfn(bi->mods[i].mod->mod_start) +

Shouldn't these be
     paddr_to_pfn(bi->mods[i].start)
?

> +                      PFN_UP(bi->mods[i].size));
>           map_pages_to_xen(
> -            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),
> -            _mfn(bi->mods[i].mod->mod_start),
> -            PFN_UP(bi->mods[i].mod->mod_end), PAGE_HYPERVISOR);
> +            (unsigned long)maddr_to_virt(bi->mods[i].start),
> +            maddr_to_mfn(bi->mods[i].start),
> +            PFN_UP(bi->mods[i].size), PAGE_HYPERVISOR);

First argument should fit on same line as map_pages_to_xen().

>       }
>   
>   #ifdef CONFIG_KEXEC

Regards,
Jason

Re: [PATCH v5 15/44] x86/boot: introduce boot module interator

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Provide an iterator to go through boot module array searching based on type.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 16/44] x86/boot: introduce consumed flag for struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Allow the tracking of when a boot module has been consumed by a handler in the
> hypervisor independent of when it is claimed. The instances where the
> hypervisor does nothing beyond claiming, the dom0 kernel, dom0 ramdisk, and a
> placeholder for itself, are updated as being consumed at the time of being
> claimed.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h | 1 +
>   xen/arch/x86/setup.c                | 3 +++
>   2 files changed, 4 insertions(+)
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> index c79678840d31..7833b065eff1 100644
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -34,6 +34,7 @@ struct boot_module {
>   
>       uint32_t flags;
>   #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
> +#define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
>   
>       paddr_t start;
>       size_t size;
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index 4f540c461b26..235b4e41f653 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -318,6 +318,7 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>   
>       /* map the last mb module for xen entry */
>       bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
> +    bi->mods[bi->nr_modules].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>       bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
>   
>       return bi;
> @@ -1196,6 +1197,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>       bitmap_fill(module_map, bi->nr_modules);
>       __clear_bit(0, module_map); /* Dom0 kernel is always first */
>       bi->mods[0].type = BOOTMOD_KERNEL;
> +    bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;

I think these first two can be straight assignments since they occur 
before relocation.

>       if ( pvh_boot )
>       {
> @@ -2085,6 +2087,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   
>       initrdidx = find_first_bit(module_map, bi->nr_modules);
>       bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
> +    bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>       if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
>           printk(XENLOG_WARNING
>                  "Multiple initrd candidates, picking module #%u\n",

This one is after relocation, so |= is necessary.

Regards,
Jason

Re: [PATCH v5 17/44] x86/boot: convert microcode loading to consume struct boot_info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Convert the microcode loading functions to take struct boot_info, and then
> using struct boot_module to map and check for microcode. To keep the changes
> focused, continue using the struct mod to hold the reference to the microcode
> that is used by the late microcode logic.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/cpu/microcode/core.c    | 37 +++++++++++++---------------
>   xen/arch/x86/include/asm/bootinfo.h  |  1 +
>   xen/arch/x86/include/asm/microcode.h | 14 ++++++-----
>   xen/arch/x86/setup.c                 |  4 +--
>   4 files changed, 28 insertions(+), 28 deletions(-)
> 
> diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
> index 8564e4d2c94c..22fea80bc97e 100644
> --- a/xen/arch/x86/cpu/microcode/core.c
> +++ b/xen/arch/x86/cpu/microcode/core.c

> @@ -205,20 +204,18 @@ static void __init microcode_scan_module(
>   }
>   
>   static void __init microcode_grab_module(
> -    unsigned long *module_map,
> -    const multiboot_info_t *mbi)
> +    unsigned long *module_map, struct boot_info *bi)
>   {
> -    module_t *mod = (module_t *)__va(mbi->mods_addr);
> -
>       if ( ucode_mod_idx < 0 )
> -        ucode_mod_idx += mbi->mods_count;
> -    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= mbi->mods_count ||
> +        ucode_mod_idx += bi->nr_modules;
> +    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
>            !__test_and_clear_bit(ucode_mod_idx, module_map) )
>           goto scan;
> -    ucode_mod = mod[ucode_mod_idx];
> +    bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
> +    ucode_mod = *bi->mods[ucode_mod_idx].mod;

Why the dereference: *bi->mods[ucode_mod_idx].mod; ?  I don't think it 
should be there.

>   scan:
>       if ( ucode_scan )
> -        microcode_scan_module(module_map, mbi);
> +        microcode_scan_module(module_map, bi);
>   }
>   
>   static struct microcode_ops __ro_after_init ucode_ops;
> @@ -822,8 +819,8 @@ static int __init early_update_cache(const void *data, size_t len)
>       return rc;
>   }
>   
> -int __init microcode_init_cache(unsigned long *module_map,
> -                                const struct multiboot_info *mbi)
> +int __init microcode_init_cache(
> +    unsigned long *module_map, const struct boot_info *bi)

Why the indent style change?  I prefer the original indent and it 
doesn't seem like the line length matters.  And it looks like you 
restore the indent later.

>   {
>       int rc = 0;
>   

> diff --git a/xen/arch/x86/include/asm/microcode.h b/xen/arch/x86/include/asm/microcode.h
> index 57c08205d475..495c8f7a7cc5 100644
> --- a/xen/arch/x86/include/asm/microcode.h
> +++ b/xen/arch/x86/include/asm/microcode.h
> @@ -4,6 +4,8 @@
>   #include <xen/types.h>
>   #include <xen/percpu.h>
>   
> +#include <asm/bootinfo.h>
> +
>   #include <public/xen.h>
>   
>   struct multiboot_info;
> @@ -22,12 +24,12 @@ struct cpu_signature {
>   DECLARE_PER_CPU(struct cpu_signature, cpu_sig);
>   
>   void microcode_set_module(unsigned int idx);
> -int microcode_update(XEN_GUEST_HANDLE(const_void) buf,
> -                     unsigned long len, unsigned int flags);
> -int early_microcode_init(unsigned long *module_map,
> -                         const struct multiboot_info *mbi);
> -int microcode_init_cache(unsigned long *module_map,
> -                         const struct multiboot_info *mbi);
> +int microcode_update(
> +    XEN_GUEST_HANDLE(const_void) buf, unsigned long len, unsigned int flags);
> +int early_microcode_init(
> +    unsigned long *module_map, struct boot_info *bi);
> +int microcode_init_cache(
> +    unsigned long *module_map, const struct boot_info *bi);

More indent churn.  The diff would be smaller without it.

>   int microcode_update_one(void);
>   
>   #endif /* ASM_X86__MICROCODE_H */

Re: [PATCH v5 01/44] x86/boot: move x86 boot module counting into a new boot_info struct

[Jan Beulich]

On 07.10.2024 19:57, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> --- /dev/null
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -0,0 +1,29 @@
>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>> +/*
>> + * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
>> + * Copyright (c) 2024 Apertus Solutions, LLC
>> + * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
>> + */
>> +
>> +#ifndef __XEN_X86_BOOTINFO_H__
>> +#define __XEN_X86_BOOTINFO_H__
> 
> I haven't been following closely, but I think if we follow Frediano's 
> naming scheme, it would be:
> ASM__X86__BOOTINFO_H

The new scheme became "official" only after Daniel posted the series, by me
actually committing what previously was only a proposal (coming from Bugseng
originally, as a result of long winded discussions). But yes, now that it's
official new headers ought to adhere to it.

Jan

Re: [PATCH v5 03/44] x86/boot: move cmdline to boot info

[Jan Beulich]

On 07.10.2024 20:09, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Transition Xen's command line to being held in struct boot_info.
>>
>> No functional change intended.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
> 
>>   #endif /* __XEN_X86_BOOTINFO_H__ */
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index aafc098ca268..0921f296075f 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -274,6 +274,8 @@ static int __init cf_check parse_acpi_param(const char *s)
>>   }
>>   custom_param("acpi", parse_acpi_param);
>>   
>> +static const char *cmdline_cook(const char *p, const char *loader_name);
> 
> Is there a reason not to move cmdline_cook() (and loader_is_grub2()) 
> earlier to avoid this forward declaration?

At a guess: To limit churn?

Jan

Re: [PATCH v5 18/44] x86/boot: convert late microcode loading to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Remove the use of struct mod to hold the reference for the microcode,
> converting the code to work with a struct boot_module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 17/44] x86/boot: convert microcode loading to consume struct boot_info

[Jason Andryuk]

On 2024-10-07 17:22, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Convert the microcode loading functions to take struct boot_info, and 
>> then
>> using struct boot_module to map and check for microcode. To keep the 
>> changes
>> focused, continue using the struct mod to hold the reference to the 
>> microcode
>> that is used by the late microcode logic.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/cpu/microcode/core.c    | 37 +++++++++++++---------------
>>   xen/arch/x86/include/asm/bootinfo.h  |  1 +
>>   xen/arch/x86/include/asm/microcode.h | 14 ++++++-----
>>   xen/arch/x86/setup.c                 |  4 +--
>>   4 files changed, 28 insertions(+), 28 deletions(-)
>>
>> diff --git a/xen/arch/x86/cpu/microcode/core.c 
>> b/xen/arch/x86/cpu/microcode/core.c
>> index 8564e4d2c94c..22fea80bc97e 100644
>> --- a/xen/arch/x86/cpu/microcode/core.c
>> +++ b/xen/arch/x86/cpu/microcode/core.c
> 
>> @@ -205,20 +204,18 @@ static void __init microcode_scan_module(
>>   }
>>   static void __init microcode_grab_module(
>> -    unsigned long *module_map,
>> -    const multiboot_info_t *mbi)
>> +    unsigned long *module_map, struct boot_info *bi)
>>   {
>> -    module_t *mod = (module_t *)__va(mbi->mods_addr);
>> -
>>       if ( ucode_mod_idx < 0 )
>> -        ucode_mod_idx += mbi->mods_count;
>> -    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= mbi->mods_count ||
>> +        ucode_mod_idx += bi->nr_modules;
>> +    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
>>            !__test_and_clear_bit(ucode_mod_idx, module_map) )
>>           goto scan;
>> -    ucode_mod = mod[ucode_mod_idx];
>> +    bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
>> +    ucode_mod = *bi->mods[ucode_mod_idx].mod;
> 
> Why the dereference: *bi->mods[ucode_mod_idx].mod; ?  I don't think it 
> should be there.

Oh, the next patch shows ucode_mod is not a pointer, so dereferencing is 
correct.   Sorry for the noise.

Regards,
Jason

Re: [PATCH v5 19/44] x86/boot: use consumed boot module flag for microcode

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> To track if the microcode boot module was loaded, a copy of the boot module is
> kept. The size element of this copy is set to zero as the indicator that the
> microcode was loaded. A side effect is that the modules have to be rescanned to
> find the boot module post-relocation, so the cache copy can be created.
> 
> Use the consumed boot module flag to track the loading of the microcode boot
> module. This removes the need to manipulate the boot module size element, no
> longer requiring the copy, thus allowing it to be replaced by a reference. As a
> result it is no longer necessary to rescan the boot modules after relocation
> has occurred.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/cpu/microcode/core.c | 28 ++++++++++++++--------------
>   1 file changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/x86/cpu/microcode/core.c b/xen/arch/x86/cpu/microcode/core.c
> index 7bcc17e0ab2f..5b42aad2fdd0 100644
> --- a/xen/arch/x86/cpu/microcode/core.c
> +++ b/xen/arch/x86/cpu/microcode/core.c

> @@ -826,14 +826,14 @@ int __init microcode_init_cache(
>       if ( !ucode_ops.apply_microcode )
>           return -ENODEV;
>   
> -    if ( ucode_scan )
> -        /* Need to rescan the modules because they might have been relocated */
> +    /* Scan if microcode was not detected earlier */
> +    if ( !ucode_mod )

ucode_scan is a user-controlled variable (ucode=scan=$bool), so I think 
it still needs to be respected.

>           microcode_scan_module(module_map, bi);
>   
> -    if ( ucode_mod.size )
> -        rc = early_update_cache(bootstrap_map_bm(&ucode_mod),
> -                                ucode_mod.size);
> -    else if ( ucode_blob.size )
> +    if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
> +        rc = early_update_cache(bootstrap_map_bm(ucode_mod),
> +                                ucode_mod->size);
> +    else if ( ucode_mod && ucode_blob.size )

ucode_blob seems independent of ucode_mod, so I don't see why this 
didn't stay `else if ( ucode_blob.size )`

>           rc = early_update_cache(ucode_blob.data, ucode_blob.size);
>   
>       return rc;
> @@ -851,10 +851,10 @@ static int __init early_microcode_update_cpu(void)

Regards,
Jason

Re: [PATCH v5 20/44] x86/boot: convert xsm policy loading to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Iterate through the unclaimed struct boot_module to see if any are an XSM FLASK
> policy. If one is located, mark it as an xsm policy.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

> @@ -161,6 +162,14 @@ int __init xsm_multiboot_init(
>       }
>   
>       ret = xsm_core_init(policy_buffer, policy_size);
> +    if ( ret == 0 )
> +    {
> +        int idx = first_boot_module_index(bi, BOOTMOD_XSM_POLICY);
> +
> +        /* If the policy was loaded from a boot module, mark it consumed */
> +        if ( idx >= 0 )
> +            bi->mods[idx].flags |= BOOTMOD_FLAG_X86_CONSUMED;

Maybe xsm_multiboot_policy_init() should return the idx used instead of 
having a second search?  (Also, xsm_multiboot_policy_init() can't fail?)

> +    }
>       bootstrap_map(NULL);
>   
>       return 0;

The other changes look okay.

Regards,
Jason

Re: [PATCH v5 21/44] x86/boot: convert ramdisk locating to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Locate the first unclaimed struct boot_module and mark it as ramdisk. If there
> are any remaining unclaimed struct boot_module instances, report to the
> console. In the change, the new boot module iterator is used to find the
> initrd index, which returns a signed int. Switch initrdidx from unsigned to
> signed.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Regards,
Jason

Re: [PATCH v5 22/44] x86/boot: remove module_map usage from microcode loading

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> With all consumers of module_map converted, remove usage of it
> by the microcode loading logic.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

> @@ -202,19 +201,18 @@ static void __init microcode_scan_module(
>       }
>   }
>   
> -static void __init microcode_grab_module(
> -    unsigned long *module_map, struct boot_info *bi)
> +static void __init microcode_grab_module(struct boot_info *bi)
>   {
>       if ( ucode_mod_idx < 0 )
>           ucode_mod_idx += bi->nr_modules;
>       if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
> -         !__test_and_clear_bit(ucode_mod_idx, module_map) )
> +         (bi->mods[ucode_mod_idx].type != BOOTMOD_UNKNOWN) )

Just
          bi->mods[ucode_mod_idx].type != BOOTMOD_UNKNOWN )

With that:

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Regards,
Jason

Re: [PATCH v5 23/44] x86/boot: remove module_map usage from xsm policy loading

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c  | 2 +-
>   xen/include/xsm/xsm.h | 9 +++------
>   xen/xsm/xsm_core.c    | 6 ++----
>   xen/xsm/xsm_policy.c  | 5 +----
>   4 files changed, 7 insertions(+), 15 deletions(-)
> 

> diff --git a/xen/xsm/xsm_policy.c b/xen/xsm/xsm_policy.c
> index 921bb254b9d1..a22367a62e93 100644
> --- a/xen/xsm/xsm_policy.c
> +++ b/xen/xsm/xsm_policy.c

> @@ -42,7 +41,6 @@ int __init xsm_multiboot_policy_init(
>   
>       /*
>        * Try all modules and see whichever could be the binary policy.
> -     * Adjust module_map for the module that is the binary policy.
>        */

You can collapse to a single line comment /* ... */

With that:
Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

>       for ( i = bi->nr_modules-1; i >= 1; i-- )
>       {

Re: [PATCH v5 24/44] x86/boot: remove module_map usage by ramdisk loading

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The ramdisk loading is the last user of module_map, remove
> its usage and any remaining remnants of module_map.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 11 +++++------
>   1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index b0946216ea3f..0d2ee19998aa 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1037,7 +1037,7 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>       struct boot_info *bi;
>       multiboot_info_t *mbi;
>       module_t *mod;
> -    unsigned long nr_pages, raw_max_page, module_map[1];
> +    unsigned long nr_pages, raw_max_page;
>       int i, j, e820_warn = 0, bytes = 0;
>       unsigned long eb_start, eb_end;
>       bool acpi_boot_table_init_done = false, relocated = false;
> @@ -1187,15 +1187,14 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>           panic("dom0 kernel not specified. Check bootloader configuration\n");
>   
>       /* Check that we don't have a silly number of modules. */
> -    if ( bi->nr_modules > sizeof(module_map) * 8 )
> +    if ( bi->nr_modules > MAX_NR_BOOTMODS + 1 )

Don't you want to check MAX_NR_BOOTMODS, to keep the last module for Xen 
itself?

Regards,
Jason

>       {
> -        bi->nr_modules = sizeof(module_map) * 8;
> -        printk("Excessive boot modules - using the first %u only\n",
> +        bi->nr_modules = MAX_NR_BOOTMODS + 1;
> +        printk("Excessive multiboot modules - using the first %u only\n",
>                  bi->nr_modules);
>       }
>   
> -    bitmap_fill(module_map, bi->nr_modules);
> -    __clear_bit(0, module_map); /* Dom0 kernel is always first */
> +    /* Dom0 kernel is always first */
>       bi->mods[0].type = BOOTMOD_KERNEL;
>       bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>   

Re: [PATCH v5 25/44] x86/boot: convert create_dom0 to use boot info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This commit changes create_dom0 to no longer take the individual components and
> take struct boot_info instead. Internally, it is changed to locate the kernel
> and ramdisk details from struct boot_info.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 25 ++++++++++++++++---------
>   1 file changed, 16 insertions(+), 9 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index 0d2ee19998aa..c2bcddc50990 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -946,10 +946,8 @@ static unsigned int __init copy_bios_e820(struct e820entry *map, unsigned int li
>       return n;
>   }
>   
> -static struct domain *__init create_dom0(const module_t *image,
> -                                         unsigned long headroom,
> -                                         module_t *initrd, const char *kextra,
> -                                         const char *loader)
> +static struct domain *__init create_dom0(const struct boot_info *bi,
> +                                         const char *kextra)
>   {
>       static char __initdata cmdline[MAX_GUEST_CMDLINE];
>   
> @@ -964,9 +962,21 @@ static struct domain *__init create_dom0(const module_t *image,
>               .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
>           },
>       };
> +    int headroom, mod_idx = first_boot_module_index(bi, BOOTMOD_RAMDISK);

I think headroom should stay unsigned long, which matches struct 
boot_module.

With that

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Regards,
Jason

Re: [PATCH v5 26/44] x86/boot: convert construct_dom0 to use struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The construct_dom0 function is converted to consume struct boot_module
> instances for the kernel and ramdisk. With this change, it is no longer
> necessary for the internal use of struct mod by create_dom0, so they are
> changed to struct boot_module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 27/44] x86/boot: relocate kextra into boot info

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Move kextra into struct boot_info, thus no longer needed to be passed as a
> parameter to create_dom0.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 28/44] x86/boot: add cmdline to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Add a char pointer field, cmdline, to struct boot_module to hold the address
> pointed to by the string field of struct mod. This removes the need to use the
> early_mod field to get to the dom0 kernel command line.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootinfo.h | 2 ++
>   xen/arch/x86/setup.c                | 9 ++++++---
>   2 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
> index 9ed260629012..3b6bfbe88770 100644
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -38,6 +38,8 @@ struct boot_module {
>   #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
>   #define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
>   
> +    const char *cmdline;
> +
>       paddr_t start;
>       size_t size;
>   };
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index aed0837902c4..d5916e85f68e 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -312,6 +312,8 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>       {
>           bi->mods[i].mod = &mods[i];
>   
> +        bi->mods[i].cmdline = (char *)(paddr_t)mods[i].string;
> +
>           bi->mods[i].start = (paddr_t)mods[i].mod_start;
>           bi->mods[i].size = mods[i].mod_end - mods[i].mod_start;
>       }
> @@ -1000,10 +1002,11 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>           panic("Error creating d%uv0\n", domid);
>   
>       /* Grab the DOM0 command line. */
> -    if ( image->mod->string || bi->kextra )
> +    if ( image->cmdline || bi->kextra )
>       {
> -        if ( image->mod->string )
> -            safe_strcpy(cmdline, cmdline_cook(__va(image->mod->string),
> +        if ( image->cmdline )
> +            safe_strcpy(cmdline,
> +                        cmdline_cook(__va((unsigned long)image->cmdline),

char * seems inappropriate if cmdline isn't usable as a string.  Maybe 
have cmdline as a paddr_t, or can __va() be used at assignment time?

Regards,
Jason

>                           bi->loader));
>   
>           if ( bi->kextra )

Re: [PATCH v5 29/44] x86/boot: convert dom0_construct_pv image param to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This changes the type for the image parameter of dom0_construct_pv to be struct
> boot_module. Removing the usage of early_mod field for kernel module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 30/44] x86/boot: convert dom0_construct_pv initrd param to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This changes the type for the initrd parameter of dom0_construct_pv to be struct
> boot_module. This conversion requires several adjustments throughout dom0_construct_pv
> to account for the type change. Removes the usage of early_mod field for ramdisk module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

> diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
> index 7b6afe64d799..16b8c1e40998 100644
> --- a/xen/arch/x86/pv/dom0_build.c
> +++ b/xen/arch/x86/pv/dom0_build.c

> @@ -367,7 +367,8 @@ static int __init dom0_construct(struct domain *d,
>       unsigned long nr_pt_pages;
>       unsigned long alloc_spfn;
>       unsigned long alloc_epfn;
> -    unsigned long initrd_pfn = -1, initrd_mfn = 0;
> +    unsigned long initrd_pfn = -1;
> +    mfn_t  initrd_mfn = { 0 };

= _mfn(0);

With that:

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

>       unsigned long count;
>       struct page_info *page = NULL;
>       unsigned int flush_flags = 0;

Re: [PATCH v5 31/44] x86/boot: convert dom0_construct_pvh to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This changes both the kernel and ramdisk parameters over to struct
> boot_module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 32/44] x86/boot: convert pvh_load_kernel to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This changes both the kernel and ramdisk parameters over to struct boot_module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 33/44] x86/boot: convert initial_images to struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The variable initial_images is used for tracking the boot modules passed in by
> the boot loader. Convert to a struct boot_module and adjust the code that uses
> it accordingly.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 15 +++++++++------
>   1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index d5916e85f68e..30a139074833 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c

> @@ -336,8 +336,9 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
>   
>       for ( nr = i = 0; i < bi->nr_modules; ++i )
>       {
> -        unsigned long start = initial_images[i].mod_start;
> -        unsigned long end = start + PFN_UP(initial_images[i].mod_end);
> +        unsigned long start = initial_images[i].mod->mod_start;
> +        unsigned long end = start +
> +                            PFN_UP(initial_images[i].mod->mod_end);

This can fit on a single line.

>   
>           if ( end > node_start && node_end > start )
>               nr += min(node_end, end) - max(node_start, start);
> @@ -353,10 +354,12 @@ void __init discard_initial_images(void)
>   
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
> -        uint64_t start = (uint64_t)initial_images[i].mod_start << PAGE_SHIFT;
> +        uint64_t start =
> +            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
>   
>           init_domheap_pages(start,
> -                           start + PAGE_ALIGN(initial_images[i].mod_end));
> +                           start +
> +                           PAGE_ALIGN(initial_images[i].mod->mod_end));

This can fit on a single line.

>       }
>   
>       bi->nr_modules = 0;

With those fixed:
Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 34/44] x86/boot: drop the use of initial_images unit global

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 13 ++++---------
>   1 file changed, 4 insertions(+), 9 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index 30a139074833..b3b6e6f38622 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -276,8 +276,6 @@ custom_param("acpi", parse_acpi_param);
>   
>   static const char *cmdline_cook(const char *p, const char *loader_name);
>   
> -static const struct boot_module *__initdata initial_images;
> -
>   struct boot_info __initdata xen_boot_info;
>   
>   static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
> @@ -336,9 +334,9 @@ unsigned long __init initial_images_nrpages(nodeid_t node)
>   
>       for ( nr = i = 0; i < bi->nr_modules; ++i )
>       {
> -        unsigned long start = initial_images[i].mod->mod_start;
> +        unsigned long start = bi->mods[i].mod->mod_start;
>           unsigned long end = start +
> -                            PFN_UP(initial_images[i].mod->mod_end);
> +                            PFN_UP(bi->mods[i].mod->mod_end);

Fits on a single line.

>   
>           if ( end > node_start && node_end > start )
>               nr += min(node_end, end) - max(node_start, start);
> @@ -355,15 +353,14 @@ void __init discard_initial_images(void)
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
>           uint64_t start =
> -            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
> +            (uint64_t)bi->mods[i].mod->mod_start << PAGE_SHIFT;

Fits on one line.  Can also be pfn_to_paddr(), which applies to earlier 
patches.  Having said that, maybe it's okay to skip pfn_to_paddr as at 
the end of the series mods[i].start is used without a shift.  i.e. fewer 
transformations in these "mechanical" changes make review easier. 
Unless someone else wants pfn_to_addr(), I am okay without that conversion.

>   
>           init_domheap_pages(start,
>                              start +
> -                           PAGE_ALIGN(initial_images[i].mod->mod_end));
> +                           PAGE_ALIGN(bi->mods[i].mod->mod_end));

One line.

>       }
>   
>       bi->nr_modules = 0;
> -    initial_images = NULL;
>   }
>   
>   static void __init init_idle_domain(void)

With the line fixups:

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 35/44] x86/boot: remove usage of mod_end by discard_initial_images

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> This eliminates usage of early_mod by discard_initial_images
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Any direct usages of struct mod have been transitioned, remove the remaining
> references to early_mod fields.

This is unclear, please try to re-word.  "struct mod" and "early_mod" 
don't exist.

> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/setup.c | 31 +++++++++++--------------------
>   1 file changed, 11 insertions(+), 20 deletions(-)
> 
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index e9e3da3204f1..0ffe8d3ff8dd 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c

> @@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>            */
>           bi->mods[xen].start = virt_to_mfn(_stext);
>           bi->mods[xen].size = __2M_rwdata_end - _stext;
> -
> -        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
> -        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>       }
>   
> -    bi->mods[0].headroom =
> -        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
> -                         bi->mods[0].mod->mod_end);
> -
> -    bootstrap_map(NULL);
> +    bi->mods[0].headroom = bzimage_headroom(
> +                        bootstrap_map_bm(&bi->mods[0]),
> +                        bi->mods[0].size);

Thunderbird might corrupt this, bit the above can fit on two lines:
     bi->mods[0].headroom = bzimage_headroom(bootstrap_map_bm(&bi->mods[0]),
                                             bi->mods[0].size);

> +    bootstrap_map_bm(NULL);
>   
>   #ifndef highmem_start
>       /* Don't allow split below 4Gb. */

> @@ -1708,8 +1699,8 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>   
>       for ( i = 0; i < bi->nr_modules; ++i )
>       {
> -        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
> -                      paddr_to_pfn(bi->mods[i].mod->mod_start) +
> +        set_pdx_range(paddr_to_pfn(bi->mods[i].start),
> +                      paddr_to_pfn(bi->mods[i].start) +

This belongs in patch 14 as mentioned there.

>                         PFN_UP(bi->mods[i].size));
>           map_pages_to_xen(
>               (unsigned long)maddr_to_virt(bi->mods[i].start),

Re: [PATCH v5 37/44] x86/boot: remove mod from struct boot_module

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> With all references to mod field removed, remove the mod field from struct
> boot_module.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 38/44] x86/boot: introduce boot domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> To begin moving toward allowing the hypervisor to construct more than one
> domain at boot, a container is needed for a domain's build information.
> 
> Introduce a new header, <xen/asm/bootdomain.h>, that contains the initial
> struct boot_domain that encapsulate the build information for a domain.
> 
> No functional change intended.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootdomain.h | 28 +++++++++++++++++++++++++++
>   xen/arch/x86/include/asm/bootinfo.h   |  5 +++++
>   xen/arch/x86/setup.c                  | 24 ++++++++---------------
>   3 files changed, 41 insertions(+), 16 deletions(-)
>   create mode 100644 xen/arch/x86/include/asm/bootdomain.h
> 
> diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
> new file mode 100644
> index 000000000000..4285223ac5ab
> --- /dev/null
> +++ b/xen/arch/x86/include/asm/bootdomain.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +/*
> + * Copyright (c) 2024 Apertus Solutions, LLC
> + * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
> + * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
> + */
> +
> +#ifndef __XEN_X86_BOOTDOMAIN_H__
> +#define __XEN_X86_BOOTDOMAIN_H__

ASM__X86__BOOTDOMAIN_H

With that:

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 39/44] x86/boot: introduce domid field to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Add a domid field to struct boot_domain to hold the assigned domain id for the
> domain. During initialization, ensure all instances of struct boot_domain have
> the invalid domid to ensure that the domid must be set either by convention or
> configuration.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 39/44] x86/boot: introduce domid field to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Add a domid field to struct boot_domain to hold the assigned domain id for the
> domain. During initialization, ensure all instances of struct boot_domain have
> the invalid domid to ensure that the domid must be set either by convention or
> configuration.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootdomain.h |  2 ++
>   xen/arch/x86/setup.c                  | 12 +++++++-----
>   2 files changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
> index 4285223ac5ab..d6264d554dba 100644
> --- a/xen/arch/x86/include/asm/bootdomain.h
> +++ b/xen/arch/x86/include/asm/bootdomain.h
> @@ -11,6 +11,8 @@
>   struct boot_module;
>   
>   struct boot_domain {
> +    domid_t domid;
> +
>       struct boot_module *kernel;
>       struct boot_module *ramdisk;
>   };

Oh, you should probably move domid after the pointers to avoid a hole.

Regards,
Jason

Re: [PATCH v5 42/44] x86/boot: convert construct_dom0 to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> A struct boot_domain now encapsulates the domain reference, kernel, ramdisk,
> and command line for the domain being constructed. As a result of this
> encapsulation, construct_dom0 can now take a single struct boot_domain instead
> of these four parameters.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/dom0_build.c        | 19 +++++++++----------
>   xen/arch/x86/include/asm/setup.h |  4 +---
>   xen/arch/x86/setup.c             |  2 +-
>   3 files changed, 11 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index 71b2e3afc1a1..e552f2e9abef 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -597,22 +597,21 @@ int __init dom0_setup_permissions(struct domain *d)
>       return rc;
>   }
>   
> -int __init construct_dom0(struct domain *d, const struct boot_module *image,
> -                          struct boot_module *initrd, const char *cmdline)
> +int __init construct_dom0(struct boot_domain *bd)
>   {
>       int rc;

I think a local variable would be better:

     struct domain *d = bd->d;

The patch is smaller, and using just d is common in the hypervisor.

Regards,
Jason

>   
>       /* Sanity! */
> -    BUG_ON(!pv_shim && d->domain_id != 0);
> -    BUG_ON(d->vcpu[0] == NULL);
> -    BUG_ON(d->vcpu[0]->is_initialised);
> +    BUG_ON(!pv_shim && bd->d->domain_id != 0);
> +    BUG_ON(bd->d->vcpu[0] == NULL);
> +    BUG_ON(bd->d->vcpu[0]->is_initialised);
>   
>       process_pending_softirqs();
>   
> -    if ( is_hvm_domain(d) )
> -        rc = dom0_construct_pvh(d, image, initrd, cmdline);
> -    else if ( is_pv_domain(d) )
> -        rc = dom0_construct_pv(d, image, initrd, cmdline);
> +    if ( is_hvm_domain(bd->d) )
> +        rc = dom0_construct_pvh(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
> +    else if ( is_pv_domain(bd->d) )
> +        rc = dom0_construct_pv(bd->d, bd->kernel, bd->ramdisk, bd->cmdline);
>       else
>           panic("Cannot construct Dom0. No guest interface available\n");
>   

Re: [PATCH v5 41/44] x86/boot: add struct domain to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Store a reference to the created domain in struct boot_domain.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---

> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index f250638edf09..e6a231bd2d42 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -959,7 +959,6 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>           },
>       };
>       struct boot_domain *bd = &bi->domains[0];
> -    struct domain *d;
>   
>       if ( opt_dom0_pvh )
>       {
> @@ -976,13 +975,13 @@ static struct domain *__init create_dom0(struct boot_info *bi)
>   
>       /* Create initial domain.  Not d0 for pvshim. */
>       bd->domid = get_initial_domain_id();
> -    d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);

It's a smaller patch if you keep `d` and the line above and then do:
        bd->d = d;

Regards,
Jason

> -    if ( IS_ERR(d) )
> -        panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
> +    bd->d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : CDF_privileged);
> +    if ( IS_ERR(bd->d) )
> +        panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(bd->d));
>   
> -    init_dom0_cpuid_policy(d);
> +    init_dom0_cpuid_policy(bd->d);
>   
> -    if ( alloc_dom0_vcpu0(d) == NULL )
> +    if ( alloc_dom0_vcpu0(bd->d) == NULL )
>           panic("Error creating d%uv0\n", bd->domid);
>   
>       /* Grab the DOM0 command line. */

Re: [PATCH v5 43/44] x86/boot: convert dom0_construct_pv to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> With construct_dom0 consuming struct boot_domain, continue passing the
> structure down to dom0_construct_pv.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

> diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
> index 16b8c1e40998..34974aa7cd56 100644
> --- a/xen/arch/x86/pv/dom0_build.c
> +++ b/xen/arch/x86/pv/dom0_build.c
> @@ -354,10 +354,7 @@ static struct page_info * __init alloc_chunk(struct domain *d,
>       return page;
>   }
>   
> -static int __init dom0_construct(struct domain *d,
> -                                 const struct boot_module *image,
> -                                 struct boot_module *initrd,
> -                                 const char *cmdline)
> +static int __init dom0_construct(struct boot_domain *bd)
>   {
>       int i, rc, order, machine;
>       bool compatible, compat;
> @@ -373,11 +370,12 @@ static int __init dom0_construct(struct domain *d,
>       struct page_info *page = NULL;
>       unsigned int flush_flags = 0;
>       start_info_t *si;
> -    struct vcpu *v = d->vcpu[0];
> -    void *image_base = bootstrap_map_bm(image);
> -    unsigned long image_len = image->size;
> -    void *image_start = image_base + image->headroom;
> -    unsigned long initrd_len = initrd ? initrd->size : 0;
> +    struct domain *d = bd->d;
> +    struct vcpu *v = bd->d->vcpu[0];

This can stay:
     struct vcpu *v = d->vcpu[0];

With that:

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Re: [PATCH v5 44/44] x86/boot: convert dom0_construct_pvh to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> With construct_dom0 consuming struct boot_domain, continue passing the
> structure down to dom0_construct_pvh.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

> diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
> index a3fd5e762dc4..755e257cdf30 100644
> --- a/xen/arch/x86/hvm/dom0_build.c
> +++ b/xen/arch/x86/hvm/dom0_build.c
> @@ -1299,25 +1299,23 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
>       }
>   }
>   
> -int __init dom0_construct_pvh(
> -    struct domain *d, const struct boot_module *image,
> -    struct boot_module *initrd, const char *cmdline)
> +int __init dom0_construct_pvh(const struct boot_domain *bd)
>   {
>       paddr_t entry, start_info;
>       int rc;

Again, I recommend using a local struct domain *d to cut down on the churn.

Regards,
Jason

>   
> -    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", d->domain_id);
> +    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", bd->domid);
>   
> -    if ( is_hardware_domain(d) )
> +    if ( is_hardware_domain(bd->d) )

Re: [PATCH v5 40/44] x86/boot: add cmdline to struct boot_domain

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> Add a container for the "cooked" command line for a domain.
> 
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>   xen/arch/x86/include/asm/bootdomain.h |  4 ++++
>   xen/arch/x86/setup.c                  | 18 ++++++++----------
>   2 files changed, 12 insertions(+), 10 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
> index d6264d554dba..00f7d9267965 100644
> --- a/xen/arch/x86/include/asm/bootdomain.h
> +++ b/xen/arch/x86/include/asm/bootdomain.h
> @@ -8,9 +8,13 @@
>   #ifndef __XEN_X86_BOOTDOMAIN_H__
>   #define __XEN_X86_BOOTDOMAIN_H__
>   
> +#include <public/xen.h>
> +
>   struct boot_module;
>   
>   struct boot_domain {
> +    char cmdline[MAX_GUEST_CMDLINE];
> +

1024 bytes for just dom0 isn't too much.  But when hyperlaunch has 64 
boot_domains, that's a good bit more.  But I suppose it isn't too much 
RAM for a modern system.  This is __initdata, so it increases the binary 
size.  I just want to highlight this in case others want to chime in.

The code changes seem fine.

Regards,
Jason

Re: [PATCH v5 00/44] Boot modules for Hyperlaunch

[Jason Andryuk]

On 2024-10-06 17:49, Daniel P. Smith wrote:
> The Boot Modules for Hyperlaunch series is an effort to split out preliminary
> changes necessary for the introduction of the Hyperlaunch domain builder
> logic. These preliminary changes revolve around introducing the struct
> boot_module and struct boot_domain structures. This includes converting the
> dom0 construction path to use these structures. These abstractions lay the
> groundwork to transform and extend the dom0 construction logic into a limited,
> but general domain builder.
> 
> The splitting of Hyperlaunch into a set of series are twofold, to reduce the
> effort in reviewing a much larger series, and to reduce the effort in handling
> the knock-on effects to the construction logic from requested review changes.
> 
> Much thanks to AMD for supporting this work.
> 
> Documentation on Hyperlaunch:
> https://wiki.xenproject.org/wiki/Hyperlaunch
> 
> Original Hyperlaunch v1 patch series:
> https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00345.html

There is a lot of re-formatting of function arguments like:

-static int __init pvh_load_kernel(struct domain *d, const module_t *image,
-                                  unsigned long image_headroom,
-                                  module_t *initrd, void *image_base,
-                                  const char *cmdline, paddr_t *entry,
-                                  paddr_t *start_info_addr)
+static int __init pvh_load_kernel(
+    struct domain *d, const struct boot_module *image,
+    struct boot_module *initrd, void *image_base,
+    const char *cmdline, paddr_t *entry, paddr_t *start_info_addr)

I feel like the old style is more common and I prefer it.  But I also 
don't see it specified in CODING_STYLE.  As I am not a maintainer, I'd 
like them to weigh in.

Also, it is nicer to include a per-patch change log instead of just a 
cover-letter one.  That will be useful in subsequent review rounds to 
clearly identified changed patches.

Thanks,
Jason

Re: [PATCH v5 00/44] Boot modules for Hyperlaunch

[Andrew Cooper]

On 08/10/2024 9:07 pm, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The Boot Modules for Hyperlaunch series is an effort to split out
>> preliminary
>> changes necessary for the introduction of the Hyperlaunch domain builder
>> logic. These preliminary changes revolve around introducing the struct
>> boot_module and struct boot_domain structures. This includes
>> converting the
>> dom0 construction path to use these structures. These abstractions
>> lay the
>> groundwork to transform and extend the dom0 construction logic into a
>> limited,
>> but general domain builder.
>>
>> The splitting of Hyperlaunch into a set of series are twofold, to
>> reduce the
>> effort in reviewing a much larger series, and to reduce the effort in
>> handling
>> the knock-on effects to the construction logic from requested review
>> changes.
>>
>> Much thanks to AMD for supporting this work.
>>
>> Documentation on Hyperlaunch:
>> https://wiki.xenproject.org/wiki/Hyperlaunch
>>
>> Original Hyperlaunch v1 patch series:
>> https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00345.html
>>
>
> There is a lot of re-formatting of function arguments like:
>
> -static int __init pvh_load_kernel(struct domain *d, const module_t
> *image,
> -                                  unsigned long image_headroom,
> -                                  module_t *initrd, void *image_base,
> -                                  const char *cmdline, paddr_t *entry,
> -                                  paddr_t *start_info_addr)
> +static int __init pvh_load_kernel(
> +    struct domain *d, const struct boot_module *image,
> +    struct boot_module *initrd, void *image_base,
> +    const char *cmdline, paddr_t *entry, paddr_t *start_info_addr)
>
> I feel like the old style is more common and I prefer it.  But I also
> don't see it specified in CODING_STYLE.  As I am not a maintainer, I'd
> like them to weigh in.

I already did.  :)

This isn't a terribly bad example, but there are others which are much
worse.  Given a choice between an intractable mess of parameters
squeezed onto the RHS, and the same mess spread out across the whole
width, prefer the latter.

~Andrew

Re: [PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Jan Beulich]

On 08.10.2024 21:15, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Any direct usages of struct mod have been transitioned, remove the remaining
>> references to early_mod fields.
> 
> This is unclear, please try to re-word.  "struct mod" and "early_mod" 
> don't exist.
> 
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 31 +++++++++++--------------------
>>   1 file changed, 11 insertions(+), 20 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index e9e3da3204f1..0ffe8d3ff8dd 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
> 
>> @@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>>            */
>>           bi->mods[xen].start = virt_to_mfn(_stext);
>>           bi->mods[xen].size = __2M_rwdata_end - _stext;
>> -
>> -        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
>> -        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>>       }
>>   
>> -    bi->mods[0].headroom =
>> -        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
>> -                         bi->mods[0].mod->mod_end);
>> -
>> -    bootstrap_map(NULL);
>> +    bi->mods[0].headroom = bzimage_headroom(
>> +                        bootstrap_map_bm(&bi->mods[0]),
>> +                        bi->mods[0].size);
> 
> Thunderbird might corrupt this, bit the above can fit on two lines:
>      bi->mods[0].headroom = bzimage_headroom(bootstrap_map_bm(&bi->mods[0]),
>                                              bi->mods[0].size);

Or else at least indentation wants to change, to one of the two possible
forms:

    bi->mods[0].headroom = bzimage_headroom(
        bootstrap_map_bm(&bi->mods[0]),
        bi->mods[0].size);

(indentation increased by a level from the start of the statement) or

    bi->mods[0].headroom = bzimage_headroom(
                               bootstrap_map_bm(&bi->mods[0]),
                               bi->mods[0].size);

(indentation by one level biased from the start of the function call).
Personally, if already wrapping like this, I'd prefer the former.

Jan

Re: [PATCH v5 39/44] x86/boot: introduce domid field to struct boot_domain

[Jan Beulich]

On 08.10.2024 21:36, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Add a domid field to struct boot_domain to hold the assigned domain id for the
>> domain. During initialization, ensure all instances of struct boot_domain have
>> the invalid domid to ensure that the domid must be set either by convention or
>> configuration.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootdomain.h |  2 ++
>>   xen/arch/x86/setup.c                  | 12 +++++++-----
>>   2 files changed, 9 insertions(+), 5 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootdomain.h b/xen/arch/x86/include/asm/bootdomain.h
>> index 4285223ac5ab..d6264d554dba 100644
>> --- a/xen/arch/x86/include/asm/bootdomain.h
>> +++ b/xen/arch/x86/include/asm/bootdomain.h
>> @@ -11,6 +11,8 @@
>>   struct boot_module;
>>   
>>   struct boot_domain {
>> +    domid_t domid;
>> +
>>       struct boot_module *kernel;
>>       struct boot_module *ramdisk;
>>   };
> 
> Oh, you should probably move domid after the pointers to avoid a hole.

That would only move the hole to the end of the struct.

Jan

Re: [PATCH v5 01/44] x86/boot: move x86 boot module counting into a new boot_info struct

[Daniel P. Smith]

On 10/8/24 02:41, Jan Beulich wrote:
> On 07.10.2024 19:57, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> --- /dev/null
>>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>>> @@ -0,0 +1,29 @@
>>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>>> +/*
>>> + * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
>>> + * Copyright (c) 2024 Apertus Solutions, LLC
>>> + * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
>>> + */
>>> +
>>> +#ifndef __XEN_X86_BOOTINFO_H__
>>> +#define __XEN_X86_BOOTINFO_H__
>>
>> I haven't been following closely, but I think if we follow Frediano's
>> naming scheme, it would be:
>> ASM__X86__BOOTINFO_H
> 
> The new scheme became "official" only after Daniel posted the series, by me
> actually committing what previously was only a proposal (coming from Bugseng
> originally, as a result of long winded discussions). But yes, now that it's
> official new headers ought to adhere to it.

Ack.

v/r,
dps

Re: [PATCH v5 03/44] x86/boot: move cmdline to boot info

[Daniel P. Smith]

On 10/8/24 02:42, Jan Beulich wrote:
> On 07.10.2024 20:09, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> Transition Xen's command line to being held in struct boot_info.
>>>
>>> No functional change intended.
>>>
>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>> ---
>>
>>>    #endif /* __XEN_X86_BOOTINFO_H__ */
>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>> index aafc098ca268..0921f296075f 100644
>>> --- a/xen/arch/x86/setup.c
>>> +++ b/xen/arch/x86/setup.c
>>> @@ -274,6 +274,8 @@ static int __init cf_check parse_acpi_param(const char *s)
>>>    }
>>>    custom_param("acpi", parse_acpi_param);
>>>    
>>> +static const char *cmdline_cook(const char *p, const char *loader_name);
>>
>> Is there a reason not to move cmdline_cook() (and loader_is_grub2())
>> earlier to avoid this forward declaration?
> 
> At a guess: To limit churn?

Correct.

v/r,
dps

Re: [PATCH v5 05/44] x86/boot: introduce struct boot_module

[Daniel P. Smith]

On 10/7/24 14:29, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> This will introduce a new struct boot_module to provide a rich state
>> representation around modules provided by the boot loader. Support is 
>> for 64
>> boot modules, one held in reserve for Xen, and up to 63 can be 
>> provided by the
>> boot loader. The array of struct boot_modules will be accessible via a
>> reference held in struct boot_info.
>>
>> A temporary `mod` parameter is included in struct boot_module to ease the
>> transition from using Multiboot v1 structures over to struct 
>> boot_module. Once
>> the transition is complete, the parameter will be dropped from the 
>> structure.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h | 14 ++++++++++++--
>>   xen/arch/x86/setup.c                |  9 +++++++++
>>   2 files changed, 21 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h 
>> b/xen/arch/x86/include/asm/bootinfo.h
>> index 87d311ac1399..d19473d8941e 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -8,20 +8,30 @@
>>   #ifndef __XEN_X86_BOOTINFO_H__
>>   #define __XEN_X86_BOOTINFO_H__
>> +#include <xen/multiboot.h>
>>   #include <xen/types.h>
>> +/* Max number of boot modules a bootloader can provide in addition to 
>> Xen */
>> +#define MAX_NR_BOOTMODS 63
>> +
>> +struct boot_module {
>> +    /* Transitionary only */
>> +    module_t *mod;
>> +};
>> +
>>   /*
>>    * Xen internal representation of information provided by the
>>    * bootloader/environment, or derived from the information.
>>    */
>>   struct boot_info {
>> -    unsigned int nr_modules;
>> -
> 
> (You should probably re-work the other patches to insert ahead of this 
> and avoid the movement.)

Ack.

>>       const char *loader;
>>       const char *cmdline;
>>       paddr_t memmap_addr;
>>       size_t memmap_length;
>> +
>> +    unsigned int nr_modules;
>> +    struct boot_module mods[MAX_NR_BOOTMODS + 1];
>>   };
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thank you.

v/r,
dps

Re: [PATCH v5 07/44] x86/boot: move headroom to boot modules

[Daniel P. Smith]

On 10/7/24 14:55, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The purpose of struct boot_module is to encapsulate the state of boot 
>> module as
>> it is processed by Xen. Locating boot module state struct boot_module 
>> reduces
>> the number of global variables as well as the number of state 
>> variables that
>> must be passed around. It also lays the groundwork for hyperlaunch 
>> mult-domain
>> construction, where multiple instances of state variables like 
>> headroom will be
>> needed.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h |  5 +++++
>>   xen/arch/x86/setup.c                | 23 ++++++++++++++---------
>>   2 files changed, 19 insertions(+), 9 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h 
>> b/xen/arch/x86/include/asm/bootinfo.h
>> index d19473d8941e..c7e6b4ebf0da 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -17,6 +17,11 @@
>>   struct boot_module {
>>       /* Transitionary only */
>>       module_t *mod;
>> +    /*
>> +     * A boot module may contain a compressed kernel that Xen will 
>> need space
>> +     * reserved, into which it will be decompressed.
> 
> Maybe "Extra space, before the module data, for compressed kernel 
> modules to be decompressed into."

I will rework it with your suggestions.

> And some ascii art could help:
> 
> [ headroom ][ compressed data ]
>   <decompression>
> [ decompressed data ]
> 
> (Not sure how to create a down arrow...)

Yes, in fact I would just show the three states like this:

At boot:
   [ compressed kernel ]
After boot module relocation:
   [ estimated headroom + PAGE_SIZE rounding ][ compressed kernel ]
After kernel decompression:
   [ decompressed kernel ][ unused rounding ]

>> +     */
>> +    unsigned long headroom;
>>   };
>>   /*
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index ba9f110d98c6..dd82ca3d43e2 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1012,7 +1012,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       struct boot_info *bi;
>>       multiboot_info_t *mbi;
>>       module_t *mod;
>> -    unsigned long nr_pages, raw_max_page, modules_headroom, 
>> module_map[1];
>> +    unsigned long nr_pages, raw_max_page, module_map[1];
>>       int i, j, e820_warn = 0, bytes = 0;
>>       unsigned long eb_start, eb_end;
>>       bool acpi_boot_table_init_done = false, relocated = false;
>> @@ -1371,7 +1371,10 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>           mod[bi->nr_modules].mod_end = __2M_rwdata_end - _stext;
>>       }
>> -    modules_headroom = bzimage_headroom(bootstrap_map(mod), 
>> mod->mod_end);
>> +    bi->mods[0].headroom =
>> +        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
>> +                         bi->mods[0].mod->mod_end);
>> +
>>       bootstrap_map(NULL);
>>   #ifndef highmem_start
>> @@ -1456,8 +1459,10 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>                * decompressor overheads of mod[0] (the dom0 kernel).  
>> When we
>>                * move mod[0], we incorporate this as extra space at 
>> the start.
>>                */
>> -            unsigned long headroom = j ? 0 : modules_headroom;
>> -            unsigned long size = PAGE_ALIGN(headroom + mod[j].mod_end);
>> +            struct boot_module *bm = &bi->mods[j];
>> +            unsigned long size;
>> +
>> +            size = PAGE_ALIGN(bm->headroom + mod[j].mod_end);
> 
> Just do
>              unsigned long size = PAGE_ALIGN(bm->headroom + 
> mod[j].mod_end);
> ?

yep, not sure why I even split it.

v/r,
dps

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Daniel P. Smith]

On 10/7/24 15:34, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> To allow a slow conversion of x86 over to struct boot_module, start with
>> replacing all references to struct mod to the early_mod element of struct
>> boot_module. These serves twofold, first to allow the incremental 
>> transition
>> from struct mod fields to struct boot_module fields. The second is to 
>> allow
>> the conversion of function definitions from taking struct mod 
>> parameters to
>> accepting struct boot_module as needed when a transitioned field will be
>> accessed.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 61 ++++++++++++++++++++++++--------------------
>>   1 file changed, 34 insertions(+), 27 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index dd82ca3d43e2..ba4bee6b93af 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1341,15 +1341,15 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
>>       kexec_reserve_area();
>> -    initial_images = mod;
>> +    initial_images = bi->mods[0].mod;
> 
> Isn't this wrong?
> mod is the array of module_t * of *all* modules, but bi->mods[0].mod is 
> a single module_t *?

No it is not wrong:
   bi->mods[0].mod == __va(mbi->mods_addr)[0]

While the modules themselves get relocated, the location of the array of 
module_t never change.

The question does give me pause to double check the patch ordering, just 
to be sure that mod_start and mod_end are correct up until we transition
to using the boot_module fields.

>>       for ( i = 0; !efi_enabled(EFI_LOADER) && i < bi->nr_modules; i++ )
>>       {
>> -        if ( mod[i].mod_start & (PAGE_SIZE - 1) )
>> +        if ( bi->mods[i].mod->mod_start & (PAGE_SIZE - 1) )
>>               panic("Bootloader didn't honor module alignment 
>> request\n");
>> -        mod[i].mod_end -= mod[i].mod_start;
>> -        mod[i].mod_start >>= PAGE_SHIFT;
>> -        mod[i].reserved = 0;
>> +        bi->mods[i].mod->mod_end -= bi->mods[i].mod->mod_start;
>> +        bi->mods[i].mod->mod_start >>= PAGE_SHIFT;
>> +        bi->mods[i].mod->reserved = 0;
>>       }
>>       /*
> 
>> @@ -1509,13 +1510,15 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>   #endif
>>       }
>> -    if ( bi->mods[0].headroom && !mod->reserved )
>> +    if ( bi->mods[0].headroom && !bi->mods[0].mod->reserved )
>>           panic("Not enough memory to relocate the dom0 kernel image\n");
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        uint64_t s = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
>> +        uint64_t s = (uint64_t)bi->mods[i].mod->mod_start
>> +                        << PAGE_SHIFT;
> 
> pfn_to_paddr() ?

Yep, missed one ( ._.)

>> -        reserve_e820_ram(&boot_e820, s, s + PAGE_ALIGN(mod[i].mod_end));
>> +        reserve_e820_ram(&boot_e820, s,
>> +                         s + PAGE_ALIGN(bi->mods[i].mod->mod_end));
>>       }
>>       if ( !xen_phys_start )
>> @@ -1593,8 +1596,9 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>                   map_e = boot_e820.map[j].addr + boot_e820.map[j].size;
>>                   for ( j = 0; j < bi->nr_modules; ++j )
>>                   {
>> -                    uint64_t end = pfn_to_paddr(mod[j].mod_start) +
>> -                                   mod[j].mod_end;
>> +                    uint64_t end = pfn_to_paddr(
>> +                                   bi->mods[j].mod->mod_start) +
>> +                                   bi->mods[j].mod->mod_end;
> 
> I think you want a different indent.  I think
>      uint64_t end = pfn_to_paddr(bi->mods[j].mod->mod_start)
> 
> will all fit on one line (indented all the way).  (Thunderbird makes it 
> difficult me to send indented.)

Yes, it will fit on one line without the '+', but I believe one of the 
many unwritten coding style rules is that the '+' stays with the LHS, so 
I wrapped the LHS with the '+'.

>>                       if ( map_e < end )
>>                           map_e = end;
>> @@ -1668,11 +1672,13 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        set_pdx_range(mod[i].mod_start,
>> -                      mod[i].mod_start + PFN_UP(mod[i].mod_end));
>> -        map_pages_to_xen((unsigned long)mfn_to_virt(mod[i].mod_start),
>> -                         _mfn(mod[i].mod_start),
>> -                         PFN_UP(mod[i].mod_end), PAGE_HYPERVISOR);
>> +        set_pdx_range(bi->mods[i].mod->mod_start,
>> +                      bi->mods[i].mod->mod_start +
>> +                      PFN_UP(bi->mods[i].mod->mod_end));
>> +        map_pages_to_xen(
>> +            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),
> 
> map_pages_to_xen((unsigned long)maddr_to_virt(bi->mods[i].start),
> 
> All fits on one line.

If it does, I will bring it back up.

v/r,
dps

Re: [PATCH v5 09/44] x86/boot: introduce boot module types

[Daniel P. Smith]

On 10/7/24 15:50, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> This commit introduces module types of xen, kernel, and ramdisk to 
>> allow boot
>> module detect code to tag the purpose of a boot module. This reduces 
>> the need
>> for hard coded order assumptions and global variables to be used by 
>> consumers
>> of boot modules, such as domain construction.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
> 
>> @@ -2058,6 +2063,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>              cpu_has_nx ? "" : "not ");
>>       initrdidx = find_first_bit(module_map, bi->nr_modules);
>> +    bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
> 
> This is incorrect if an initrd isn't present.

Correct, will put behind guard.

v/r,
dps

Re: [PATCH v5 10/44] x86/boot: introduce boot module flags

[Daniel P. Smith]

On 10/7/24 16:02, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The existing startup code employs various ad-hoc state tracking about 
>> certain
>> boot module types by each area of the code. A boot module flags is 
>> added to
>> enable tracking these different states.  The first state to be 
>> transition by
>> this commit is module relocation.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h | 4 ++++
>>   xen/arch/x86/setup.c                | 8 ++++----
>>   2 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h 
>> b/xen/arch/x86/include/asm/bootinfo.h
>> index 6941a8975ea6..021ff0d93643 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -31,6 +31,10 @@ struct boot_module {
>>        */
>>       unsigned long headroom;
>>       enum bootmod_type type;
>> +
>> +    uint32_t flags;
>> +#define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
>> +
> 
> Stray newline.  Otherwise:
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Ack and thank you.

v/r,
dps

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Jan Beulich]

On 09.10.2024 16:23, Daniel P. Smith wrote:
> On 10/7/24 15:34, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> --- a/xen/arch/x86/setup.c
>>> +++ b/xen/arch/x86/setup.c
>>> @@ -1341,15 +1341,15 @@ void asmlinkage __init noreturn 
>>> __start_xen(unsigned long mbi_p)
>>>       set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
>>>       kexec_reserve_area();
>>> -    initial_images = mod;
>>> +    initial_images = bi->mods[0].mod;
>>
>> Isn't this wrong?
>> mod is the array of module_t * of *all* modules, but bi->mods[0].mod is 
>> a single module_t *?
> 
> No it is not wrong:
>    bi->mods[0].mod == __va(mbi->mods_addr)[0]

Yet as it's seemingly wrong, a comment appears to be necessary.

Jan

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Daniel P. Smith]

On 10/9/24 10:29, Jan Beulich wrote:
> On 09.10.2024 16:23, Daniel P. Smith wrote:
>> On 10/7/24 15:34, Jason Andryuk wrote:
>>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>>> --- a/xen/arch/x86/setup.c
>>>> +++ b/xen/arch/x86/setup.c
>>>> @@ -1341,15 +1341,15 @@ void asmlinkage __init noreturn
>>>> __start_xen(unsigned long mbi_p)
>>>>        set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT);
>>>>        kexec_reserve_area();
>>>> -    initial_images = mod;
>>>> +    initial_images = bi->mods[0].mod;
>>>
>>> Isn't this wrong?
>>> mod is the array of module_t * of *all* modules, but bi->mods[0].mod is
>>> a single module_t *?
>>
>> No it is not wrong:
>>     bi->mods[0].mod == __va(mbi->mods_addr)[0]
> 
> Yet as it's seemingly wrong, a comment appears to be necessary.

That is fair, will add comment.

v/r,
dps

Re: [PATCH v5 13/44] x86/boot: update struct boot_module on module relocation

[Daniel P. Smith]

On 10/7/24 16:31, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> When a boot module is relocated, ensure struct boot_module start and size
>> fields are updated along with early_mod.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 7 +++++--
>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index 093a4f5380d1..f968758048ed 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1392,8 +1392,11 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>            * respective reserve_e820_ram() invocation below. No need to
>>            * query efi_boot_mem_unused() here, though.
>>            */
>> -        bi->mods[xen].mod->mod_start = virt_to_mfn(_stext);
>> -        bi->mods[xen].mod->mod_end = __2M_rwdata_end - _stext;
>> +        bi->mods[xen].start = virt_to_mfn(_stext);
>> +        bi->mods[xen].size = __2M_rwdata_end - _stext;
> 
> The last patch did:
> bi->mods[i].start = (paddr_t)mods[i].mod_start;
> 
> and start is a paddr_t.
> 
> Is virt_to_mfn() wrong?

Yah, that is seriously wrong. Will fix.

v/r,
dps

Re: [PATCH v5 14/44] x86/boot: transition relocation calculations to struct boot_module

[Daniel P. Smith]

On 10/7/24 16:44, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Use struct boot_module fields, start and size, when calculating the 
>> relocation
>> address and size. It also ensures that early_mod references are kept 
>> in sync.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 36 +++++++++++++++++-------------------
>>   1 file changed, 17 insertions(+), 19 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index f968758048ed..4f540c461b26 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1490,7 +1490,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>               struct boot_module *bm = &bi->mods[j];
>>               unsigned long size;
>> -            size = PAGE_ALIGN(bm->headroom + bm->mod->mod_end);
>> +            size = PAGE_ALIGN(bm->headroom + bm->size);
> 
> Is there a mismatch from mod_end in PFNs to bm->size in bytes?  Or is 
> mod_start in pfns and mod_end in bytes?

The conversion is the latter, mod_start is in PFN and mod_end is in bytes.

>>               if ( bi->mods[j].flags & BOOTMOD_FLAG_X86_RELOCATED )
>>                   continue;
>> @@ -1504,13 +1504,13 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>               if ( s < end &&
>>                    (bm->headroom ||
>> -                  ((end - size) >> PAGE_SHIFT) > bm->mod->mod_start) )
>> +                  paddr_to_pfn(end - size) > paddr_to_pfn(bm->start)) )
> 
> Drop the paddr_to_pfn if both sides are now in bytes?

True, both are in addresses, will drop.

>>               {
>> -                move_memory(end - size + bm->headroom,
>> -                            (uint64_t)bm->mod->mod_start << PAGE_SHIFT,
>> -                            bm->mod->mod_end);
>> -                bm->mod->mod_start = (end - size) >> PAGE_SHIFT;
>> -                bm->mod->mod_end += bm->headroom;
>> +                move_memory(end - size + bm->headroom, bm->start, 
>> bm->size);
>> +                bm->start = (end - size);
>> +                bm->mod->mod_start = paddr_to_pfn(bm->start);
>> +                bm->size += bm->headroom;
>> +                bm->mod->mod_end = bm->size;
>>                   bm->flags |= BOOTMOD_FLAG_X86_RELOCATED;
>>               }
>>           }
> 
>> @@ -1700,13 +1698,13 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        set_pdx_range(bi->mods[i].mod->mod_start,
>> -                      bi->mods[i].mod->mod_start +
>> -                      PFN_UP(bi->mods[i].mod->mod_end));
>> +        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
>> +                      paddr_to_pfn(bi->mods[i].mod->mod_start) +
> 
> Shouldn't these be
>      paddr_to_pfn(bi->mods[i].start)
> ?

Correct, will fix.

>> +                      PFN_UP(bi->mods[i].size));
>>           map_pages_to_xen(
>> -            (unsigned long)mfn_to_virt(bi->mods[i].mod->mod_start),
>> -            _mfn(bi->mods[i].mod->mod_start),
>> -            PFN_UP(bi->mods[i].mod->mod_end), PAGE_HYPERVISOR);
>> +            (unsigned long)maddr_to_virt(bi->mods[i].start),
>> +            maddr_to_mfn(bi->mods[i].start),
>> +            PFN_UP(bi->mods[i].size), PAGE_HYPERVISOR);
> 
> First argument should fit on same line as map_pages_to_xen().

If it fits, yes I will move it up.

v/r,
dps

Re: [PATCH v5 16/44] x86/boot: introduce consumed flag for struct boot_module

[Daniel P. Smith]

On 10/7/24 17:06, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Allow the tracking of when a boot module has been consumed by a 
>> handler in the
>> hypervisor independent of when it is claimed. The instances where the
>> hypervisor does nothing beyond claiming, the dom0 kernel, dom0 
>> ramdisk, and a
>> placeholder for itself, are updated as being consumed at the time of 
>> being
>> claimed.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h | 1 +
>>   xen/arch/x86/setup.c                | 3 +++
>>   2 files changed, 4 insertions(+)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h 
>> b/xen/arch/x86/include/asm/bootinfo.h
>> index c79678840d31..7833b065eff1 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -34,6 +34,7 @@ struct boot_module {
>>       uint32_t flags;
>>   #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
>> +#define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
>>       paddr_t start;
>>       size_t size;
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index 4f540c461b26..235b4e41f653 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -318,6 +318,7 @@ static struct boot_info __init 
>> *multiboot_fill_boot_info(unsigned long mbi_p)
>>       /* map the last mb module for xen entry */
>>       bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
>> +    bi->mods[bi->nr_modules].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>>       bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
>>       return bi;
>> @@ -1196,6 +1197,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       bitmap_fill(module_map, bi->nr_modules);
>>       __clear_bit(0, module_map); /* Dom0 kernel is always first */
>>       bi->mods[0].type = BOOTMOD_KERNEL;
>> +    bi->mods[0].flags |= BOOTMOD_FLAG_X86_CONSUMED;
> 
> I think these first two can be straight assignments since they occur 
> before relocation.

True, will change to straight assignment.

>>       if ( pvh_boot )
>>       {
>> @@ -2085,6 +2087,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       initrdidx = find_first_bit(module_map, bi->nr_modules);
>>       bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
>> +    bi->mods[initrdidx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>>       if ( bitmap_weight(module_map, bi->nr_modules) > 1 )
>>           printk(XENLOG_WARNING
>>                  "Multiple initrd candidates, picking module #%u\n",
> 
> This one is after relocation, so |= is necessary.

I am going to assume that is just a reminder since it is already '|='.

v/r,
dps

Re: [PATCH v5 17/44] x86/boot: convert microcode loading to consume struct boot_info

[Daniel P. Smith]

On 10/7/24 17:22, Jason Andryuk wrote:
>> @@ -822,8 +819,8 @@ static int __init early_update_cache(const void 
>> *data, size_t len)
>>       return rc;
>>   }
>> -int __init microcode_init_cache(unsigned long *module_map,
>> -                                const struct multiboot_info *mbi)
>> +int __init microcode_init_cache(
>> +    unsigned long *module_map, const struct boot_info *bi)
> 
> Why the indent style change?  I prefer the original indent and it 
> doesn't seem like the line length matters.  And it looks like you 
> restore the indent later.

This is one I missed unrolling earlier version. Originally was applying 
the style requested but then realized it was unnecessary churn as by the
end of the series the only param would be *bi and was waiting until the 
last change to move to requested format.

Re: [PATCH v5 17/44] x86/boot: convert microcode loading to consume struct boot_info

[Daniel P. Smith]

On 10/8/24 08:50, Jason Andryuk wrote:
> On 2024-10-07 17:22, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> @@ -205,20 +204,18 @@ static void __init microcode_scan_module(
>>>   }
>>>   static void __init microcode_grab_module(
>>> -    unsigned long *module_map,
>>> -    const multiboot_info_t *mbi)
>>> +    unsigned long *module_map, struct boot_info *bi)
>>>   {
>>> -    module_t *mod = (module_t *)__va(mbi->mods_addr);
>>> -
>>>       if ( ucode_mod_idx < 0 )
>>> -        ucode_mod_idx += mbi->mods_count;
>>> -    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= mbi->mods_count ||
>>> +        ucode_mod_idx += bi->nr_modules;
>>> +    if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
>>>            !__test_and_clear_bit(ucode_mod_idx, module_map) )
>>>           goto scan;
>>> -    ucode_mod = mod[ucode_mod_idx];
>>> +    bi->mods[ucode_mod_idx].type = BOOTMOD_MICROCODE;
>>> +    ucode_mod = *bi->mods[ucode_mod_idx].mod;
>>
>> Why the dereference: *bi->mods[ucode_mod_idx].mod; ?  I don't think it 
>> should be there.
> 
> Oh, the next patch shows ucode_mod is not a pointer, so dereferencing is 
> correct.   Sorry for the noise.

No worries.

v/r,
dps

Re: [PATCH v5 01/44] x86/boot: move x86 boot module counting into a new boot_info struct

[Jan Beulich]

On 07.10.2024 19:57, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> --- /dev/null
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -0,0 +1,29 @@
>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>> +/*
>> + * Copyright (c) 2024 Christopher Clark <christopher.w.clark@gmail.com>
>> + * Copyright (c) 2024 Apertus Solutions, LLC
>> + * Author: Daniel P. Smith <dpsmith@apertussolutions.com>
>> + */
>> +
>> +#ifndef __XEN_X86_BOOTINFO_H__
>> +#define __XEN_X86_BOOTINFO_H__
> 
> I haven't been following closely, but I think if we follow Frediano's 
> naming scheme, it would be:
> ASM__X86__BOOTINFO_H
> 
> With that (or whatever it should be),
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

And then also
Acked-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH v5 02/44] x86/boot: move boot loader name to boot info

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -285,6 +285,9 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>  
>      bi->nr_modules = (mbi->flags & MBI_MODULES) ? mbi->mods_count : 0;
>  
> +    bi->loader = (mbi->flags & MBI_LOADERNAME) ?
> +                            __va(mbi->boot_loader_name) : "unknown";

Either (noting that generally we exempt ?: from the operator-on-earlier-line
rule)

    bi->loader = (mbi->flags & MBI_LOADERNAME) ? __va(mbi->boot_loader_name)
                                               : "unknown";

or 

    bi->loader = (mbi->flags & MBI_LOADERNAME)
                 ? __va(mbi->boot_loader_name) : "unknown";

or

    bi->loader = (mbi->flags & MBI_LOADERNAME)
        ? __va(mbi->boot_loader_name) : "unknown";

(in the order of my personal preference).

Jan

Re: [PATCH v5 04/44] x86/boot: move mmap info to boot info

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -296,6 +296,12 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>      else
>          bi->cmdline = "";
>  
> +    if ( mbi->flags & MBI_MEMMAP )
> +    {
> +        bi->memmap_addr = mbi->mmap_addr;
> +        bi->memmap_length = mbi->mmap_length;
> +    }
> +
>      return bi;
>  }
>  
> @@ -1185,13 +1191,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>      {
>          memmap_type = "Xen-e820";
>      }
> -    else if ( mbi->flags & MBI_MEMMAP )
> +    else if ( bi->memmap_addr )

I'd like to note that this isn't an exact transformation, as with the flag
set the memory map could theoretically also like at address 0. As long as
the legacy BIOS layout of low memory is as it is, that won't happen. I'm
less certain going forward, for legacy-free hardware/firmware. Imo at the
very least this needs mentioning as intentional in the description, for
archeologists to later be able to tell whether this was an oversight.

Or maybe it would be better to check ->memmap_length? That being zero
clearly means there's effectively no map.

Jan

Re: [PATCH v5 05/44] x86/boot: introduce struct boot_module

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> @@ -302,6 +304,13 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>          bi->memmap_length = mbi->mmap_length;
>      }
>  
> +    /*
> +     * This will iterate over all modules to include an extra mb module, which
> +     * should have been reserved to hold an entry for Xen.
> +     */
> +    for ( i = 0; i <= bi->nr_modules; i++ )
> +        bi->mods[i].mod = &mods[i];

I find the comment difficult to follow / match with code here and elsewhere.
How about "Iterate over all modules, including the extra one which should
have been reserved for Xen itself"?

Jan

Re: [PATCH v5 06/44] x86/boot: convert consider_modules to struct boot_module

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> @@ -639,20 +639,20 @@ static uint64_t __init consider_modules(
>  
>      for ( i = 0; i < nr_mods ; ++i )
>      {
> -        uint64_t start = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
> -        uint64_t end = start + PAGE_ALIGN(mod[i].mod_end);
> +        uint64_t start = (uint64_t)pfn_to_paddr(mods[i].mod->mod_start);

With the switch to pfn_to_paddr() the cast isn't needed anymore.

Jan

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> @@ -2061,8 +2067,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>       * We're going to setup domain0 using the module(s) that we stashed safely
>       * above our heap. The second module, if present, is an initrd ramdisk.
>       */
> -    dom0 = create_dom0(mod, bi->mods[0].headroom,
> -                       initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
> +    dom0 = create_dom0(bi->mods[0].mod, bi->mods[0].headroom,
> +                       initrdidx < bi->nr_modules ?
> +                            bi->mods[initrdidx].mod : NULL,

See an earlier comment regarding wrapped ?:. We certainly never have
indentation levels of 5 blanks.

Jan

Re: [PATCH v5 09/44] x86/boot: introduce boot module types

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -311,6 +311,10 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>      for ( i = 0; i <= bi->nr_modules; i++ )
>          bi->mods[i].mod = &mods[i];

This loop, on its last iteration, has done ...

> +    /* map the last mb module for xen entry */
> +    bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
> +    bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];

... this assignment already, hasn't it?

Jan

Re: [PATCH v5 10/44] x86/boot: introduce boot module flags

[Jan Beulich]

On 07.10.2024 22:02, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The existing startup code employs various ad-hoc state tracking about certain
>> boot module types by each area of the code. A boot module flags is added to
>> enable tracking these different states.  The first state to be transition by
>> this commit is module relocation.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h | 4 ++++
>>   xen/arch/x86/setup.c                | 8 ++++----
>>   2 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h b/xen/arch/x86/include/asm/bootinfo.h
>> index 6941a8975ea6..021ff0d93643 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -31,6 +31,10 @@ struct boot_module {
>>        */
>>       unsigned long headroom;
>>       enum bootmod_type type;
>> +
>> +    uint32_t flags;
>> +#define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
>> +
> 
> Stray newline.  Otherwise:
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Acked-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v5 11/44] x86/boot: split bootstrap_map_addr() out of bootstrap_map()

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> From: Andrew Cooper <andrew.cooper3@citrix.com>
> 
> Using an interface based on addresses directly, not modules.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> ---
>  xen/arch/x86/include/asm/setup.h |  1 +
>  xen/arch/x86/setup.c             | 19 +++++++++++++------
>  2 files changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
> index 3d189521189d..213584b05fb2 100644
> --- a/xen/arch/x86/include/asm/setup.h
> +++ b/xen/arch/x86/include/asm/setup.h
> @@ -36,6 +36,7 @@ extern struct boot_info xen_boot_info;
>  
>  unsigned long initial_images_nrpages(nodeid_t node);
>  void discard_initial_images(void);
> +void *bootstrap_map_addr(paddr_t start, paddr_t end);

Nothing is being said about why this function needs a declaration here
and ...

> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -437,24 +437,22 @@ static void __init normalise_cpu_order(void)
>   * Ensure a given physical memory range is present in the bootstrap mappings.
>   * Use superpage mappings to ensure that pagetable memory needn't be allocated.
>   */
> -void *__init bootstrap_map(const module_t *mod)
> +void *__init bootstrap_map_addr(paddr_t start, paddr_t end)

... isn't instead static here. Bugseng folks have put in quite a bit of
effort to remove such anomalies (which Misra doesn't like) from the code
base; I don't think we should introduce new ones. I didn't peek ahead
further than just the next patch, where the function gains a new use,
but could still be static, so it's possible I'm simply missing a
subsequent use from another CU. Yet then the function ought to become
non-static only there.

Jan

Re: [PATCH v5 12/44] x86/boot: add start and size fields to struct boot_module

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> This commit introduces the start and size fields to struct boot_module and adds
> a corresponding bootstrap mapping function, bootstrap_map_bm.

Which then is left with no caller. Misra doesn't like unreachable code.

Jan

Re: [PATCH v5 15/44] x86/boot: introduce boot module interator

[Jan Beulich]

On 06.10.2024 23:49, Daniel P. Smith wrote:
> --- a/xen/arch/x86/include/asm/bootinfo.h
> +++ b/xen/arch/x86/include/asm/bootinfo.h
> @@ -54,8 +54,24 @@ struct boot_info {
>      struct boot_module mods[MAX_NR_BOOTMODS + 1];
>  };
>  
> -#endif /* __XEN_X86_BOOTINFO_H__ */
> +static inline int __init next_boot_module_index(
> +    const struct boot_info *bi, enum bootmod_type t, int offset)

Instead of "offset" maybe better "start" or "from"? Further, plain int
(as also used ...

> +{
> +    int i;

... here) isn't really liked for ...

> +    for ( i = offset; i < bi->nr_modules; i++ )
> +    {
> +        if ( bi->mods[i].type == t )

... array indexing. Perhaps the function itself would better have
unsigned int return type as well, ...

> +            return i;
> +    }
> +
> +    return -1;

... using UINT_MAX or some other suitable constant here instead?

Jan

Re: [PATCH v5 19/44] x86/boot: use consumed boot module flag for microcode

[Daniel P. Smith]

On 10/8/24 11:56, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> To track if the microcode boot module was loaded, a copy of the boot 
>> module is
>> kept. The size element of this copy is set to zero as the indicator 
>> that the
>> microcode was loaded. A side effect is that the modules have to be 
>> rescanned to
>> find the boot module post-relocation, so the cache copy can be created.
>>
>> Use the consumed boot module flag to track the loading of the 
>> microcode boot
>> module. This removes the need to manipulate the boot module size 
>> element, no
>> longer requiring the copy, thus allowing it to be replaced by a 
>> reference. As a
>> result it is no longer necessary to rescan the boot modules after 
>> relocation
>> has occurred.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/cpu/microcode/core.c | 28 ++++++++++++++--------------
>>   1 file changed, 14 insertions(+), 14 deletions(-)
>>
>> diff --git a/xen/arch/x86/cpu/microcode/core.c 
>> b/xen/arch/x86/cpu/microcode/core.c
>> index 7bcc17e0ab2f..5b42aad2fdd0 100644
>> --- a/xen/arch/x86/cpu/microcode/core.c
>> +++ b/xen/arch/x86/cpu/microcode/core.c
> 
>> @@ -826,14 +826,14 @@ int __init microcode_init_cache(
>>       if ( !ucode_ops.apply_microcode )
>>           return -ENODEV;
>> -    if ( ucode_scan )
>> -        /* Need to rescan the modules because they might have been 
>> relocated */
>> +    /* Scan if microcode was not detected earlier */
>> +    if ( !ucode_mod )
> 
> ucode_scan is a user-controlled variable (ucode=scan=$bool), so I think 
> it still needs to be respected.

The ucode_scan was introduced due to the complex situation attempting to 
be addressed. The microcode needs to be loaded earlier before it is 
possible to safely store a cached copy. Multiboot's module_t had no 
method of state tracking, identification and consumption. To address 
this short coming, the early loading made a copy of the module so it 
could use the mod_end field as a flag without breaking the relocation
logic later. And now because it made a copy instead of holding a 
reference, when the relocation occurs, the mod_start is no longer valid. 
I am not sure why the scan was a user exposed flag, but with boot_module
having identification and state, it is no longer necessary to hold a 
copy and a reference can now be used. Since it is now a reference, when 
the relocation occurs, there is no longer a need to rescan because of a 
relocation. I did leave a rescan if there wasn't microcode detected 
during the early load. Though, honestly that probably should go since it 
should be the exact same modules that were scanned during early load.

>>           microcode_scan_module(module_map, bi);
>> -    if ( ucode_mod.size )
>> -        rc = early_update_cache(bootstrap_map_bm(&ucode_mod),
>> -                                ucode_mod.size);
>> -    else if ( ucode_blob.size )
>> +    if ( ucode_mod && !(ucode_mod->flags & BOOTMOD_FLAG_X86_CONSUMED) )
>> +        rc = early_update_cache(bootstrap_map_bm(ucode_mod),
>> +                                ucode_mod->size);
>> +    else if ( ucode_mod && ucode_blob.size )
> 
> ucode_blob seems independent of ucode_mod, so I don't see why this 
> didn't stay `else if ( ucode_blob.size )`
 From my inspection, looks like that should have been an '||" and not a 
'&&'. The reason being is that the function will fall back to ucode_mod 
if ucode_blob is not set.

v/r,
dps

Re: [PATCH v5 20/44] x86/boot: convert xsm policy loading to struct boot_module

[Daniel P. Smith]

On 10/8/24 12:13, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Iterate through the unclaimed struct boot_module to see if any are an 
>> XSM FLASK
>> policy. If one is located, mark it as an xsm policy.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> 
>> @@ -161,6 +162,14 @@ int __init xsm_multiboot_init(
>>       }
>>       ret = xsm_core_init(policy_buffer, policy_size);
>> +    if ( ret == 0 )
>> +    {
>> +        int idx = first_boot_module_index(bi, BOOTMOD_XSM_POLICY);
>> +
>> +        /* If the policy was loaded from a boot module, mark it 
>> consumed */
>> +        if ( idx >= 0 )
>> +            bi->mods[idx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
> 
> Maybe xsm_multiboot_policy_init() should return the idx used instead of 
> having a second search?  (Also, xsm_multiboot_policy_init() can't fail?)

I was debating on whether to make similar changes because the existing 
logic just seems sub-optimal. Currently I am looking to just write an 
independent XSM patch that looks at both this function and the device 
tree version of the function. Specifically, looking to use the 
IS_ENABLED() macro instead of #ifdef to reduce code, provide better code 
coverage, and to refine the logic.

>> +    }
>>       bootstrap_map(NULL);
>>       return 0;
> 
> The other changes look okay.

R-b then?

v/r,
dps

Re: [PATCH v5 22/44] x86/boot: remove module_map usage from microcode loading

[Daniel P. Smith]

On 10/8/24 12:30, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> With all consumers of module_map converted, remove usage of it
>> by the microcode loading logic.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> 
>> @@ -202,19 +201,18 @@ static void __init microcode_scan_module(
>>       }
>>   }
>> -static void __init microcode_grab_module(
>> -    unsigned long *module_map, struct boot_info *bi)
>> +static void __init microcode_grab_module(struct boot_info *bi)
>>   {
>>       if ( ucode_mod_idx < 0 )
>>           ucode_mod_idx += bi->nr_modules;
>>       if ( ucode_mod_idx <= 0 || ucode_mod_idx >= bi->nr_modules ||
>> -         !__test_and_clear_bit(ucode_mod_idx, module_map) )
>> +         (bi->mods[ucode_mod_idx].type != BOOTMOD_UNKNOWN) )
> 
> Just
>           bi->mods[ucode_mod_idx].type != BOOTMOD_UNKNOWN )

Ack

> With that:
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thank you.

v/r,
dps

Re: [PATCH v5 23/44] x86/boot: remove module_map usage from xsm policy loading

[Daniel P. Smith]

On 10/8/24 12:36, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c  | 2 +-
>>   xen/include/xsm/xsm.h | 9 +++------
>>   xen/xsm/xsm_core.c    | 6 ++----
>>   xen/xsm/xsm_policy.c  | 5 +----
>>   4 files changed, 7 insertions(+), 15 deletions(-)
>>
> 
>> diff --git a/xen/xsm/xsm_policy.c b/xen/xsm/xsm_policy.c
>> index 921bb254b9d1..a22367a62e93 100644
>> --- a/xen/xsm/xsm_policy.c
>> +++ b/xen/xsm/xsm_policy.c
> 
>> @@ -42,7 +41,6 @@ int __init xsm_multiboot_policy_init(
>>       /*
>>        * Try all modules and see whichever could be the binary policy.
>> -     * Adjust module_map for the module that is the binary policy.
>>        */
> 
> You can collapse to a single line comment /* ... */

Ack.

> With that:
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thank you.

v/r,
dps

Re: [PATCH v5 24/44] x86/boot: remove module_map usage by ramdisk loading

[Daniel P. Smith]

On 10/8/24 12:46, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The ramdisk loading is the last user of module_map, remove
>> its usage and any remaining remnants of module_map.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 11 +++++------
>>   1 file changed, 5 insertions(+), 6 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index b0946216ea3f..0d2ee19998aa 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -1037,7 +1037,7 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       struct boot_info *bi;
>>       multiboot_info_t *mbi;
>>       module_t *mod;
>> -    unsigned long nr_pages, raw_max_page, module_map[1];
>> +    unsigned long nr_pages, raw_max_page;
>>       int i, j, e820_warn = 0, bytes = 0;
>>       unsigned long eb_start, eb_end;
>>       bool acpi_boot_table_init_done = false, relocated = false;
>> @@ -1187,15 +1187,14 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>           panic("dom0 kernel not specified. Check bootloader 
>> configuration\n");
>>       /* Check that we don't have a silly number of modules. */
>> -    if ( bi->nr_modules > sizeof(module_map) * 8 )
>> +    if ( bi->nr_modules > MAX_NR_BOOTMODS + 1 )
> 
> Don't you want to check MAX_NR_BOOTMODS, to keep the last module for Xen 
> itself?

Good question. I went back to confirm and it does not look like any of 
the module_map bits was used for tracking xen in the module list. so 
yes, drop it back down to just MAX_NR_BOOTMODS.

v/r,
dps

Re: [PATCH v5 25/44] x86/boot: convert create_dom0 to use boot info

[Daniel P. Smith]

On 10/8/24 12:52, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> This commit changes create_dom0 to no longer take the individual 
>> components and
>> take struct boot_info instead. Internally, it is changed to locate the 
>> kernel
>> and ramdisk details from struct boot_info.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 25 ++++++++++++++++---------
>>   1 file changed, 16 insertions(+), 9 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index 0d2ee19998aa..c2bcddc50990 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -946,10 +946,8 @@ static unsigned int __init copy_bios_e820(struct 
>> e820entry *map, unsigned int li
>>       return n;
>>   }
>> -static struct domain *__init create_dom0(const module_t *image,
>> -                                         unsigned long headroom,
>> -                                         module_t *initrd, const char 
>> *kextra,
>> -                                         const char *loader)
>> +static struct domain *__init create_dom0(const struct boot_info *bi,
>> +                                         const char *kextra)
>>   {
>>       static char __initdata cmdline[MAX_GUEST_CMDLINE];
>> @@ -964,9 +962,21 @@ static struct domain *__init create_dom0(const 
>> module_t *image,
>>               .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED 
>> : 0,
>>           },
>>       };
>> +    int headroom, mod_idx = first_boot_module_index(bi, 
>> BOOTMOD_RAMDISK);
> 
> I think headroom should stay unsigned long, which matches struct 
> boot_module.

Yes, that was not intentional. Will preserve the size.

> With that
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thanks!

v/r,
dps

Re: [PATCH v5 28/44] x86/boot: add cmdline to struct boot_module

[Daniel P. Smith]

On 10/8/24 13:08, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Add a char pointer field, cmdline, to struct boot_module to hold the 
>> address
>> pointed to by the string field of struct mod. This removes the need to 
>> use the
>> early_mod field to get to the dom0 kernel command line.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootinfo.h | 2 ++
>>   xen/arch/x86/setup.c                | 9 ++++++---
>>   2 files changed, 8 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootinfo.h 
>> b/xen/arch/x86/include/asm/bootinfo.h
>> index 9ed260629012..3b6bfbe88770 100644
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -38,6 +38,8 @@ struct boot_module {
>>   #define BOOTMOD_FLAG_X86_RELOCATED     (1U << 0)
>>   #define BOOTMOD_FLAG_X86_CONSUMED      (1U << 1)
>> +    const char *cmdline;
>> +
>>       paddr_t start;
>>       size_t size;
>>   };
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index aed0837902c4..d5916e85f68e 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -312,6 +312,8 @@ static struct boot_info __init 
>> *multiboot_fill_boot_info(unsigned long mbi_p)
>>       {
>>           bi->mods[i].mod = &mods[i];
>> +        bi->mods[i].cmdline = (char *)(paddr_t)mods[i].string;
>> +
>>           bi->mods[i].start = (paddr_t)mods[i].mod_start;
>>           bi->mods[i].size = mods[i].mod_end - mods[i].mod_start;
>>       }
>> @@ -1000,10 +1002,11 @@ static struct domain *__init 
>> create_dom0(struct boot_info *bi)
>>           panic("Error creating d%uv0\n", domid);
>>       /* Grab the DOM0 command line. */
>> -    if ( image->mod->string || bi->kextra )
>> +    if ( image->cmdline || bi->kextra )
>>       {
>> -        if ( image->mod->string )
>> -            safe_strcpy(cmdline, cmdline_cook(__va(image->mod->string),
>> +        if ( image->cmdline )
>> +            safe_strcpy(cmdline,
>> +                        cmdline_cook(__va((unsigned 
>> long)image->cmdline),
> 
> char * seems inappropriate if cmdline isn't usable as a string.  Maybe 
> have cmdline as a paddr_t, or can __va() be used at assignment time?
> 

Yah, char * was not the correct choice, paddr_t is more appropriate.

v/r,
dps

Re: [PATCH v5 30/44] x86/boot: convert dom0_construct_pv initrd param to struct boot_module

[Daniel P. Smith]

On 10/8/24 14:30, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> This changes the type for the initrd parameter of dom0_construct_pv to 
>> be struct
>> boot_module. This conversion requires several adjustments throughout 
>> dom0_construct_pv
>> to account for the type change. Removes the usage of early_mod field 
>> for ramdisk module.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> 
>> diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
>> index 7b6afe64d799..16b8c1e40998 100644
>> --- a/xen/arch/x86/pv/dom0_build.c
>> +++ b/xen/arch/x86/pv/dom0_build.c
> 
>> @@ -367,7 +367,8 @@ static int __init dom0_construct(struct domain *d,
>>       unsigned long nr_pt_pages;
>>       unsigned long alloc_spfn;
>>       unsigned long alloc_epfn;
>> -    unsigned long initrd_pfn = -1, initrd_mfn = 0;
>> +    unsigned long initrd_pfn = -1;
>> +    mfn_t  initrd_mfn = { 0 };
> 
> = _mfn(0);

Ack.

> With that:
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thanks!

v/r,
dps

Re: [PATCH v5 33/44] x86/boot: convert initial_images to struct boot_module

[Daniel P. Smith]

On 10/8/24 14:52, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> The variable initial_images is used for tracking the boot modules 
>> passed in by
>> the boot loader. Convert to a struct boot_module and adjust the code 
>> that uses
>> it accordingly.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 15 +++++++++------
>>   1 file changed, 9 insertions(+), 6 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index d5916e85f68e..30a139074833 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
> 
>> @@ -336,8 +336,9 @@ unsigned long __init 
>> initial_images_nrpages(nodeid_t node)
>>       for ( nr = i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        unsigned long start = initial_images[i].mod_start;
>> -        unsigned long end = start + PFN_UP(initial_images[i].mod_end);
>> +        unsigned long start = initial_images[i].mod->mod_start;
>> +        unsigned long end = start +
>> +                            PFN_UP(initial_images[i].mod->mod_end);
> 
> This can fit on a single line.

Ack.

>>           if ( end > node_start && node_end > start )
>>               nr += min(node_end, end) - max(node_start, start);
>> @@ -353,10 +354,12 @@ void __init discard_initial_images(void)
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        uint64_t start = (uint64_t)initial_images[i].mod_start << 
>> PAGE_SHIFT;
>> +        uint64_t start =
>> +            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
>>           init_domheap_pages(start,
>> -                           start + 
>> PAGE_ALIGN(initial_images[i].mod_end));
>> +                           start +
>> +                           PAGE_ALIGN(initial_images[i].mod->mod_end));
> 
> This can fit on a single line.

Ack.

>>       }
>>       bi->nr_modules = 0;
> 
> With those fixed:
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thanks!

v/r,
dps

Re: [PATCH v5 34/44] x86/boot: drop the use of initial_images unit global

[Daniel P. Smith]

On 10/8/24 15:04, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 13 ++++---------
>>   1 file changed, 4 insertions(+), 9 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index 30a139074833..b3b6e6f38622 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -276,8 +276,6 @@ custom_param("acpi", parse_acpi_param);
>>   static const char *cmdline_cook(const char *p, const char 
>> *loader_name);
>> -static const struct boot_module *__initdata initial_images;
>> -
>>   struct boot_info __initdata xen_boot_info;
>>   static struct boot_info __init *multiboot_fill_boot_info(unsigned 
>> long mbi_p)
>> @@ -336,9 +334,9 @@ unsigned long __init 
>> initial_images_nrpages(nodeid_t node)
>>       for ( nr = i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        unsigned long start = initial_images[i].mod->mod_start;
>> +        unsigned long start = bi->mods[i].mod->mod_start;
>>           unsigned long end = start +
>> -                            PFN_UP(initial_images[i].mod->mod_end);
>> +                            PFN_UP(bi->mods[i].mod->mod_end);
> 
> Fits on a single line.

Ack.

>>           if ( end > node_start && node_end > start )
>>               nr += min(node_end, end) - max(node_start, start);
>> @@ -355,15 +353,14 @@ void __init discard_initial_images(void)
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>>           uint64_t start =
>> -            (uint64_t)initial_images[i].mod->mod_start << PAGE_SHIFT;
>> +            (uint64_t)bi->mods[i].mod->mod_start << PAGE_SHIFT;
> 
> Fits on one line.  Can also be pfn_to_paddr(), which applies to earlier 
> patches.  Having said that, maybe it's okay to skip pfn_to_paddr as at 
> the end of the series mods[i].start is used without a shift.  i.e. fewer 
> transformations in these "mechanical" changes make review easier. Unless 
> someone else wants pfn_to_addr(), I am okay without that conversion.

After I did a few based on a comment from Jan on v4, I realized I am 
making changes that just go away or shift to a different conversion 
macro. I would prefer to just leave them to the end and make sure that 
when arriving at the final form, none of them are using any open coded 
forms.

I will move it up to a single line.

>>           init_domheap_pages(start,
>>                              start +
>> -                           PAGE_ALIGN(initial_images[i].mod->mod_end));
>> +                           PAGE_ALIGN(bi->mods[i].mod->mod_end));
> 
> One line.

Ack.

>>       }
>>       bi->nr_modules = 0;
>> -    initial_images = NULL;
>>   }
>>   static void __init init_idle_domain(void)
> 
> With the line fixups:
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thanks!

v/r,
dps

Re: [PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Daniel P. Smith]

On 10/8/24 15:15, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Any direct usages of struct mod have been transitioned, remove the 
>> remaining
>> references to early_mod fields.
> 
> This is unclear, please try to re-word.  "struct mod" and "early_mod" 
> don't exist.

Ack.

>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/setup.c | 31 +++++++++++--------------------
>>   1 file changed, 11 insertions(+), 20 deletions(-)
>>
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index e9e3da3204f1..0ffe8d3ff8dd 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
> 
>> @@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>            */
>>           bi->mods[xen].start = virt_to_mfn(_stext);
>>           bi->mods[xen].size = __2M_rwdata_end - _stext;
>> -
>> -        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
>> -        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>>       }
>> -    bi->mods[0].headroom =
>> -        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
>> -                         bi->mods[0].mod->mod_end);
>> -
>> -    bootstrap_map(NULL);
>> +    bi->mods[0].headroom = bzimage_headroom(
>> +                        bootstrap_map_bm(&bi->mods[0]),
>> +                        bi->mods[0].size);
> 
> Thunderbird might corrupt this, bit the above can fit on two lines:
>      bi->mods[0].headroom = 
> bzimage_headroom(bootstrap_map_bm(&bi->mods[0]),
>                                              bi->mods[0].size);

I actually prefer the same formatting as Jan has suggested, will apply 
that one.

>> +    bootstrap_map_bm(NULL);
>>   #ifndef highmem_start
>>       /* Don't allow split below 4Gb. */
> 
>> @@ -1708,8 +1699,8 @@ void asmlinkage __init noreturn 
>> __start_xen(unsigned long mbi_p)
>>       for ( i = 0; i < bi->nr_modules; ++i )
>>       {
>> -        set_pdx_range(paddr_to_pfn(bi->mods[i].mod->mod_start),
>> -                      paddr_to_pfn(bi->mods[i].mod->mod_start) +
>> +        set_pdx_range(paddr_to_pfn(bi->mods[i].start),
>> +                      paddr_to_pfn(bi->mods[i].start) +
> 
> This belongs in patch 14 as mentioned there.

Ack.

v/r,
dps

Re: [PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Daniel P. Smith]

On 10/9/24 02:53, Jan Beulich wrote:
> On 08.10.2024 21:15, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> Any direct usages of struct mod have been transitioned, remove the remaining
>>> references to early_mod fields.
>>
>> This is unclear, please try to re-word.  "struct mod" and "early_mod"
>> don't exist.
>>
>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>> ---
>>>    xen/arch/x86/setup.c | 31 +++++++++++--------------------
>>>    1 file changed, 11 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>> index e9e3da3204f1..0ffe8d3ff8dd 100644
>>> --- a/xen/arch/x86/setup.c
>>> +++ b/xen/arch/x86/setup.c
>>
>>> @@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>>>             */
>>>            bi->mods[xen].start = virt_to_mfn(_stext);
>>>            bi->mods[xen].size = __2M_rwdata_end - _stext;
>>> -
>>> -        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
>>> -        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>>>        }
>>>    
>>> -    bi->mods[0].headroom =
>>> -        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
>>> -                         bi->mods[0].mod->mod_end);
>>> -
>>> -    bootstrap_map(NULL);
>>> +    bi->mods[0].headroom = bzimage_headroom(
>>> +                        bootstrap_map_bm(&bi->mods[0]),
>>> +                        bi->mods[0].size);
>>
>> Thunderbird might corrupt this, bit the above can fit on two lines:
>>       bi->mods[0].headroom = bzimage_headroom(bootstrap_map_bm(&bi->mods[0]),
>>                                               bi->mods[0].size);
> 
> Or else at least indentation wants to change, to one of the two possible
> forms:
> 
>      bi->mods[0].headroom = bzimage_headroom(
>          bootstrap_map_bm(&bi->mods[0]),
>          bi->mods[0].size);
> 
> (indentation increased by a level from the start of the statement) or
> 
>      bi->mods[0].headroom = bzimage_headroom(
>                                 bootstrap_map_bm(&bi->mods[0]),
>                                 bi->mods[0].size);
> 
> (indentation by one level biased from the start of the function call).
> Personally, if already wrapping like this, I'd prefer the former.

I agree with you, the former is more pleasing, though wouldn't line 3 
fit on line 2?

v/r,
dps

Re: [PATCH v5 39/44] x86/boot: introduce domid field to struct boot_domain

[Daniel P. Smith]

On 10/8/24 15:36, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Add a domid field to struct boot_domain to hold the assigned domain id 
>> for the
>> domain. During initialization, ensure all instances of struct 
>> boot_domain have
>> the invalid domid to ensure that the domid must be set either by 
>> convention or
>> configuration.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootdomain.h |  2 ++
>>   xen/arch/x86/setup.c                  | 12 +++++++-----
>>   2 files changed, 9 insertions(+), 5 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootdomain.h 
>> b/xen/arch/x86/include/asm/bootdomain.h
>> index 4285223ac5ab..d6264d554dba 100644
>> --- a/xen/arch/x86/include/asm/bootdomain.h
>> +++ b/xen/arch/x86/include/asm/bootdomain.h
>> @@ -11,6 +11,8 @@
>>   struct boot_module;
>>   struct boot_domain {
>> +    domid_t domid;
>> +
>>       struct boot_module *kernel;
>>       struct boot_module *ramdisk;
>>   };
> 
> Oh, you should probably move domid after the pointers to avoid a hole.

It's not a packed structure and at this point, we are not looking to 
make it an ABI. If the maintainers think there is a real concern here, 
then I can add a reserved field to fill the hole.

v/r,
dps

Re: [PATCH v5 40/44] x86/boot: add cmdline to struct boot_domain

[Daniel P. Smith]

On 10/8/24 16:05, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Add a container for the "cooked" command line for a domain.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/bootdomain.h |  4 ++++
>>   xen/arch/x86/setup.c                  | 18 ++++++++----------
>>   2 files changed, 12 insertions(+), 10 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/bootdomain.h 
>> b/xen/arch/x86/include/asm/bootdomain.h
>> index d6264d554dba..00f7d9267965 100644
>> --- a/xen/arch/x86/include/asm/bootdomain.h
>> +++ b/xen/arch/x86/include/asm/bootdomain.h
>> @@ -8,9 +8,13 @@
>>   #ifndef __XEN_X86_BOOTDOMAIN_H__
>>   #define __XEN_X86_BOOTDOMAIN_H__
>> +#include <public/xen.h>
>> +
>>   struct boot_module;
>>   struct boot_domain {
>> +    char cmdline[MAX_GUEST_CMDLINE];
>> +
> 
> 1024 bytes for just dom0 isn't too much.  But when hyperlaunch has 64 
> boot_domains, that's a good bit more.  But I suppose it isn't too much 
> RAM for a modern system.  This is __initdata, so it increases the binary 
> size.  I just want to highlight this in case others want to chime in.

I would prefer to leave it in the __initdata for now to avoid dealing 
with dynamic memory at this time. Later, if a need to optimize for 
binary size, then options could be explored.

> The code changes seem fine.

R-b then?

v/r
dps

Re: [PATCH v5 41/44] x86/boot: add struct domain to struct boot_domain

[Daniel P. Smith]

On 10/8/24 15:48, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> Store a reference to the created domain in struct boot_domain.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
> 
>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>> index f250638edf09..e6a231bd2d42 100644
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -959,7 +959,6 @@ static struct domain *__init create_dom0(struct 
>> boot_info *bi)
>>           },
>>       };
>>       struct boot_domain *bd = &bi->domains[0];
>> -    struct domain *d;
>>       if ( opt_dom0_pvh )
>>       {
>> @@ -976,13 +975,13 @@ static struct domain *__init create_dom0(struct 
>> boot_info *bi)
>>       /* Create initial domain.  Not d0 for pvshim. */
>>       bd->domid = get_initial_domain_id();
>> -    d = domain_create(bd->domid, &dom0_cfg, pv_shim ? 0 : 
>> CDF_privileged);
> 
> It's a smaller patch if you keep `d` and the line above and then do:
>         bd->d = d;

Yah, this can be done.

v/r,
dps

Re: [PATCH v5 42/44] x86/boot: convert construct_dom0 to struct boot_domain

[Daniel P. Smith]

On 10/8/24 15:47, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> A struct boot_domain now encapsulates the domain reference, kernel, 
>> ramdisk,
>> and command line for the domain being constructed. As a result of this
>> encapsulation, construct_dom0 can now take a single struct boot_domain 
>> instead
>> of these four parameters.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/dom0_build.c        | 19 +++++++++----------
>>   xen/arch/x86/include/asm/setup.h |  4 +---
>>   xen/arch/x86/setup.c             |  2 +-
>>   3 files changed, 11 insertions(+), 14 deletions(-)
>>
>> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
>> index 71b2e3afc1a1..e552f2e9abef 100644
>> --- a/xen/arch/x86/dom0_build.c
>> +++ b/xen/arch/x86/dom0_build.c
>> @@ -597,22 +597,21 @@ int __init dom0_setup_permissions(struct domain *d)
>>       return rc;
>>   }
>> -int __init construct_dom0(struct domain *d, const struct boot_module 
>> *image,
>> -                          struct boot_module *initrd, const char 
>> *cmdline)
>> +int __init construct_dom0(struct boot_domain *bd)
>>   {
>>       int rc;
> 
> I think a local variable would be better:
> 
>      struct domain *d = bd->d;
> 
> The patch is smaller, and using just d is common in the hypervisor.

Ack.

v/r,
dps

Re: [PATCH v5 43/44] x86/boot: convert dom0_construct_pv to struct boot_domain

[Daniel P. Smith]

On 10/8/24 15:54, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> With construct_dom0 consuming struct boot_domain, continue passing the
>> structure down to dom0_construct_pv.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> 
>> diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
>> index 16b8c1e40998..34974aa7cd56 100644
>> --- a/xen/arch/x86/pv/dom0_build.c
>> +++ b/xen/arch/x86/pv/dom0_build.c
>> @@ -354,10 +354,7 @@ static struct page_info * __init 
>> alloc_chunk(struct domain *d,
>>       return page;
>>   }
>> -static int __init dom0_construct(struct domain *d,
>> -                                 const struct boot_module *image,
>> -                                 struct boot_module *initrd,
>> -                                 const char *cmdline)
>> +static int __init dom0_construct(struct boot_domain *bd)
>>   {
>>       int i, rc, order, machine;
>>       bool compatible, compat;
>> @@ -373,11 +370,12 @@ static int __init dom0_construct(struct domain *d,
>>       struct page_info *page = NULL;
>>       unsigned int flush_flags = 0;
>>       start_info_t *si;
>> -    struct vcpu *v = d->vcpu[0];
>> -    void *image_base = bootstrap_map_bm(image);
>> -    unsigned long image_len = image->size;
>> -    void *image_start = image_base + image->headroom;
>> -    unsigned long initrd_len = initrd ? initrd->size : 0;
>> +    struct domain *d = bd->d;
>> +    struct vcpu *v = bd->d->vcpu[0];
> 
> This can stay:
>      struct vcpu *v = d->vcpu[0];

Ack.

> With that:
> 
> Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Thanks!

v/r,
dps

Re: [PATCH v5 44/44] x86/boot: convert dom0_construct_pvh to struct boot_domain

[Daniel P. Smith]

On 10/8/24 15:56, Jason Andryuk wrote:
> On 2024-10-06 17:49, Daniel P. Smith wrote:
>> With construct_dom0 consuming struct boot_domain, continue passing the
>> structure down to dom0_construct_pvh.
>>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
> 
>> diff --git a/xen/arch/x86/hvm/dom0_build.c 
>> b/xen/arch/x86/hvm/dom0_build.c
>> index a3fd5e762dc4..755e257cdf30 100644
>> --- a/xen/arch/x86/hvm/dom0_build.c
>> +++ b/xen/arch/x86/hvm/dom0_build.c
>> @@ -1299,25 +1299,23 @@ static void __hwdom_init 
>> pvh_setup_mmcfg(struct domain *d)
>>       }
>>   }
>> -int __init dom0_construct_pvh(
>> -    struct domain *d, const struct boot_module *image,
>> -    struct boot_module *initrd, const char *cmdline)
>> +int __init dom0_construct_pvh(const struct boot_domain *bd)
>>   {
>>       paddr_t entry, start_info;
>>       int rc;
> 
> Again, I recommend using a local struct domain *d to cut down on the churn.

Ack.

v/r,
dps

Re: [PATCH v5 02/44] x86/boot: move boot loader name to boot info

[Daniel P. Smith]

On 10/9/24 11:07, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -285,6 +285,9 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>>   
>>       bi->nr_modules = (mbi->flags & MBI_MODULES) ? mbi->mods_count : 0;
>>   
>> +    bi->loader = (mbi->flags & MBI_LOADERNAME) ?
>> +                            __va(mbi->boot_loader_name) : "unknown";
> 
> Either (noting that generally we exempt ?: from the operator-on-earlier-line
> rule)
> 
>      bi->loader = (mbi->flags & MBI_LOADERNAME) ? __va(mbi->boot_loader_name)
>                                                 : "unknown";
> 
> or
> 
>      bi->loader = (mbi->flags & MBI_LOADERNAME)
>                   ? __va(mbi->boot_loader_name) : "unknown";
> 
> or
> 
>      bi->loader = (mbi->flags & MBI_LOADERNAME)
>          ? __va(mbi->boot_loader_name) : "unknown";
> 
> (in the order of my personal preference).

I would prefer the first as well. Will adjust.

v/r,
dps

Re: [PATCH v5 04/44] x86/boot: move mmap info to boot info

[Daniel P. Smith]

On 10/9/24 11:13, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -296,6 +296,12 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>>       else
>>           bi->cmdline = "";
>>   
>> +    if ( mbi->flags & MBI_MEMMAP )
>> +    {
>> +        bi->memmap_addr = mbi->mmap_addr;
>> +        bi->memmap_length = mbi->mmap_length;
>> +    }
>> +
>>       return bi;
>>   }
>>   
>> @@ -1185,13 +1191,13 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>>       {
>>           memmap_type = "Xen-e820";
>>       }
>> -    else if ( mbi->flags & MBI_MEMMAP )
>> +    else if ( bi->memmap_addr )
> 
> I'd like to note that this isn't an exact transformation, as with the flag
> set the memory map could theoretically also like at address 0. As long as
> the legacy BIOS layout of low memory is as it is, that won't happen. I'm
> less certain going forward, for legacy-free hardware/firmware. Imo at the
> very least this needs mentioning as intentional in the description, for
> archeologists to later be able to tell whether this was an oversight.
> 
> Or maybe it would be better to check ->memmap_length? That being zero
> clearly means there's effectively no map.

I think checking memmap_length is a better approach because as you say, 
the only time it will be zero is if no map was provided.

v/r,
dps

Re: [PATCH v5 05/44] x86/boot: introduce struct boot_module

[Daniel P. Smith]

On 10/9/24 11:17, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> @@ -302,6 +304,13 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>>           bi->memmap_length = mbi->mmap_length;
>>       }
>>   
>> +    /*
>> +     * This will iterate over all modules to include an extra mb module, which
>> +     * should have been reserved to hold an entry for Xen.
>> +     */
>> +    for ( i = 0; i <= bi->nr_modules; i++ )
>> +        bi->mods[i].mod = &mods[i];
> 
> I find the comment difficult to follow / match with code here and elsewhere.
> How about "Iterate over all modules, including the extra one which should
> have been reserved for Xen itself"?

Okay.

v/r,
dps

Re: [PATCH v5 06/44] x86/boot: convert consider_modules to struct boot_module

[Daniel P. Smith]

On 10/9/24 11:22, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> @@ -639,20 +639,20 @@ static uint64_t __init consider_modules(
>>   
>>       for ( i = 0; i < nr_mods ; ++i )
>>       {
>> -        uint64_t start = (uint64_t)mod[i].mod_start << PAGE_SHIFT;
>> -        uint64_t end = start + PAGE_ALIGN(mod[i].mod_end);
>> +        uint64_t start = (uint64_t)pfn_to_paddr(mods[i].mod->mod_start);
> 
> With the switch to pfn_to_paddr() the cast isn't needed anymore.

Ack.

v/r,
dps

Re: [PATCH v5 08/44] x86/boot: convert setup.c mod refs to early_mod

[Daniel P. Smith]

On 10/9/24 11:29, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> @@ -2061,8 +2067,9 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>>        * We're going to setup domain0 using the module(s) that we stashed safely
>>        * above our heap. The second module, if present, is an initrd ramdisk.
>>        */
>> -    dom0 = create_dom0(mod, bi->mods[0].headroom,
>> -                       initrdidx < bi->nr_modules ? mod + initrdidx : NULL,
>> +    dom0 = create_dom0(bi->mods[0].mod, bi->mods[0].headroom,
>> +                       initrdidx < bi->nr_modules ?
>> +                            bi->mods[initrdidx].mod : NULL,
> 
> See an earlier comment regarding wrapped ?:. We certainly never have
> indentation levels of 5 blanks.

Can format similar as earlier, and just fyi, that is a stray space from 
wrapping and not a five space indent.

v/r,
dps

Re: [PATCH v5 09/44] x86/boot: introduce boot module types

[Daniel P. Smith]

On 10/9/24 11:30, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -311,6 +311,10 @@ static struct boot_info __init *multiboot_fill_boot_info(unsigned long mbi_p)
>>       for ( i = 0; i <= bi->nr_modules; i++ )
>>           bi->mods[i].mod = &mods[i];
> 
> This loop, on its last iteration, has done ...
> 
>> +    /* map the last mb module for xen entry */
>> +    bi->mods[bi->nr_modules].type = BOOTMOD_XEN;
>> +    bi->mods[bi->nr_modules].mod = &mods[bi->nr_modules];
> 
> ... this assignment already, hasn't it?

Yep, looks like I missed that the rebase of this commit put the 
assignment back in. Will make sure it's not re-inserted.

v/r,
dps

Re: [PATCH v5 11/44] x86/boot: split bootstrap_map_addr() out of bootstrap_map()

[Daniel P. Smith]

On 10/9/24 11:38, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> From: Andrew Cooper <andrew.cooper3@citrix.com>
>>
>> Using an interface based on addresses directly, not modules.
>>
>> No functional change.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>> ---
>>   xen/arch/x86/include/asm/setup.h |  1 +
>>   xen/arch/x86/setup.c             | 19 +++++++++++++------
>>   2 files changed, 14 insertions(+), 6 deletions(-)
>>
>> diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
>> index 3d189521189d..213584b05fb2 100644
>> --- a/xen/arch/x86/include/asm/setup.h
>> +++ b/xen/arch/x86/include/asm/setup.h
>> @@ -36,6 +36,7 @@ extern struct boot_info xen_boot_info;
>>   
>>   unsigned long initial_images_nrpages(nodeid_t node);
>>   void discard_initial_images(void);
>> +void *bootstrap_map_addr(paddr_t start, paddr_t end);
> 
> Nothing is being said about why this function needs a declaration here
> and ...

You are correct, as far as this series is concerned, nothing external 
uses this. Will drop this declaration....

>> --- a/xen/arch/x86/setup.c
>> +++ b/xen/arch/x86/setup.c
>> @@ -437,24 +437,22 @@ static void __init normalise_cpu_order(void)
>>    * Ensure a given physical memory range is present in the bootstrap mappings.
>>    * Use superpage mappings to ensure that pagetable memory needn't be allocated.
>>    */
>> -void *__init bootstrap_map(const module_t *mod)
>> +void *__init bootstrap_map_addr(paddr_t start, paddr_t end)
> 
> ... isn't instead static here. Bugseng folks have put in quite a bit of
> effort to remove such anomalies (which Misra doesn't like) from the code
> base; I don't think we should introduce new ones. I didn't peek ahead
> further than just the next patch, where the function gains a new use,
> but could still be static, so it's possible I'm simply missing a
> subsequent use from another CU. Yet then the function ought to become
> non-static only there.

...and will make this static.

v/r,
dps

Re: [PATCH v5 12/44] x86/boot: add start and size fields to struct boot_module

[Daniel P. Smith]

On 10/9/24 11:39, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> This commit introduces the start and size fields to struct boot_module and adds
>> a corresponding bootstrap mapping function, bootstrap_map_bm.
> 
> Which then is left with no caller. Misra doesn't like unreachable code.

Only until the upcoming commit makes use, but yes these should be 
standalone and thus complaint. I will rework to ensure there is a use of 
the function when it is introduced.

v/r,
dps

Re: [PATCH v5 15/44] x86/boot: introduce boot module interator

[Daniel P. Smith]

On 10/9/24 11:53, Jan Beulich wrote:
> On 06.10.2024 23:49, Daniel P. Smith wrote:
>> --- a/xen/arch/x86/include/asm/bootinfo.h
>> +++ b/xen/arch/x86/include/asm/bootinfo.h
>> @@ -54,8 +54,24 @@ struct boot_info {
>>       struct boot_module mods[MAX_NR_BOOTMODS + 1];
>>   };
>>   
>> -#endif /* __XEN_X86_BOOTINFO_H__ */
>> +static inline int __init next_boot_module_index(
>> +    const struct boot_info *bi, enum bootmod_type t, int offset)
> 
> Instead of "offset" maybe better "start" or "from"? Further, plain int
> (as also used ...

Will change to start.

>> +{
>> +    int i;
> 
> ... here) isn't really liked for ...
> 
>> +    for ( i = offset; i < bi->nr_modules; i++ )
>> +    {
>> +        if ( bi->mods[i].type == t )
> 
> ... array indexing. Perhaps the function itself would better have
> unsigned int return type as well, ...
> 
>> +            return i;
>> +    }
>> +
>> +    return -1;
> 
> ... using UINT_MAX or some other suitable constant here instead?

I was initially going to disagree as returning a value less than zero is 
much more natural/reasonable than UINIT_MAX. But then thinking about it,
another natural value to reflect an error/not found is a value larger 
than MAX_NR_BOOTMODS.

Will switch to unsigned and add code comment that larger than 
MAX_NR_BOOTMODS is error/not found condition.

v/r,
dps

Re: [PATCH v5 25/44] x86/boot: convert create_dom0 to use boot info

[Jan Beulich]

On 10.10.2024 01:02, Daniel P. Smith wrote:
> On 10/8/24 12:52, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> This commit changes create_dom0 to no longer take the individual 
>>> components and
>>> take struct boot_info instead. Internally, it is changed to locate the 
>>> kernel
>>> and ramdisk details from struct boot_info.
>>>
>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>> ---
>>>   xen/arch/x86/setup.c | 25 ++++++++++++++++---------
>>>   1 file changed, 16 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>> index 0d2ee19998aa..c2bcddc50990 100644
>>> --- a/xen/arch/x86/setup.c
>>> +++ b/xen/arch/x86/setup.c
>>> @@ -946,10 +946,8 @@ static unsigned int __init copy_bios_e820(struct 
>>> e820entry *map, unsigned int li
>>>       return n;
>>>   }
>>> -static struct domain *__init create_dom0(const module_t *image,
>>> -                                         unsigned long headroom,
>>> -                                         module_t *initrd, const char 
>>> *kextra,
>>> -                                         const char *loader)
>>> +static struct domain *__init create_dom0(const struct boot_info *bi,
>>> +                                         const char *kextra)
>>>   {
>>>       static char __initdata cmdline[MAX_GUEST_CMDLINE];
>>> @@ -964,9 +962,21 @@ static struct domain *__init create_dom0(const 
>>> module_t *image,
>>>               .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED 
>>> : 0,
>>>           },
>>>       };
>>> +    int headroom, mod_idx = first_boot_module_index(bi, 
>>> BOOTMOD_RAMDISK);
>>
>> I think headroom should stay unsigned long, which matches struct 
>> boot_module.
> 
> Yes, that was not intentional. Will preserve the size.

And just to mention, with what was said on another patch mod_idx also
shouldn't be plain int.

Jan

Re: [PATCH v5 36/44] x86/boot: remove remaining early_mod references

[Jan Beulich]

On 10.10.2024 01:42, Daniel P. Smith wrote:
> On 10/9/24 02:53, Jan Beulich wrote:
>> On 08.10.2024 21:15, Jason Andryuk wrote:
>>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>>> Any direct usages of struct mod have been transitioned, remove the remaining
>>>> references to early_mod fields.
>>>
>>> This is unclear, please try to re-word.  "struct mod" and "early_mod"
>>> don't exist.
>>>
>>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>>> ---
>>>>    xen/arch/x86/setup.c | 31 +++++++++++--------------------
>>>>    1 file changed, 11 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>> index e9e3da3204f1..0ffe8d3ff8dd 100644
>>>> --- a/xen/arch/x86/setup.c
>>>> +++ b/xen/arch/x86/setup.c
>>>
>>>> @@ -1404,16 +1401,12 @@ void asmlinkage __init noreturn __start_xen(unsigned long mbi_p)
>>>>             */
>>>>            bi->mods[xen].start = virt_to_mfn(_stext);
>>>>            bi->mods[xen].size = __2M_rwdata_end - _stext;
>>>> -
>>>> -        bi->mods[xen].mod->mod_start = bi->mods[xen].start;
>>>> -        bi->mods[xen].mod->mod_end = bi->mods[xen].size;
>>>>        }
>>>>    
>>>> -    bi->mods[0].headroom =
>>>> -        bzimage_headroom(bootstrap_map(bi->mods[0].mod),
>>>> -                         bi->mods[0].mod->mod_end);
>>>> -
>>>> -    bootstrap_map(NULL);
>>>> +    bi->mods[0].headroom = bzimage_headroom(
>>>> +                        bootstrap_map_bm(&bi->mods[0]),
>>>> +                        bi->mods[0].size);
>>>
>>> Thunderbird might corrupt this, bit the above can fit on two lines:
>>>       bi->mods[0].headroom = bzimage_headroom(bootstrap_map_bm(&bi->mods[0]),
>>>                                               bi->mods[0].size);
>>
>> Or else at least indentation wants to change, to one of the two possible
>> forms:
>>
>>      bi->mods[0].headroom = bzimage_headroom(
>>          bootstrap_map_bm(&bi->mods[0]),
>>          bi->mods[0].size);
>>
>> (indentation increased by a level from the start of the statement) or
>>
>>      bi->mods[0].headroom = bzimage_headroom(
>>                                 bootstrap_map_bm(&bi->mods[0]),
>>                                 bi->mods[0].size);
>>
>> (indentation by one level biased from the start of the function call).
>> Personally, if already wrapping like this, I'd prefer the former.
> 
> I agree with you, the former is more pleasing, though wouldn't line 3 
> fit on line 2?

Yes, looks like it would.

Jan

Re: [PATCH v5 25/44] x86/boot: convert create_dom0 to use boot info

[Daniel P. Smith]

On 10/10/24 04:03, Jan Beulich wrote:
> On 10.10.2024 01:02, Daniel P. Smith wrote:
>> On 10/8/24 12:52, Jason Andryuk wrote:
>>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>>> This commit changes create_dom0 to no longer take the individual
>>>> components and
>>>> take struct boot_info instead. Internally, it is changed to locate the
>>>> kernel
>>>> and ramdisk details from struct boot_info.
>>>>
>>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>>> ---
>>>>    xen/arch/x86/setup.c | 25 ++++++++++++++++---------
>>>>    1 file changed, 16 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
>>>> index 0d2ee19998aa..c2bcddc50990 100644
>>>> --- a/xen/arch/x86/setup.c
>>>> +++ b/xen/arch/x86/setup.c
>>>> @@ -946,10 +946,8 @@ static unsigned int __init copy_bios_e820(struct
>>>> e820entry *map, unsigned int li
>>>>        return n;
>>>>    }
>>>> -static struct domain *__init create_dom0(const module_t *image,
>>>> -                                         unsigned long headroom,
>>>> -                                         module_t *initrd, const char
>>>> *kextra,
>>>> -                                         const char *loader)
>>>> +static struct domain *__init create_dom0(const struct boot_info *bi,
>>>> +                                         const char *kextra)
>>>>    {
>>>>        static char __initdata cmdline[MAX_GUEST_CMDLINE];
>>>> @@ -964,9 +962,21 @@ static struct domain *__init create_dom0(const
>>>> module_t *image,
>>>>                .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED
>>>> : 0,
>>>>            },
>>>>        };
>>>> +    int headroom, mod_idx = first_boot_module_index(bi,
>>>> BOOTMOD_RAMDISK);
>>>
>>> I think headroom should stay unsigned long, which matches struct
>>> boot_module.
>>
>> Yes, that was not intentional. Will preserve the size.
> 
> And just to mention, with what was said on another patch mod_idx also
> shouldn't be plain int.

Correct, and I would like to think that the compiler would flag a type 
mismatch if I missed it.

v/r,
dps

Re: [PATCH v5 20/44] x86/boot: convert xsm policy loading to struct boot_module

[Jason Andryuk]

On 2024-10-09 13:21, Daniel P. Smith wrote:
> On 10/8/24 12:13, Jason Andryuk wrote:
>> On 2024-10-06 17:49, Daniel P. Smith wrote:
>>> Iterate through the unclaimed struct boot_module to see if any are an 
>>> XSM FLASK
>>> policy. If one is located, mark it as an xsm policy.
>>>
>>> Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
>>
>>> @@ -161,6 +162,14 @@ int __init xsm_multiboot_init(
>>>       }
>>>       ret = xsm_core_init(policy_buffer, policy_size);
>>> +    if ( ret == 0 )
>>> +    {
>>> +        int idx = first_boot_module_index(bi, BOOTMOD_XSM_POLICY);
>>> +
>>> +        /* If the policy was loaded from a boot module, mark it 
>>> consumed */
>>> +        if ( idx >= 0 )
>>> +            bi->mods[idx].flags |= BOOTMOD_FLAG_X86_CONSUMED;
>> Maybe xsm_multiboot_policy_init() should return the idx used instead 
>> of having a second search?  (Also, xsm_multiboot_policy_init() can't 
>> fail?)
> 
> I was debating on whether to make similar changes because the existing 
> logic just seems sub-optimal. Currently I am looking to just write an 
> independent XSM patch that looks at both this function and the device 
> tree version of the function. Specifically, looking to use the 
> IS_ENABLED() macro instead of #ifdef to reduce code, provide better code 
> coverage, and to refine the logic.

Ok.

>>> +    }
>>>       bootstrap_map(NULL);
>>>       return 0;
>>
>> The other changes look okay.
> 
> R-b then?

Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>

Regards,
Jason