[RFCv2 00/38] Hyperlaunch domain builder

[RFCv2 00/38] Hyperlaunch domain builder

[Daniel P. Smith]

NOTE: Sending this series as an RFC as it is a follow-on to the hyperlaunch
dom0 device tree series going through rounds of review right now. This
iteration of the RFC series is based off of v6 of the dom0 device tree series.

The Hyperlaunch domain builder series is the third split out for the
introduction of the Hyperlaunch domain builder logic. These changes focus on
introducing the ability to create multiple PVH domains at boot. The definition
for those domains will come from the Device Tree capability introduced in the
dom0 device tree series.

Significant Changes in v2:
- bzimage renetrant was dropped for a proper refactoring of kernel headroom and
  expansion
- While there is concern on the affect for pv construction, event channel
  allocation has been delayed to a newly introduced builder_finalize function
  called after all construction has completed


Documentation on Hyperlaunch:
https://wiki.xenproject.org/wiki/Hyperlaunch

Original Hyperlaunch v1 patch series:
https://lists.xenproject.org/archives/html/xen-devel/2022-07/msg00345.html

V/r,
Daniel P. Smith

Daniel P. Smith (38):
  maintainers: add new section for hyperlaunch
  x86/hyperlaunch: correct the naming of domain ramdisk field
  x86/hyperlaunch: convert max vcpu determination to domain builder
  x86/hyperlaunch: convert vcpu0 creation to domain builder
  x86/hyperlaunch: move dom0 cpuid policy behind capability check
  x86/hyperlaunch: introduce pvh domain builder
  x86/hyperlaunch: move initial hwdom setup to dom_construct_pvh
  x86/boot: convert dom0 page calculation to use boot domain
  x86/boot: refactor dom0 page calculation
  x86/boot: generalize paging pages calculation
  x86/boot: generalize compute number of domain pages
  x86/hyperlaunch: move page computation to domain builder
  x86/hyperlaunch: move pvh p2m init to domain builder
  x86/hyperlaunch: move iommu init to domain builder
  x86/boot: move and rename sched_setup_dom0_vcpus
  x86/hyperlaunch: move pvh_setup_cpus to domain builder
  x86/boot: rename pvh acpi setup function
  x86/hyperlaunch: add domu memory map construction
  x86/hyperlaunch: move populating p2m under domain builder
  x86/hyperlaunch: move remaining pvh dom0 construction
  x86/hyperlaunch: relocate pvh_steal_ram to domain builder
  x86/hyperlaunch: add domu acpi construction
  x86/boot: export command line processing
  x86/hyperlaunch: convert create_dom0 to arch_create_dom
  x86/hyperlaunch: remove dom0-isms from arch_create_dom
  x86/hyperlaunch: introduce domain builder general dom creation
  x86/hyperlaunch: introduce arch builder finalize
  x86/hyperlaunch: allocate xenstore for domu
  x86/hyperlaunch: allocate console for domu
  x86/hyperlaunch: introduce concept of core domains
  common/gzip: add function to read isize field
  x86/hyperlaunch: move headroom under domain builder
  x86/hyperlaunch: move kernel extraction under domain builder
  x86/hyperlaunch: introduce multidomain kconfig option
  x86/hyperlaunch: add multidomain construction logic
  x86/hyperlaunch: enable unpausing mulitple domains
  x86/hyperlaunch: generalize domid assignment
  tools: introduce hyperlaunch domain late init

 .gitignore                                |    1 +
 MAINTAINERS                               |   11 +
 tools/helpers/Makefile                    |   12 +
 tools/helpers/late-init-domains.c         |  364 +++++++
 tools/helpers/late-init-domains.h         |   18 +
 tools/helpers/xs-helpers.c                |  117 +++
 tools/helpers/xs-helpers.h                |   26 +
 xen/arch/x86/Makefile                     |    1 +
 xen/arch/x86/bzimage.c                    |   87 +-
 xen/arch/x86/dom0_build.c                 |  120 +--
 xen/arch/x86/domain-builder/Makefile      |    2 +
 xen/arch/x86/domain-builder/core.c        |  140 +++
 xen/arch/x86/domain-builder/domain.c      |  480 ++++++++++
 xen/arch/x86/hvm/Makefile                 |    1 +
 xen/arch/x86/hvm/dom0_build.c             |  600 +-----------
 xen/arch/x86/hvm/dom_build.c              | 1047 +++++++++++++++++++++
 xen/arch/x86/include/asm/boot-domain.h    |   24 +-
 xen/arch/x86/include/asm/bootinfo.h       |   28 +-
 xen/arch/x86/include/asm/bzimage.h        |    6 +-
 xen/arch/x86/include/asm/dom0_build.h     |   19 +-
 xen/arch/x86/include/asm/domain-builder.h |   33 +
 xen/arch/x86/include/asm/setup.h          |    4 +-
 xen/arch/x86/pv/dom0_build.c              |   21 +-
 xen/arch/x86/setup.c                      |  214 +----
 xen/common/domain-builder/Kconfig         |   12 +
 xen/common/domain-builder/fdt.c           |   40 +-
 xen/common/gzip/gunzip.c                  |   12 +
 xen/common/sched/core.c                   |   12 -
 xen/include/xen/domain-builder.h          |   14 +
 xen/include/xen/gunzip.h                  |   12 +
 xen/include/xen/sched.h                   |    1 -
 31 files changed, 2526 insertions(+), 953 deletions(-)
 create mode 100644 tools/helpers/late-init-domains.c
 create mode 100644 tools/helpers/late-init-domains.h
 create mode 100644 tools/helpers/xs-helpers.c
 create mode 100644 tools/helpers/xs-helpers.h
 create mode 100644 xen/arch/x86/domain-builder/Makefile
 create mode 100644 xen/arch/x86/domain-builder/core.c
 create mode 100644 xen/arch/x86/domain-builder/domain.c
 create mode 100644 xen/arch/x86/hvm/dom_build.c
 create mode 100644 xen/arch/x86/include/asm/domain-builder.h

-- 
2.30.2

[RFCv2 01/38] maintainers: add new section for hyperlaunch

[Daniel P. Smith]

Add new section to MAINTAINERS for hyperlaunch, including the files
specifically added to this point under the hyperlaunch work.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 MAINTAINERS | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index c11b82eca98f..d9a85a0b818c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -343,6 +343,17 @@ F:	tools/misc/xenhypfs.c
 F:	xen/common/hypfs.c
 F:	xen/include/xen/hypfs.h
 
+HYPERLAUNCH
+M:	Daniel P. Smith <dpsmith@apertussolutions.com>
+M:	Christopher Clark <christopher.w.clark@gmail.com>
+S:	Supported
+F:	xen/common/domain-builder
+F:	xen/include/xen/domain-builder.h
+F:	xen/x86/domain-builder/
+F:	xen/x86/include/asm/bootinfo.h
+F:	xen/x86/include/asm/boot-domain.h
+F:	xen/x86/include/asm/domain-builder.h
+
 IMX8QM/QXP SUPPORT
 R:	John Ernberg <john.ernberg@actia.se>
 F:	xen/arch/arm/platforms/imx8qm.c
-- 
2.30.2

[RFCv2 02/38] x86/hyperlaunch: correct the naming of domain ramdisk field

[Daniel P. Smith]

The ramdisk field was incorrectly renamed to module without providing a sound
justification. Doing so creates an unnecessary indirection that can cause more
confusion than utility. The only way the field is populated is via a match of a
boot module of type BOOTMOD_RAMDISK. All usages of the field are cast into a
variables named initrd. The attempt to generalize the field name under the
guise that it could be multiplexed for other module types did so without a
valid usage. The result is there is no consideration of how that multiplexing
would even work or be deconflict with the simultaneous presence of a ramdisk.

Moving the field name back to ramdisk to make the current code flow logical. At
a later time should there be a use case that arises where additional modules
need to be passed to a domain, a more appropriate framework will be crafted
that will like be more complicated than just renaming the field to something
other than ramdisk.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c          | 2 +-
 xen/arch/x86/include/asm/boot-domain.h | 2 +-
 xen/arch/x86/pv/dom0_build.c           | 2 +-
 xen/arch/x86/setup.c                   | 2 +-
 xen/common/domain-builder/fdt.c        | 8 ++++----
 5 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index a038e58c118b..0664f0543fd6 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -648,7 +648,7 @@ static int __init pvh_load_kernel(
 {
     struct domain *d = bd->d;
     struct boot_module *image = bd->kernel;
-    struct boot_module *initrd = bd->module;
+    struct boot_module *initrd = bd->ramdisk;
     void *image_base = bootstrap_map_bm(image);
     void *image_start = image_base + image->headroom;
     unsigned long image_len = image->size;
diff --git a/xen/arch/x86/include/asm/boot-domain.h b/xen/arch/x86/include/asm/boot-domain.h
index cb5351241a62..740bfb74121c 100644
--- a/xen/arch/x86/include/asm/boot-domain.h
+++ b/xen/arch/x86/include/asm/boot-domain.h
@@ -28,7 +28,7 @@ struct boot_domain {
     unsigned int max_vcpus;
 
     struct boot_module *kernel;
-    struct boot_module *module;
+    struct boot_module *ramdisk;
     const char *cmdline;
 
     struct domain *d;
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index e1b78d47c218..3b2baf057b75 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -375,7 +375,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
     struct vcpu *v = d->vcpu[0];
 
     struct boot_module *image = bd->kernel;
-    struct boot_module *initrd = bd->module;
+    struct boot_module *initrd = bd->ramdisk;
     void *image_base;
     unsigned long image_len;
     void *image_start;
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 6a1e874b2ecf..afc133b4d562 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -2187,7 +2187,7 @@ void asmlinkage __init noreturn __start_xen(void)
     if ( !bi->hyperlaunch_enabled && initrdidx < MAX_NR_BOOTMODS )
     {
         bi->mods[initrdidx].type = BOOTMOD_RAMDISK;
-        bi->domains[0].module = &bi->mods[initrdidx];
+        bi->domains[0].ramdisk = &bi->mods[initrdidx];
         if ( first_boot_module_index(bi, BOOTMOD_UNKNOWN) < MAX_NR_BOOTMODS )
             printk(XENLOG_WARNING
                    "Multiple initrd candidates, picking module #%u\n",
diff --git a/xen/common/domain-builder/fdt.c b/xen/common/domain-builder/fdt.c
index 232621ae46f9..1b3492571b15 100644
--- a/xen/common/domain-builder/fdt.c
+++ b/xen/common/domain-builder/fdt.c
@@ -374,10 +374,10 @@ static int __init fdt_process_domain_node(
         {
             int idx;
 
-            if ( bd->module )
+            if ( bd->ramdisk )
             {
                 printk(XENLOG_WARNING
-                       "Duplicate module for domain %s\n", name);
+                       "Duplicate multiboot,ramdisk for domain %s\n", name);
                 continue;
             }
 
@@ -390,9 +390,9 @@ static int __init fdt_process_domain_node(
                 return idx;
             }
 
-            printk(XENLOG_INFO "  module: multiboot-index=%d\n", idx);
+            printk(XENLOG_INFO "  ramdisk: multiboot-index=%d\n", idx);
             bi->mods[idx].type = BOOTMOD_RAMDISK;
-            bd->module = &bi->mods[idx];
+            bd->ramdisk = &bi->mods[idx];
         }
     }
 
-- 
2.30.2

[RFCv2 03/38] x86/hyperlaunch: convert max vcpu determination to domain builder

[Daniel P. Smith]

The domain configuration may request more vcpus than are present in the system.
For dom0, the function dom0_max_vcpus() was used to clamp down to physically
available vcpus. Here we are introducing a generalized version,
dom_max_vcpus(), that takes a boot domain and sets the max vcpus based on the
lesser of the requested max and the available vcpus.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/Makefile                |  1 +
 xen/arch/x86/domain-builder/Makefile |  1 +
 xen/arch/x86/domain-builder/domain.c | 40 ++++++++++++++++++++++++++++
 xen/arch/x86/setup.c                 |  4 +--
 xen/include/xen/domain-builder.h     |  3 +++
 5 files changed, 47 insertions(+), 2 deletions(-)
 create mode 100644 xen/arch/x86/domain-builder/Makefile
 create mode 100644 xen/arch/x86/domain-builder/domain.c

diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index c2f1dcf301d6..80269c5e6b45 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -31,6 +31,7 @@ obj-y += desc.o
 obj-bin-y += dmi_scan.init.o
 obj-y += domain.o
 obj-bin-y += dom0_build.init.o
+obj-y += domain-builder/
 obj-y += domain_page.o
 obj-y += e820.o
 obj-y += emul-i8254.o
diff --git a/xen/arch/x86/domain-builder/Makefile b/xen/arch/x86/domain-builder/Makefile
new file mode 100644
index 000000000000..0c2e7085e21b
--- /dev/null
+++ b/xen/arch/x86/domain-builder/Makefile
@@ -0,0 +1 @@
+obj-y += domain.init.o
diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
new file mode 100644
index 000000000000..d8eba73dac28
--- /dev/null
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025, Apertus Solutions, LLC
+ */
+
+#include <xen/cpumask.h>
+#include <xen/domain.h>
+#include <xen/init.h>
+#include <xen/sched.h>
+
+#include <public/bootfdt.h>
+
+#include <asm/bootinfo.h>
+
+unsigned int __init dom_max_vcpus(struct boot_domain *bd)
+{
+    unsigned int limit = bd->mode & BUILD_MODE_PARAVIRT ?
+                                    MAX_VIRT_CPUS : HVM_MAX_VCPUS;
+
+    if ( bd->capabilities & DOMAIN_CAPS_CONTROL )
+        limit = dom0_max_vcpus();
+    else
+        limit = min(limit,
+                    (uint32_t)cpumask_weight(cpupool_valid_cpus(cpupool0)));
+
+    if ( bd->max_vcpus == 0 || bd->max_vcpus > limit )
+        bd->max_vcpus = limit;
+
+    return bd->max_vcpus;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index afc133b4d562..d3bde6c43e8d 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1020,18 +1020,18 @@ static struct domain *__init create_dom0(struct boot_info *bi)
 {
     char *cmdline = NULL;
     size_t cmdline_size;
+    struct boot_domain *bd = &bi->domains[0];
     struct xen_domctl_createdomain dom0_cfg = {
         .flags = IS_ENABLED(CONFIG_TBOOT) ? XEN_DOMCTL_CDF_s3_integrity : 0,
         .max_evtchn_port = -1,
         .max_grant_frames = -1,
         .max_maptrack_frames = -1,
         .grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version),
-        .max_vcpus = dom0_max_vcpus(),
+        .max_vcpus = dom_max_vcpus(bd),
         .arch = {
             .misc_flags = opt_dom0_msr_relaxed ? XEN_X86_MSR_RELAXED : 0,
         },
     };
-    struct boot_domain *bd = &bi->domains[0];
     struct domain *d;
 
     if ( opt_dom0_pvh ||
diff --git a/xen/include/xen/domain-builder.h b/xen/include/xen/domain-builder.h
index 350e2eb2af8c..82c929cc48a1 100644
--- a/xen/include/xen/domain-builder.h
+++ b/xen/include/xen/domain-builder.h
@@ -5,6 +5,7 @@
 #include <xen/types.h>
 
 struct boot_info;
+struct boot_domain;
 
 /* Return status of builder_init(). Shows which boot mechanism was detected */
 enum fdt_kind
@@ -34,4 +35,6 @@ size_t builder_get_cmdline_size(const struct boot_info *bi, int offset);
 int builder_get_cmdline(const struct boot_info *bi, int offset,
                         char *cmdline, size_t size);
 
+unsigned int dom_max_vcpus(struct boot_domain *bd);
+
 #endif /* __XEN_DOMAIN_BUILDER_H__ */
-- 
2.30.2

[RFCv2 04/38] x86/hyperlaunch: convert vcpu0 creation to domain builder

[Daniel P. Smith]

Convert alloc_dom0_vcpu0() to dom0_set_affinity(), making it only set up the
node affinity based on command line parameters passed. At the same time,
introduce alloc_dom_vcpu0() as the replacement for alloc_dom0_vcpu(). Then have
alloc_dom_vcpu0() call dom0_set_affinity() when the boot domain is the control
domain, otherwise set the affinity to auto.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>

---

Changes in v2:
- name vcpu0 allocation function domain_vcpu0_create()
---
 xen/arch/x86/dom0_build.c              |  4 +---
 xen/arch/x86/domain-builder/domain.c   | 11 +++++++++++
 xen/arch/x86/include/asm/boot-domain.h |  9 +++++++++
 xen/arch/x86/include/asm/dom0_build.h  |  2 ++
 xen/arch/x86/setup.c                   |  5 +++--
 xen/include/xen/domain-builder.h       |  1 +
 6 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index f734104c7488..22d9ff3f1e8c 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -254,12 +254,10 @@ unsigned int __init dom0_max_vcpus(void)
     return max_vcpus;
 }
 
-struct vcpu *__init alloc_dom0_vcpu0(struct domain *dom0)
+void __init dom0_set_affinity(struct domain *dom0)
 {
     dom0->node_affinity = dom0_nodes;
     dom0->auto_node_affinity = !dom0_nr_pxms;
-
-    return vcpu_create(dom0, 0);
 }
 
 #ifdef CONFIG_SHADOW_PAGING
diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
index d8eba73dac28..0512dde54746 100644
--- a/xen/arch/x86/domain-builder/domain.c
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -11,6 +11,7 @@
 #include <public/bootfdt.h>
 
 #include <asm/bootinfo.h>
+#include <asm/dom0_build.h>
 
 unsigned int __init dom_max_vcpus(struct boot_domain *bd)
 {
@@ -29,6 +30,16 @@ unsigned int __init dom_max_vcpus(struct boot_domain *bd)
     return bd->max_vcpus;
 }
 
+struct vcpu *__init domain_vcpu0_create(struct boot_domain *bd)
+{
+    if ( has_dom0_caps(bd) )
+        dom0_set_affinity(bd->d);
+    else
+        bd->d->auto_node_affinity = true;
+
+    return vcpu_create(bd->d, 0);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/x86/include/asm/boot-domain.h b/xen/arch/x86/include/asm/boot-domain.h
index 740bfb74121c..c60b19c58aef 100644
--- a/xen/arch/x86/include/asm/boot-domain.h
+++ b/xen/arch/x86/include/asm/boot-domain.h
@@ -9,6 +9,7 @@
 #define __XEN_X86_BOOTDOMAIN_H__
 
 #include <public/xen.h>
+#include <public/bootfdt.h>
 
 struct boot_domain {
     domid_t domid;
@@ -34,6 +35,14 @@ struct boot_domain {
     struct domain *d;
 };
 
+static inline bool __init has_dom0_caps(const struct boot_domain *bd)
+{
+    unsigned int dom0_caps = (DOMAIN_CAPS_CONTROL | DOMAIN_CAPS_HARDWARE | \
+                             DOMAIN_CAPS_XENSTORE);
+
+    return (bd->capabilities & dom0_caps) == dom0_caps;
+}
+
 #endif
 
 /*
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index ff021c24af9d..426def4115ce 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -8,6 +8,8 @@
 
 extern unsigned int dom0_memflags;
 
+void dom0_set_affinity(struct domain *dom0);
+
 unsigned long dom0_compute_nr_pages(struct domain *d,
                                     struct elf_dom_parms *parms,
                                     unsigned long initrd_len);
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index d3bde6c43e8d..409630089d29 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1057,9 +1057,11 @@ static struct domain *__init create_dom0(struct boot_info *bi)
     if ( IS_ERR(d) )
         panic("Error creating d%u: %ld\n", bd->domid, PTR_ERR(d));
 
+    bd->d = d;
+
     init_dom0_cpuid_policy(d);
 
-    if ( alloc_dom0_vcpu0(d) == NULL )
+    if ( domain_vcpu0_create(bd) == NULL )
         panic("Error creating %pdv0\n", d);
 
     cmdline_size = domain_cmdline_size(bi, bd);
@@ -1102,7 +1104,6 @@ static struct domain *__init create_dom0(struct boot_info *bi)
         bd->cmdline = cmdline;
     }
 
-    bd->d = d;
     if ( construct_dom0(bd) != 0 )
         panic("Could not construct domain 0\n");
 
diff --git a/xen/include/xen/domain-builder.h b/xen/include/xen/domain-builder.h
index 82c929cc48a1..d3390d368afb 100644
--- a/xen/include/xen/domain-builder.h
+++ b/xen/include/xen/domain-builder.h
@@ -36,5 +36,6 @@ int builder_get_cmdline(const struct boot_info *bi, int offset,
                         char *cmdline, size_t size);
 
 unsigned int dom_max_vcpus(struct boot_domain *bd);
+struct vcpu *domain_vcpu0_create(struct boot_domain *bd);
 
 #endif /* __XEN_DOMAIN_BUILDER_H__ */
-- 
2.30.2

[RFCv2 05/38] x86/hyperlaunch: move dom0 cpuid policy behind capability check

[Daniel P. Smith]

To incrementally convert create_dom0() into being a generalized domain
construction function, move the dom0 specific cpuid policy behind the control
domain capability.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/setup.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index 409630089d29..298e27848dda 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -1059,7 +1059,8 @@ static struct domain *__init create_dom0(struct boot_info *bi)
 
     bd->d = d;
 
-    init_dom0_cpuid_policy(d);
+    if ( has_dom0_caps(bd) )
+        init_dom0_cpuid_policy(bd->d);
 
     if ( domain_vcpu0_create(bd) == NULL )
         panic("Error creating %pdv0\n", d);
-- 
2.30.2

[RFCv2 06/38] x86/hyperlaunch: introduce pvh domain builder

[Daniel P. Smith]

Introduce dom_construct_pvh() as a wrapper around dom0_construct_pvh(). This
function will be expanded as dom0 specific construction functions are
generalized.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c                 |  5 ++--
 xen/arch/x86/hvm/Makefile                 |  1 +
 xen/arch/x86/hvm/dom_build.c              | 31 +++++++++++++++++++++++
 xen/arch/x86/include/asm/domain-builder.h |  8 ++++++
 xen/arch/x86/include/asm/setup.h          |  2 +-
 5 files changed, 44 insertions(+), 3 deletions(-)
 create mode 100644 xen/arch/x86/hvm/dom_build.c
 create mode 100644 xen/arch/x86/include/asm/domain-builder.h

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 22d9ff3f1e8c..2cf6535ce190 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -15,6 +15,7 @@
 #include <asm/amd.h>
 #include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
+#include <asm/domain-builder.h>
 #include <asm/guest.h>
 #include <asm/hpet.h>
 #include <asm/hvm/emulate.h>
@@ -613,7 +614,7 @@ int __init dom0_setup_permissions(struct domain *d)
     return rc;
 }
 
-int __init construct_dom0(const struct boot_domain *bd)
+int __init construct_dom0(struct boot_domain *bd)
 {
     int rc;
     const struct domain *d = bd->d;
@@ -637,7 +638,7 @@ int __init construct_dom0(const struct boot_domain *bd)
         opt_dom0_max_vcpus_max = bd->max_vcpus;
 
     if ( is_hvm_domain(d) )
-        rc = dom0_construct_pvh(bd);
+        rc = dom_construct_pvh(bd);
     else if ( is_pv_domain(d) )
         rc = dom0_construct_pv(bd);
     else
diff --git a/xen/arch/x86/hvm/Makefile b/xen/arch/x86/hvm/Makefile
index 6ec2c8f2db56..0830a5b2a56f 100644
--- a/xen/arch/x86/hvm/Makefile
+++ b/xen/arch/x86/hvm/Makefile
@@ -5,6 +5,7 @@ obj-y += viridian/
 obj-y += asid.o
 obj-y += dm.o
 obj-bin-y += dom0_build.init.o
+obj-bin-y += dom_build.init.o
 obj-y += domain.o
 obj-y += emulate.o
 obj-$(CONFIG_GRANT_TABLE) += grant_table.o
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
new file mode 100644
index 000000000000..8546cfff1fbf
--- /dev/null
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * hvm/dom_build.c
+ *
+ * Dom builder for PVH guest.
+ *
+ * Copyright (C) 2017 Citrix Systems R&D
+ * Copyright (C) 2025 Apertus Solutions, LLC
+ */
+
+#include <xen/init.h>
+
+#include <asm/bootinfo.h>
+#include <asm/dom0_build.h>
+
+int __init dom_construct_pvh(struct boot_domain *bd)
+{
+    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", bd->domid);
+
+    return dom0_construct_pvh(bd);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
new file mode 100644
index 000000000000..dd429fc9ff8b
--- /dev/null
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -0,0 +1,8 @@
+#ifndef __XEN_X86_DOMBUILDER_H__
+#define __XEN_X86_DOMBUILDER_H__
+
+struct boot_domain;
+
+int dom_construct_pvh(struct boot_domain *bd);
+
+#endif
diff --git a/xen/arch/x86/include/asm/setup.h b/xen/arch/x86/include/asm/setup.h
index ac34c698551e..b517da6144de 100644
--- a/xen/arch/x86/include/asm/setup.h
+++ b/xen/arch/x86/include/asm/setup.h
@@ -27,7 +27,7 @@ void subarch_init_memory(void);
 void init_IRQ(void);
 
 struct boot_domain;
-int construct_dom0(const struct boot_domain *bd);
+int construct_dom0(struct boot_domain *bd);
 
 void setup_io_bitmap(struct domain *d);
 
-- 
2.30.2

[RFCv2 07/38] x86/hyperlaunch: move initial hwdom setup to dom_construct_pvh

[Daniel P. Smith]

Relocate the initial block of hwdom setup code from dom0_construct_pvh() over
to dom_construct_pvh().

No functional change.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c | 44 ------------------------------
 xen/arch/x86/hvm/dom_build.c  | 50 +++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 44 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 0664f0543fd6..4f2eb0f97aba 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -1311,56 +1311,12 @@ static int __init pvh_setup_acpi(struct domain *d, paddr_t start_info)
     return 0;
 }
 
-static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
-{
-    unsigned int i;
-    int rc;
-
-    for ( i = 0; i < pci_mmcfg_config_num; i++ )
-    {
-        rc = register_vpci_mmcfg_handler(d, pci_mmcfg_config[i].address,
-                                         pci_mmcfg_config[i].start_bus_number,
-                                         pci_mmcfg_config[i].end_bus_number,
-                                         pci_mmcfg_config[i].pci_segment);
-        if ( rc )
-            printk("Unable to setup MMCFG handler at %#lx for segment %u\n",
-                   pci_mmcfg_config[i].address,
-                   pci_mmcfg_config[i].pci_segment);
-    }
-}
-
 int __init dom0_construct_pvh(const struct boot_domain *bd)
 {
     paddr_t entry, start_info;
     struct domain *d = bd->d;
     int rc;
 
-    printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", d->domain_id);
-
-    if ( bd->kernel == NULL )
-        panic("Missing kernel boot module for %pd construction\n", d);
-
-    if ( is_hardware_domain(d) )
-    {
-        /*
-         * MMCFG initialization must be performed before setting domain
-         * permissions, as the MCFG areas must not be part of the domain IOMEM
-         * accessible regions.
-         */
-        pvh_setup_mmcfg(d);
-
-        /*
-         * Setup permissions early so that calls to add MMIO regions to the
-         * p2m as part of vPCI setup don't fail due to permission checks.
-         */
-        rc = dom0_setup_permissions(d);
-        if ( rc )
-        {
-            printk("%pd unable to setup permissions: %d\n", d, rc);
-            return rc;
-        }
-    }
-
     /*
      * Craft dom0 physical memory map and set the paging allocation. This must
      * be done before the iommu initializion, since iommu initialization code
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index 8546cfff1fbf..a5f259b6352d 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -8,15 +8,65 @@
  * Copyright (C) 2025 Apertus Solutions, LLC
  */
 
+#include <xen/acpi.h>
 #include <xen/init.h>
+#include <xen/types.h>
+
+#include <acpi/actables.h>
 
 #include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
+#include <asm/hvm/io.h>
+#include <asm/pci.h>
+
+static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
+{
+    unsigned int i;
+    int rc;
+
+    for ( i = 0; i < pci_mmcfg_config_num; i++ )
+    {
+        rc = register_vpci_mmcfg_handler(d, pci_mmcfg_config[i].address,
+                                         pci_mmcfg_config[i].start_bus_number,
+                                         pci_mmcfg_config[i].end_bus_number,
+                                         pci_mmcfg_config[i].pci_segment);
+        if ( rc )
+            printk("Unable to setup MMCFG handler at %#lx for segment %u\n",
+                   pci_mmcfg_config[i].address,
+                   pci_mmcfg_config[i].pci_segment);
+    }
+}
 
 int __init dom_construct_pvh(struct boot_domain *bd)
 {
+    int rc;
+
     printk(XENLOG_INFO "*** Building a PVH Dom%d ***\n", bd->domid);
 
+    if ( bd->kernel == NULL )
+        panic("Missing kernel boot module for %pd construction\n", bd->d);
+
+    if ( is_hardware_domain(bd->d) )
+    {
+        /*
+         * MMCFG initialization must be performed before setting domain
+         * permissions, as the MCFG areas must not be part of the domain IOMEM
+         * accessible regions.
+         */
+        pvh_setup_mmcfg(bd->d);
+
+        /*
+         * Setup permissions early so that calls to add MMIO regions to the
+         * p2m as part of vPCI setup don't fail due to permission checks.
+         */
+        rc = dom0_setup_permissions(bd->d);
+        if ( rc )
+        {
+            printk("%pd unable to setup permissions: %d\n", bd->d, rc);
+            return rc;
+        }
+    }
+
     return dom0_construct_pvh(bd);
 }
 
-- 
2.30.2

[RFCv2 08/38] x86/boot: convert dom0 page calculation to use boot domain

[Daniel P. Smith]

This commit seeks to rework the dom0_compute_nr_pages() function to consume a
boot domain structure that may contain requested memory pages, min pages, max
pages, and the reference for the initrd. With the passing of the boot domain
struct, the initrd_size parameter is dropped. This takes into account the PVH
case, where the value 0 was passed, which is safe as initrd_size is only used
behind the is_pv_domain() check.

It introduces the calculate_dom0_pages() function that handles the command line
override of the memory pages, min pages, and max pages values. The function
also applies a clamping of memory pages to the min/max pages value.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             | 62 ++++++++++++++++-----------
 xen/arch/x86/hvm/dom0_build.c         | 12 +++---
 xen/arch/x86/include/asm/dom0_build.h | 10 ++---
 xen/arch/x86/pv/dom0_build.c          |  6 +--
 4 files changed, 51 insertions(+), 39 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 2cf6535ce190..02a3c2d249c3 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -330,16 +330,37 @@ static unsigned long __init default_nr_pages(unsigned long avail)
                             : min(avail / 16, 128UL << (20 - PAGE_SHIFT)));
 }
 
-unsigned long __init dom0_compute_nr_pages(
-    struct domain *d, struct elf_dom_parms *parms, unsigned long initrd_len)
+static void __init calculate_dom0_pages(
+    struct boot_domain *bd, unsigned long avail)
 {
-    nodeid_t node;
-    unsigned long avail = 0, nr_pages, min_pages, max_pages, iommu_pages = 0;
+    unsigned long nr_pages = bd->mem_pages ?: default_nr_pages(avail);
 
     /* The ordering of operands is to work around a clang5 issue. */
     if ( CONFIG_DOM0_MEM[0] && !dom0_mem_set )
         parse_dom0_mem(CONFIG_DOM0_MEM);
 
+    if ( dom0_mem_set )
+    {
+        nr_pages = get_memsize(&dom0_size, avail) ?: default_nr_pages(avail);
+        bd->min_pages = get_memsize(&dom0_min_size, avail);
+        bd->max_pages = get_memsize(&dom0_max_size, avail);
+    }
+
+    nr_pages = max(nr_pages, bd->min_pages);
+    nr_pages = min(nr_pages, bd->max_pages);
+    nr_pages = min(nr_pages, avail);
+
+    bd->mem_pages = nr_pages;
+}
+
+unsigned long __init dom0_compute_nr_pages(
+    struct boot_domain *bd, struct elf_dom_parms *parms)
+{
+    nodeid_t node;
+    struct domain *d = bd->d;
+    unsigned long initrd_len = bd->ramdisk ? bd->ramdisk->size : 0;
+    unsigned long avail = 0, iommu_pages = 0;
+
     for_each_node_mask ( node, dom0_nodes )
         avail += avail_domheap_pages_region(node, 0, 0) +
                  initial_images_nrpages(node);
@@ -366,17 +387,13 @@ unsigned long __init dom0_compute_nr_pages(
     {
         unsigned long cpu_pages;
 
-        nr_pages = get_memsize(&dom0_size, avail) ?: default_nr_pages(avail);
-
         /*
          * Clamp according to min/max limits and available memory
          * (preliminary).
          */
-        nr_pages = max(nr_pages, get_memsize(&dom0_min_size, avail));
-        nr_pages = min(nr_pages, get_memsize(&dom0_max_size, avail));
-        nr_pages = min(nr_pages, avail);
+        calculate_dom0_pages(bd, avail);
 
-        cpu_pages = dom0_paging_pages(d, nr_pages);
+        cpu_pages = dom0_paging_pages(d, bd->mem_pages);
 
         if ( !iommu_use_hap_pt(d) )
             avail -= cpu_pages;
@@ -384,18 +401,12 @@ unsigned long __init dom0_compute_nr_pages(
             avail -= cpu_pages - iommu_pages;
     }
 
-    nr_pages = get_memsize(&dom0_size, avail) ?: default_nr_pages(avail);
-    min_pages = get_memsize(&dom0_min_size, avail);
-    max_pages = get_memsize(&dom0_max_size, avail);
-
     /* Clamp according to min/max limits and available memory (final). */
-    nr_pages = max(nr_pages, min_pages);
-    nr_pages = min(nr_pages, max_pages);
-    nr_pages = min(nr_pages, avail);
+    calculate_dom0_pages(bd, avail);
 
     if ( is_pv_domain(d) &&
          (parms->p2m_base == UNSET_ADDR) && !memsize_gt_zero(&dom0_size) &&
-         (!memsize_gt_zero(&dom0_min_size) || (nr_pages > min_pages)) )
+         (!memsize_gt_zero(&dom0_min_size) || (bd->mem_pages > bd->min_pages)) )
     {
         /*
          * Legacy Linux kernels (i.e. such without a XEN_ELFNOTE_INIT_P2M
@@ -412,7 +423,7 @@ unsigned long __init dom0_compute_nr_pages(
         vend = round_pgup(parms->virt_kend);
         if ( !parms->unmapped_initrd )
             vend += round_pgup(initrd_len);
-        end = vend + nr_pages * sizeof_long;
+        end = vend + bd->mem_pages * sizeof_long;
 
         if ( end > vstart )
             end += end - vstart;
@@ -420,16 +431,17 @@ unsigned long __init dom0_compute_nr_pages(
              (sizeof_long < sizeof(end) && end > (1UL << (8 * sizeof_long))) )
         {
             end = sizeof_long >= sizeof(end) ? 0 : 1UL << (8 * sizeof_long);
-            nr_pages = (end - vend) / (2 * sizeof_long);
-            if ( memsize_gt_zero(&dom0_min_size) && nr_pages < min_pages )
-                nr_pages = min_pages;
-            printk("Dom0 memory clipped to %lu pages\n", nr_pages);
+            bd->mem_pages = (end - vend) / (2 * sizeof_long);
+            if ( memsize_gt_zero(&dom0_min_size) &&
+                 bd->mem_pages < bd->min_pages )
+                bd->mem_pages = bd->min_pages;
+            printk("Dom0 memory clipped to %lu pages\n", bd->mem_pages);
         }
     }
 
-    d->max_pages = min_t(unsigned long, max_pages, UINT_MAX);
+    d->max_pages = min_t(unsigned long, bd->max_pages, UINT_MAX);
 
-    return nr_pages;
+    return bd->mem_pages;
 }
 
 static void __init process_dom0_ioports_disable(struct domain *dom0)
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 4f2eb0f97aba..1f229d7bded1 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -398,15 +398,15 @@ static __init void pvh_setup_e820(struct domain *d, unsigned long nr_pages)
     ASSERT(cur_pages == nr_pages);
 }
 
-static void __init pvh_init_p2m(struct domain *d)
+static void __init pvh_init_p2m(struct boot_domain *bd)
 {
-    unsigned long nr_pages = dom0_compute_nr_pages(d, NULL, 0);
+    unsigned long nr_pages = dom0_compute_nr_pages(bd, NULL);
     bool preempted;
 
-    pvh_setup_e820(d, nr_pages);
+    pvh_setup_e820(bd->d, nr_pages);
     do {
         preempted = false;
-        paging_set_allocation(d, dom0_paging_pages(d, nr_pages),
+        paging_set_allocation(bd->d, dom0_paging_pages(bd->d, nr_pages),
                               &preempted);
         process_pending_softirqs();
     } while ( preempted );
@@ -1311,7 +1311,7 @@ static int __init pvh_setup_acpi(struct domain *d, paddr_t start_info)
     return 0;
 }
 
-int __init dom0_construct_pvh(const struct boot_domain *bd)
+int __init dom0_construct_pvh(struct boot_domain *bd)
 {
     paddr_t entry, start_info;
     struct domain *d = bd->d;
@@ -1322,7 +1322,7 @@ int __init dom0_construct_pvh(const struct boot_domain *bd)
      * be done before the iommu initializion, since iommu initialization code
      * will likely add mappings required by devices to the p2m (ie: RMRRs).
      */
-    pvh_init_p2m(d);
+    pvh_init_p2m(bd);
 
     iommu_hwdom_init(d);
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 426def4115ce..dcf71c032a17 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -10,14 +10,14 @@ extern unsigned int dom0_memflags;
 
 void dom0_set_affinity(struct domain *dom0);
 
-unsigned long dom0_compute_nr_pages(struct domain *d,
-                                    struct elf_dom_parms *parms,
-                                    unsigned long initrd_len);
 int dom0_setup_permissions(struct domain *d);
 
 struct boot_domain;
-int dom0_construct_pv(const struct boot_domain *bd);
-int dom0_construct_pvh(const struct boot_domain *bd);
+unsigned long dom0_compute_nr_pages(
+    struct boot_domain *bd, struct elf_dom_parms *parms);
+
+int dom0_construct_pv(struct boot_domain *bd);
+int dom0_construct_pvh(struct boot_domain *bd);
 
 unsigned long dom0_paging_pages(const struct domain *d,
                                 unsigned long nr_pages);
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 3b2baf057b75..350a60b1e8fd 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -355,7 +355,7 @@ static struct page_info * __init alloc_chunk(struct domain *d,
     return page;
 }
 
-static int __init dom0_construct(const struct boot_domain *bd)
+static int __init dom0_construct(struct boot_domain *bd)
 {
     unsigned int i;
     int rc, order, machine;
@@ -503,7 +503,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
         }
     }
 
-    nr_pages = dom0_compute_nr_pages(d, &parms, initrd_len);
+    nr_pages = dom0_compute_nr_pages(bd, &parms);
 
 #ifdef CONFIG_PV32
     if ( elf_32bit(&elf) )
@@ -1070,7 +1070,7 @@ out:
     return rc;
 }
 
-int __init dom0_construct_pv(const struct boot_domain *bd)
+int __init dom0_construct_pv(struct boot_domain *bd)
 {
     unsigned long cr4 = read_cr4();
     int rc;
-- 
2.30.2

[RFCv2 09/38] x86/boot: refactor dom0 page calculation

[Daniel P. Smith]

Caution is needed when dom0 is being constructed as PV using an older kernel
that does not have the elf note XEN_ELFNOTE_INIT_P2M. The logic for handling
this situation is embedded directly and takes into account whether dom0 memory
parameters were specified using the negative allocation syntax. To prepare for
generalizing domain page allocation, isolate this logic to a separate handling
function.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c | 76 +++++++++++++++++++++------------------
 1 file changed, 41 insertions(+), 35 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 02a3c2d249c3..a72064fbae80 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -353,12 +353,50 @@ static void __init calculate_dom0_pages(
     bd->mem_pages = nr_pages;
 }
 
+static void __init dom0_pv_restrict_pages(
+    struct boot_domain *bd, struct elf_dom_parms *parms)
+{
+    if ( (parms->p2m_base == UNSET_ADDR) && !memsize_gt_zero(&dom0_size) &&
+         (!memsize_gt_zero(&dom0_min_size) || (bd->mem_pages > bd->min_pages)) )
+    {
+        /*
+         * Legacy Linux kernels (i.e. such without a XEN_ELFNOTE_INIT_P2M
+         * note) require that there is enough virtual space beyond the initial
+         * allocation to set up their initial page tables. This space is
+         * roughly the same size as the p2m table, so make sure the initial
+         * allocation doesn't consume more than about half the space that's
+         * available between params.virt_base and the address space end.
+         */
+        unsigned long vstart, vend, end;
+        unsigned long initrd_len = bd->ramdisk ? bd->ramdisk->size : 0;
+        size_t sizeof_long = is_pv_32bit_domain(bd->d) ? sizeof(int) : sizeof(long);
+
+        vstart = parms->virt_base;
+        vend = round_pgup(parms->virt_kend);
+        if ( !parms->unmapped_initrd )
+            vend += round_pgup(initrd_len);
+        end = vend + bd->mem_pages * sizeof_long;
+
+        if ( end > vstart )
+            end += end - vstart;
+        if ( end <= vstart ||
+             (sizeof_long < sizeof(end) && end > (1UL << (8 * sizeof_long))) )
+        {
+            end = sizeof_long >= sizeof(end) ? 0 : 1UL << (8 * sizeof_long);
+            bd->mem_pages = (end - vend) / (2 * sizeof_long);
+            if ( memsize_gt_zero(&dom0_min_size) &&
+                 bd->mem_pages < bd->min_pages )
+                bd->mem_pages = bd->min_pages;
+            printk("Dom0 memory clipped to %lu pages\n", bd->mem_pages);
+        }
+    }
+}
+
 unsigned long __init dom0_compute_nr_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms)
 {
     nodeid_t node;
     struct domain *d = bd->d;
-    unsigned long initrd_len = bd->ramdisk ? bd->ramdisk->size : 0;
     unsigned long avail = 0, iommu_pages = 0;
 
     for_each_node_mask ( node, dom0_nodes )
@@ -404,40 +442,8 @@ unsigned long __init dom0_compute_nr_pages(
     /* Clamp according to min/max limits and available memory (final). */
     calculate_dom0_pages(bd, avail);
 
-    if ( is_pv_domain(d) &&
-         (parms->p2m_base == UNSET_ADDR) && !memsize_gt_zero(&dom0_size) &&
-         (!memsize_gt_zero(&dom0_min_size) || (bd->mem_pages > bd->min_pages)) )
-    {
-        /*
-         * Legacy Linux kernels (i.e. such without a XEN_ELFNOTE_INIT_P2M
-         * note) require that there is enough virtual space beyond the initial
-         * allocation to set up their initial page tables. This space is
-         * roughly the same size as the p2m table, so make sure the initial
-         * allocation doesn't consume more than about half the space that's
-         * available between params.virt_base and the address space end.
-         */
-        unsigned long vstart, vend, end;
-        size_t sizeof_long = is_pv_32bit_domain(d) ? sizeof(int) : sizeof(long);
-
-        vstart = parms->virt_base;
-        vend = round_pgup(parms->virt_kend);
-        if ( !parms->unmapped_initrd )
-            vend += round_pgup(initrd_len);
-        end = vend + bd->mem_pages * sizeof_long;
-
-        if ( end > vstart )
-            end += end - vstart;
-        if ( end <= vstart ||
-             (sizeof_long < sizeof(end) && end > (1UL << (8 * sizeof_long))) )
-        {
-            end = sizeof_long >= sizeof(end) ? 0 : 1UL << (8 * sizeof_long);
-            bd->mem_pages = (end - vend) / (2 * sizeof_long);
-            if ( memsize_gt_zero(&dom0_min_size) &&
-                 bd->mem_pages < bd->min_pages )
-                bd->mem_pages = bd->min_pages;
-            printk("Dom0 memory clipped to %lu pages\n", bd->mem_pages);
-        }
-    }
+    if ( is_pv_domain(d) )
+        dom0_pv_restrict_pages(bd, parms);
 
     d->max_pages = min_t(unsigned long, bd->max_pages, UINT_MAX);
 
-- 
2.30.2

[RFCv2 10/38] x86/boot: generalize paging pages calculation

[Daniel P. Smith]

Modeling after libxl__get_required_paging_memory(), refactor
dom0_paging_pages() to calculate the number of paging pages required for a
domain that is not the control or hardware domain. As the function is being
refactored, rename to dom_paging_pages() and move under the domain builder.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c                 | 17 +----------------
 xen/arch/x86/domain-builder/domain.c      | 20 ++++++++++++++++++++
 xen/arch/x86/hvm/dom0_build.c             |  3 ++-
 xen/arch/x86/include/asm/dom0_build.h     |  3 ---
 xen/arch/x86/include/asm/domain-builder.h |  3 +++
 xen/arch/x86/pv/dom0_build.c              |  3 ++-
 6 files changed, 28 insertions(+), 21 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index a72064fbae80..0bcdfcb97e6c 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -305,21 +305,6 @@ boolean_param("ro-hpet", ro_hpet);
 
 unsigned int __initdata dom0_memflags = MEMF_no_dma|MEMF_exact_node;
 
-unsigned long __init dom0_paging_pages(const struct domain *d,
-                                       unsigned long nr_pages)
-{
-    /* Keep in sync with libxl__get_required_paging_memory(). */
-    unsigned long memkb = nr_pages * (PAGE_SIZE / 1024);
-
-    memkb = 4 * (256 * d->max_vcpus +
-                 (is_pv_domain(d) ? opt_dom0_shadow || opt_pv_l1tf_hwdom
-                                  : 1 + opt_dom0_shadow) *
-                 (memkb / 1024));
-
-    return DIV_ROUND_UP(memkb, 1024) << (20 - PAGE_SHIFT);
-}
-
-
 /*
  * If allocation isn't specified, reserve 1/16th of available memory for
  * things like DMA buffers. This reservation is clamped to a maximum of 128MB.
@@ -431,7 +416,7 @@ unsigned long __init dom0_compute_nr_pages(
          */
         calculate_dom0_pages(bd, avail);
 
-        cpu_pages = dom0_paging_pages(d, bd->mem_pages);
+        cpu_pages = dom_paging_pages(bd, bd->mem_pages);
 
         if ( !iommu_use_hap_pt(d) )
             avail -= cpu_pages;
diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
index 0512dde54746..a2e5807b60a5 100644
--- a/xen/arch/x86/domain-builder/domain.c
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -12,6 +12,8 @@
 
 #include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
+#include <asm/paging.h>
+#include <asm/spec_ctrl.h>
 
 unsigned int __init dom_max_vcpus(struct boot_domain *bd)
 {
@@ -40,6 +42,24 @@ struct vcpu *__init domain_vcpu0_create(struct boot_domain *bd)
     return vcpu_create(bd->d, 0);
 }
 
+unsigned long __init dom_paging_pages(
+    const struct boot_domain *bd, unsigned long nr_pages)
+{
+    /* Keep in sync with libxl__get_required_paging_memory(). */
+    unsigned long memkb = bd->mem_pages * (PAGE_SIZE / 1024);
+    unsigned long factor = 0;
+
+    if ( has_dom0_caps(bd) )
+        factor = is_pv_domain(bd->d) ? opt_dom0_shadow || opt_pv_l1tf_hwdom
+                                     : 1 + opt_dom0_shadow;
+    else
+        factor = !is_pv_domain(bd->d) + !paging_mode_hap(bd->d);
+
+    memkb = 4 * (256 * bd->d->max_vcpus + (factor * (memkb / 1024)));
+
+    return DIV_ROUND_UP(memkb, 1024) << (20 - PAGE_SHIFT);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 1f229d7bded1..3f0d157f82c8 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -19,6 +19,7 @@
 #include <asm/bootinfo.h>
 #include <asm/bzimage.h>
 #include <asm/dom0_build.h>
+#include <asm/domain-builder.h>
 #include <asm/hvm/support.h>
 #include <asm/io_apic.h>
 #include <asm/p2m.h>
@@ -406,7 +407,7 @@ static void __init pvh_init_p2m(struct boot_domain *bd)
     pvh_setup_e820(bd->d, nr_pages);
     do {
         preempted = false;
-        paging_set_allocation(bd->d, dom0_paging_pages(bd->d, nr_pages),
+        paging_set_allocation(bd->d, dom_paging_pages(bd, nr_pages),
                               &preempted);
         process_pending_softirqs();
     } while ( preempted );
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index dcf71c032a17..81717b49b4ae 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -19,9 +19,6 @@ unsigned long dom0_compute_nr_pages(
 int dom0_construct_pv(struct boot_domain *bd);
 int dom0_construct_pvh(struct boot_domain *bd);
 
-unsigned long dom0_paging_pages(const struct domain *d,
-                                unsigned long nr_pages);
-
 void dom0_update_physmap(bool compat, unsigned long pfn,
                          unsigned long mfn, unsigned long vphysmap_s);
 
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
index dd429fc9ff8b..c5a71fae5ccb 100644
--- a/xen/arch/x86/include/asm/domain-builder.h
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -3,6 +3,9 @@
 
 struct boot_domain;
 
+unsigned long dom_paging_pages(
+    const struct boot_domain *d, unsigned long nr_pages);
+
 int dom_construct_pvh(struct boot_domain *bd);
 
 #endif
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 350a60b1e8fd..f8844b858082 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -17,6 +17,7 @@
 #include <asm/bootinfo.h>
 #include <asm/bzimage.h>
 #include <asm/dom0_build.h>
+#include <asm/domain-builder.h>
 #include <asm/guest.h>
 #include <asm/page.h>
 #include <asm/pv/mm.h>
@@ -1043,7 +1044,7 @@ static int __init dom0_construct(struct boot_domain *bd)
     {
         bool preempted;
 
-        nr_pt_pages = dom0_paging_pages(d, nr_pages);
+        nr_pt_pages = dom_paging_pages(bd, nr_pages);
 
         do {
             preempted = false;
-- 
2.30.2

[RFCv2 11/38] x86/boot: generalize compute number of domain pages

[Daniel P. Smith]

The amount of pages for a domain to be allocated is based on the physical nodes
a domain may be scheduled. For dom0, this can be restricted down from available
nodes via the dom0_nodes command line parameter.

Refactor dom0_compute_nr_pages() such that only apply the dom0_nodes
restriction only if the domain has the control domain or hardware domain
capability flag set. In doing so, also rename the function to
dom_compute_nr_pages().

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c             | 16 +++++++++++++---
 xen/arch/x86/hvm/dom0_build.c         |  2 +-
 xen/arch/x86/include/asm/dom0_build.h |  2 +-
 xen/arch/x86/pv/dom0_build.c          |  2 +-
 4 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 0bcdfcb97e6c..2a22cd4e125e 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -377,14 +377,23 @@ static void __init dom0_pv_restrict_pages(
     }
 }
 
-unsigned long __init dom0_compute_nr_pages(
+unsigned long __init dom_compute_nr_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms)
 {
     nodeid_t node;
+    nodemask_t nodes = { 0 };
     struct domain *d = bd->d;
     unsigned long avail = 0, iommu_pages = 0;
 
-    for_each_node_mask ( node, dom0_nodes )
+    nodes_or(nodes, nodes, node_online_map);
+
+    /* If building dom0 or hwdom, apply command line restriction. */
+    if ( has_dom0_caps(bd) )
+        nodes_and(nodes, nodes, dom0_nodes);
+
+    ASSERT(nodes_weight(nodes) != 0);
+
+    for_each_node_mask ( node, nodes )
         avail += avail_domheap_pages_region(node, 0, 0) +
                  initial_images_nrpages(node);
 
@@ -396,7 +405,8 @@ unsigned long __init dom0_compute_nr_pages(
         avail -= d->max_vcpus - 1;
 
     /* Reserve memory for iommu_dom0_init() (rough estimate). */
-    if ( is_iommu_enabled(d) && !iommu_hwdom_passthrough )
+    if ( is_hardware_domain(d) && is_iommu_enabled(d)
+         && !iommu_hwdom_passthrough )
     {
         unsigned int s;
 
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 3f0d157f82c8..1e63e19589a1 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -401,7 +401,7 @@ static __init void pvh_setup_e820(struct domain *d, unsigned long nr_pages)
 
 static void __init pvh_init_p2m(struct boot_domain *bd)
 {
-    unsigned long nr_pages = dom0_compute_nr_pages(bd, NULL);
+    unsigned long nr_pages = dom_compute_nr_pages(bd, NULL);
     bool preempted;
 
     pvh_setup_e820(bd->d, nr_pages);
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 81717b49b4ae..7275bcf9ba6b 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -13,7 +13,7 @@ void dom0_set_affinity(struct domain *dom0);
 int dom0_setup_permissions(struct domain *d);
 
 struct boot_domain;
-unsigned long dom0_compute_nr_pages(
+unsigned long dom_compute_nr_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms);
 
 int dom0_construct_pv(struct boot_domain *bd);
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index f8844b858082..ad4d1cc3520c 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -504,7 +504,7 @@ static int __init dom0_construct(struct boot_domain *bd)
         }
     }
 
-    nr_pages = dom0_compute_nr_pages(bd, &parms);
+    nr_pages = dom_compute_nr_pages(bd, &parms);
 
 #ifdef CONFIG_PV32
     if ( elf_32bit(&elf) )
-- 
2.30.2

[RFCv2 12/38] x86/hyperlaunch: move page computation to domain builder

[Daniel P. Smith]

The function dom_compute_nr_pages() is being moved to the domain builder. For
this to happen, the variable dom0_nodes, and the functions
calculate_dom0_pages() and dom0_pv_restrict_pages() must be exported.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/dom0_build.c                 | 75 +----------------------
 xen/arch/x86/domain-builder/domain.c      | 71 +++++++++++++++++++++
 xen/arch/x86/include/asm/dom0_build.h     |  4 +-
 xen/arch/x86/include/asm/domain-builder.h |  4 ++
 4 files changed, 81 insertions(+), 73 deletions(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 2a22cd4e125e..75969887b933 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -209,7 +209,7 @@ static int __init cf_check parse_dom0_nodes(const char *s)
 custom_param("dom0_nodes", parse_dom0_nodes);
 
 cpumask_t __initdata dom0_cpus;
-static nodemask_t __initdata dom0_nodes;
+nodemask_t __initdata dom0_nodes;
 
 unsigned int __init dom0_max_vcpus(void)
 {
@@ -315,8 +315,7 @@ static unsigned long __init default_nr_pages(unsigned long avail)
                             : min(avail / 16, 128UL << (20 - PAGE_SHIFT)));
 }
 
-static void __init calculate_dom0_pages(
-    struct boot_domain *bd, unsigned long avail)
+void __init calculate_dom0_pages(struct boot_domain *bd, unsigned long avail)
 {
     unsigned long nr_pages = bd->mem_pages ?: default_nr_pages(avail);
 
@@ -338,7 +337,7 @@ static void __init calculate_dom0_pages(
     bd->mem_pages = nr_pages;
 }
 
-static void __init dom0_pv_restrict_pages(
+void __init dom0_pv_restrict_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms)
 {
     if ( (parms->p2m_base == UNSET_ADDR) && !memsize_gt_zero(&dom0_size) &&
@@ -377,74 +376,6 @@ static void __init dom0_pv_restrict_pages(
     }
 }
 
-unsigned long __init dom_compute_nr_pages(
-    struct boot_domain *bd, struct elf_dom_parms *parms)
-{
-    nodeid_t node;
-    nodemask_t nodes = { 0 };
-    struct domain *d = bd->d;
-    unsigned long avail = 0, iommu_pages = 0;
-
-    nodes_or(nodes, nodes, node_online_map);
-
-    /* If building dom0 or hwdom, apply command line restriction. */
-    if ( has_dom0_caps(bd) )
-        nodes_and(nodes, nodes, dom0_nodes);
-
-    ASSERT(nodes_weight(nodes) != 0);
-
-    for_each_node_mask ( node, nodes )
-        avail += avail_domheap_pages_region(node, 0, 0) +
-                 initial_images_nrpages(node);
-
-    /* Reserve memory for further dom0 vcpu-struct allocations... */
-    avail -= (d->max_vcpus - 1UL)
-             << get_order_from_bytes(sizeof(struct vcpu));
-    /* ...and compat_l4's, if needed. */
-    if ( is_pv_32bit_domain(d) )
-        avail -= d->max_vcpus - 1;
-
-    /* Reserve memory for iommu_dom0_init() (rough estimate). */
-    if ( is_hardware_domain(d) && is_iommu_enabled(d)
-         && !iommu_hwdom_passthrough )
-    {
-        unsigned int s;
-
-        for ( s = 9; s < BITS_PER_LONG; s += 9 )
-            iommu_pages += max_pdx >> s;
-
-        avail -= iommu_pages;
-    }
-
-    if ( paging_mode_enabled(d) || opt_dom0_shadow || opt_pv_l1tf_hwdom )
-    {
-        unsigned long cpu_pages;
-
-        /*
-         * Clamp according to min/max limits and available memory
-         * (preliminary).
-         */
-        calculate_dom0_pages(bd, avail);
-
-        cpu_pages = dom_paging_pages(bd, bd->mem_pages);
-
-        if ( !iommu_use_hap_pt(d) )
-            avail -= cpu_pages;
-        else if ( cpu_pages > iommu_pages )
-            avail -= cpu_pages - iommu_pages;
-    }
-
-    /* Clamp according to min/max limits and available memory (final). */
-    calculate_dom0_pages(bd, avail);
-
-    if ( is_pv_domain(d) )
-        dom0_pv_restrict_pages(bd, parms);
-
-    d->max_pages = min_t(unsigned long, bd->max_pages, UINT_MAX);
-
-    return bd->mem_pages;
-}
-
 static void __init process_dom0_ioports_disable(struct domain *dom0)
 {
     unsigned long io_from, io_to;
diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
index a2e5807b60a5..258f777cd9ee 100644
--- a/xen/arch/x86/domain-builder/domain.c
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -6,6 +6,9 @@
 #include <xen/cpumask.h>
 #include <xen/domain.h>
 #include <xen/init.h>
+#include <xen/lib.h> /* get types.h for libefl.h */
+#include <xen/libelf.h>
+#include <xen/nodemask.h>
 #include <xen/sched.h>
 
 #include <public/bootfdt.h>
@@ -60,6 +63,74 @@ unsigned long __init dom_paging_pages(
     return DIV_ROUND_UP(memkb, 1024) << (20 - PAGE_SHIFT);
 }
 
+unsigned long __init dom_compute_nr_pages(
+    struct boot_domain *bd, struct elf_dom_parms *parms)
+{
+    nodeid_t node;
+    nodemask_t nodes = { 0 };
+    struct domain *d = bd->d;
+    unsigned long avail = 0, iommu_pages = 0;
+
+    nodes_or(nodes, nodes, node_online_map);
+
+    /* If building dom0 or hwdom, apply command line restriction. */
+    if ( has_dom0_caps(bd) )
+        nodes_and(nodes, nodes, dom0_nodes);
+
+    ASSERT(nodes_weight(nodes) != 0);
+
+    for_each_node_mask ( node, nodes )
+        avail += avail_domheap_pages_region(node, 0, 0) +
+                 initial_images_nrpages(node);
+
+    /* Reserve memory for further dom0 vcpu-struct allocations... */
+    avail -= (d->max_vcpus - 1UL)
+             << get_order_from_bytes(sizeof(struct vcpu));
+    /* ...and compat_l4's, if needed. */
+    if ( is_pv_32bit_domain(d) )
+        avail -= d->max_vcpus - 1;
+
+    /* Reserve memory for iommu_dom0_init() (rough estimate). */
+    if ( is_hardware_domain(d) && is_iommu_enabled(d)
+         && !iommu_hwdom_passthrough )
+    {
+        unsigned int s;
+
+        for ( s = 9; s < BITS_PER_LONG; s += 9 )
+            iommu_pages += max_pdx >> s;
+
+        avail -= iommu_pages;
+    }
+
+    if ( paging_mode_enabled(d) || opt_dom0_shadow || opt_pv_l1tf_hwdom )
+    {
+        unsigned long cpu_pages;
+
+        /*
+         * Clamp according to min/max limits and available memory
+         * (preliminary).
+         */
+        calculate_dom0_pages(bd, avail);
+
+        cpu_pages = dom_paging_pages(bd, bd->mem_pages);
+
+        if ( !iommu_use_hap_pt(d) )
+            avail -= cpu_pages;
+        else if ( cpu_pages > iommu_pages )
+            avail -= cpu_pages - iommu_pages;
+    }
+
+    /* Clamp according to min/max limits and available memory (final). */
+    calculate_dom0_pages(bd, avail);
+
+    if ( is_pv_domain(d) )
+        dom0_pv_restrict_pages(bd, parms);
+
+    d->max_pages = min_t(unsigned long, bd->max_pages, UINT_MAX);
+
+    return bd->mem_pages;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 7275bcf9ba6b..43a402af15b7 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -7,13 +7,15 @@
 #include <asm/setup.h>
 
 extern unsigned int dom0_memflags;
+extern nodemask_t dom0_nodes;
 
 void dom0_set_affinity(struct domain *dom0);
 
 int dom0_setup_permissions(struct domain *d);
 
 struct boot_domain;
-unsigned long dom_compute_nr_pages(
+void calculate_dom0_pages(struct boot_domain *bd, unsigned long avail);
+void dom0_pv_restrict_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms);
 
 int dom0_construct_pv(struct boot_domain *bd);
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
index c5a71fae5ccb..243ca09c045a 100644
--- a/xen/arch/x86/include/asm/domain-builder.h
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -2,10 +2,14 @@
 #define __XEN_X86_DOMBUILDER_H__
 
 struct boot_domain;
+struct elf_dom_parms;
 
 unsigned long dom_paging_pages(
     const struct boot_domain *d, unsigned long nr_pages);
 
+unsigned long dom_compute_nr_pages(
+    struct boot_domain *bd, struct elf_dom_parms *parms);
+
 int dom_construct_pvh(struct boot_domain *bd);
 
 #endif
-- 
2.30.2

[RFCv2 13/38] x86/hyperlaunch: move pvh p2m init to domain builder

[Daniel P. Smith]

Generalize pvh_init_p2m() for use on domU and relocate under the domain
builder. To support moving the function, dom0_pvh_setup_e820() was exported.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c         | 23 +----------------------
 xen/arch/x86/hvm/dom_build.c          | 25 +++++++++++++++++++++++++
 xen/arch/x86/include/asm/dom0_build.h |  2 ++
 3 files changed, 28 insertions(+), 22 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 1e63e19589a1..378a1f29fb1b 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -323,7 +323,7 @@ static int __init pvh_setup_vmx_realmode_helpers(struct domain *d)
     return 0;
 }
 
-static __init void pvh_setup_e820(struct domain *d, unsigned long nr_pages)
+void __init dom0_pvh_setup_e820(struct domain *d, unsigned long nr_pages)
 {
     struct e820entry *entry, *entry_guest;
     unsigned int i;
@@ -399,20 +399,6 @@ static __init void pvh_setup_e820(struct domain *d, unsigned long nr_pages)
     ASSERT(cur_pages == nr_pages);
 }
 
-static void __init pvh_init_p2m(struct boot_domain *bd)
-{
-    unsigned long nr_pages = dom_compute_nr_pages(bd, NULL);
-    bool preempted;
-
-    pvh_setup_e820(bd->d, nr_pages);
-    do {
-        preempted = false;
-        paging_set_allocation(bd->d, dom_paging_pages(bd, nr_pages),
-                              &preempted);
-        process_pending_softirqs();
-    } while ( preempted );
-}
-
 static int __init pvh_populate_p2m(struct domain *d)
 {
     struct vcpu *v = d->vcpu[0];
@@ -1318,13 +1304,6 @@ int __init dom0_construct_pvh(struct boot_domain *bd)
     struct domain *d = bd->d;
     int rc;
 
-    /*
-     * Craft dom0 physical memory map and set the paging allocation. This must
-     * be done before the iommu initializion, since iommu initialization code
-     * will likely add mappings required by devices to the p2m (ie: RMRRs).
-     */
-    pvh_init_p2m(bd);
-
     iommu_hwdom_init(d);
 
     rc = pvh_populate_p2m(d);
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index a5f259b6352d..eec65e3e805e 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -10,13 +10,16 @@
 
 #include <xen/acpi.h>
 #include <xen/init.h>
+#include <xen/softirq.h>
 #include <xen/types.h>
 
 #include <acpi/actables.h>
 
 #include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
+#include <asm/domain-builder.h>
 #include <asm/hvm/io.h>
+#include <asm/paging.h>
 #include <asm/pci.h>
 
 static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
@@ -37,6 +40,20 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
     }
 }
 
+static void __init pvh_init_p2m(struct boot_domain *bd)
+{
+    unsigned long nr_pages = dom_compute_nr_pages(bd, NULL);
+    bool preempted;
+
+    dom0_pvh_setup_e820(bd->d, nr_pages);
+    do {
+        preempted = false;
+        paging_set_allocation(bd->d, dom_paging_pages(bd, nr_pages),
+                              &preempted);
+        process_pending_softirqs();
+    } while ( preempted );
+}
+
 int __init dom_construct_pvh(struct boot_domain *bd)
 {
     int rc;
@@ -67,6 +84,14 @@ int __init dom_construct_pvh(struct boot_domain *bd)
         }
     }
 
+    /*
+     * Craft domain physical memory map and set the paging allocation. This
+     * must be done before the iommu initializion, since iommu initialization
+     * code will likely add mappings required by devices to the p2m (ie:
+     * RMRRs).
+     */
+    pvh_init_p2m(bd);
+
     return dom0_construct_pvh(bd);
 }
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 43a402af15b7..e5debd5adf9f 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -18,6 +18,8 @@ void calculate_dom0_pages(struct boot_domain *bd, unsigned long avail);
 void dom0_pv_restrict_pages(
     struct boot_domain *bd, struct elf_dom_parms *parms);
 
+void dom0_pvh_setup_e820(struct domain *d, unsigned long nr_pages);
+
 int dom0_construct_pv(struct boot_domain *bd);
 int dom0_construct_pvh(struct boot_domain *bd);
 
-- 
2.30.2

[RFCv2 14/38] x86/hyperlaunch: move iommu init to domain builder

[Daniel P. Smith]

Move invocation of iommu_hwdom_init() to dom_construct_pvh() and guard it
with a hardware domain check.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c | 2 --
 xen/arch/x86/hvm/dom_build.c  | 4 ++++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 378a1f29fb1b..6990c1d3a882 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -1304,8 +1304,6 @@ int __init dom0_construct_pvh(struct boot_domain *bd)
     struct domain *d = bd->d;
     int rc;
 
-    iommu_hwdom_init(d);
-
     rc = pvh_populate_p2m(d);
     if ( rc )
     {
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index eec65e3e805e..450e77d190a2 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -9,6 +9,7 @@
  */
 
 #include <xen/acpi.h>
+#include <xen/iommu.h>
 #include <xen/init.h>
 #include <xen/softirq.h>
 #include <xen/types.h>
@@ -92,6 +93,9 @@ int __init dom_construct_pvh(struct boot_domain *bd)
      */
     pvh_init_p2m(bd);
 
+    if ( is_hardware_domain(bd->d) )
+        iommu_hwdom_init(bd->d);
+
     return dom0_construct_pvh(bd);
 }
 
-- 
2.30.2

[RFCv2 15/38] x86/boot: move and rename sched_setup_dom0_vcpus

[Daniel P. Smith]

Relocated the function sched_setup_dom0_vcpus(), which was protected by an
ifdef CONFIG_X86, from common/sched to the hyperlaunch domain builder. Rename
it to alloc_dom_vcpus() to better reflect the purpose of the function.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/domain-builder/domain.c | 10 ++++++++++
 xen/arch/x86/hvm/dom0_build.c        |  3 ++-
 xen/arch/x86/pv/dom0_build.c         |  3 ++-
 xen/common/sched/core.c              | 12 ------------
 xen/include/xen/domain-builder.h     |  2 ++
 xen/include/xen/sched.h              |  1 -
 6 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/xen/arch/x86/domain-builder/domain.c b/xen/arch/x86/domain-builder/domain.c
index 258f777cd9ee..d934b240229f 100644
--- a/xen/arch/x86/domain-builder/domain.c
+++ b/xen/arch/x86/domain-builder/domain.c
@@ -45,6 +45,16 @@ struct vcpu *__init domain_vcpu0_create(struct boot_domain *bd)
     return vcpu_create(bd->d, 0);
 }
 
+void __init domain_vcpus_create(struct domain *d)
+{
+    unsigned int i;
+
+    for ( i = 1; i < d->max_vcpus; i++ )
+        vcpu_create(d, i);
+
+    domain_update_node_affinity(d);
+}
+
 unsigned long __init dom_paging_pages(
     const struct boot_domain *bd, unsigned long nr_pages)
 {
diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 6990c1d3a882..a900138b0311 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -8,6 +8,7 @@
  */
 
 #include <xen/acpi.h>
+#include <xen/domain-builder.h>
 #include <xen/init.h>
 #include <xen/libelf.h>
 #include <xen/multiboot.h>
@@ -833,7 +834,7 @@ static int __init pvh_setup_cpus(struct domain *d, paddr_t entry,
         .cpu_regs.x86_32.tr_ar = 0x8b,
     };
 
-    sched_setup_dom0_vcpus(d);
+    domain_vcpus_create(d);
 
     rc = arch_set_info_hvm_guest(v, &cpu_ctx);
     if ( rc )
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index ad4d1cc3520c..195c0902d5a1 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -5,6 +5,7 @@
  */
 
 #include <xen/domain.h>
+#include <xen/domain-builder.h>
 #include <xen/domain_page.h>
 #include <xen/init.h>
 #include <xen/libelf.h>
@@ -827,7 +828,7 @@ static int __init dom0_construct(struct boot_domain *bd)
 
     printk("Dom%u has maximum %u VCPUs\n", d->domain_id, d->max_vcpus);
 
-    sched_setup_dom0_vcpus(d);
+    domain_vcpus_create(d);
 
     d->arch.paging.mode = 0;
 
diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c
index 9043414290a8..d679d766a4b6 100644
--- a/xen/common/sched/core.c
+++ b/xen/common/sched/core.c
@@ -3479,18 +3479,6 @@ void wait(void)
     schedule();
 }
 
-#ifdef CONFIG_X86
-void __init sched_setup_dom0_vcpus(struct domain *d)
-{
-    unsigned int i;
-
-    for ( i = 1; i < d->max_vcpus; i++ )
-        vcpu_create(d, i);
-
-    domain_update_node_affinity(d);
-}
-#endif
-
 #ifdef CONFIG_COMPAT
 #include "compat.c"
 #endif
diff --git a/xen/include/xen/domain-builder.h b/xen/include/xen/domain-builder.h
index d3390d368afb..79e4c50ddf85 100644
--- a/xen/include/xen/domain-builder.h
+++ b/xen/include/xen/domain-builder.h
@@ -6,6 +6,7 @@
 
 struct boot_info;
 struct boot_domain;
+struct domain;
 
 /* Return status of builder_init(). Shows which boot mechanism was detected */
 enum fdt_kind
@@ -37,5 +38,6 @@ int builder_get_cmdline(const struct boot_info *bi, int offset,
 
 unsigned int dom_max_vcpus(struct boot_domain *bd);
 struct vcpu *domain_vcpu0_create(struct boot_domain *bd);
+void domain_vcpus_create(struct domain *d);
 
 #endif /* __XEN_DOMAIN_BUILDER_H__ */
diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h
index 559d201e0c7e..4f184cd76206 100644
--- a/xen/include/xen/sched.h
+++ b/xen/include/xen/sched.h
@@ -1084,7 +1084,6 @@ static inline bool sched_has_urgent_vcpu(void)
 }
 
 void vcpu_set_periodic_timer(struct vcpu *v, s_time_t value);
-void sched_setup_dom0_vcpus(struct domain *d);
 int vcpu_temporary_affinity(struct vcpu *v, unsigned int cpu, uint8_t reason);
 int vcpu_set_hard_affinity(struct vcpu *v, const cpumask_t *affinity);
 int vcpu_affinity_domctl(struct domain *d, uint32_t cmd,
-- 
2.30.2

[RFCv2 16/38] x86/hyperlaunch: move pvh_setup_cpus to domain builder

[Daniel P. Smith]

The function pvh_setup_cpus() is a very general function that is usable by all
HVM domains, not just PVH. As such, renaming to hvm_setup_cpus during move.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c             | 45 +---------------------
 xen/arch/x86/hvm/dom_build.c              | 46 +++++++++++++++++++++++
 xen/arch/x86/include/asm/domain-builder.h |  2 +
 3 files changed, 49 insertions(+), 44 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index a900138b0311..7b9a2cccabce 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -808,49 +808,6 @@ static int __init pvh_load_kernel(
     return 0;
 }
 
-static int __init pvh_setup_cpus(struct domain *d, paddr_t entry,
-                                 paddr_t start_info)
-{
-    struct vcpu *v = d->vcpu[0];
-    int rc;
-    /*
-     * This sets the vCPU state according to the state described in
-     * docs/misc/pvh.pandoc.
-     */
-    vcpu_hvm_context_t cpu_ctx = {
-        .mode = VCPU_HVM_MODE_32B,
-        .cpu_regs.x86_32.ebx = start_info,
-        .cpu_regs.x86_32.eip = entry,
-        .cpu_regs.x86_32.cr0 = X86_CR0_PE | X86_CR0_ET,
-        .cpu_regs.x86_32.cs_limit = ~0u,
-        .cpu_regs.x86_32.ds_limit = ~0u,
-        .cpu_regs.x86_32.es_limit = ~0u,
-        .cpu_regs.x86_32.ss_limit = ~0u,
-        .cpu_regs.x86_32.tr_limit = 0x67,
-        .cpu_regs.x86_32.cs_ar = 0xc9b,
-        .cpu_regs.x86_32.ds_ar = 0xc93,
-        .cpu_regs.x86_32.es_ar = 0xc93,
-        .cpu_regs.x86_32.ss_ar = 0xc93,
-        .cpu_regs.x86_32.tr_ar = 0x8b,
-    };
-
-    domain_vcpus_create(d);
-
-    rc = arch_set_info_hvm_guest(v, &cpu_ctx);
-    if ( rc )
-    {
-        printk("Unable to setup Dom0 BSP context: %d\n", rc);
-        return rc;
-    }
-
-    update_domain_wallclock_time(d);
-
-    v->is_initialised = 1;
-    clear_bit(_VPF_down, &v->pause_flags);
-
-    return 0;
-}
-
 static int __init cf_check acpi_count_intr_ovr(
     struct acpi_subtable_header *header, const unsigned long end)
 {
@@ -1319,7 +1276,7 @@ int __init dom0_construct_pvh(struct boot_domain *bd)
         return rc;
     }
 
-    rc = pvh_setup_cpus(d, entry, start_info);
+    rc = hvm_setup_cpus(bd->d, entry, start_info);
     if ( rc )
     {
         printk("Failed to setup Dom0 CPUs: %d\n", rc);
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index 450e77d190a2..c182847147b0 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -9,6 +9,7 @@
  */
 
 #include <xen/acpi.h>
+#include <xen/domain-builder.h>
 #include <xen/iommu.h>
 #include <xen/init.h>
 #include <xen/softirq.h>
@@ -16,6 +17,8 @@
 
 #include <acpi/actables.h>
 
+#include <public/hvm/hvm_vcpu.h>
+
 #include <asm/bootinfo.h>
 #include <asm/dom0_build.h>
 #include <asm/domain-builder.h>
@@ -55,6 +58,49 @@ static void __init pvh_init_p2m(struct boot_domain *bd)
     } while ( preempted );
 }
 
+int __init hvm_setup_cpus(
+    struct domain *d, paddr_t entry, paddr_t start_info)
+{
+    struct vcpu *v = d->vcpu[0];
+    int rc;
+    /*
+     * This sets the vCPU state according to the state described in
+     * docs/misc/pvh.pandoc.
+     */
+    vcpu_hvm_context_t cpu_ctx = {
+        .mode = VCPU_HVM_MODE_32B,
+        .cpu_regs.x86_32.ebx = start_info,
+        .cpu_regs.x86_32.eip = entry,
+        .cpu_regs.x86_32.cr0 = X86_CR0_PE | X86_CR0_ET,
+        .cpu_regs.x86_32.cs_limit = ~0u,
+        .cpu_regs.x86_32.ds_limit = ~0u,
+        .cpu_regs.x86_32.es_limit = ~0u,
+        .cpu_regs.x86_32.ss_limit = ~0u,
+        .cpu_regs.x86_32.tr_limit = 0x67,
+        .cpu_regs.x86_32.cs_ar = 0xc9b,
+        .cpu_regs.x86_32.ds_ar = 0xc93,
+        .cpu_regs.x86_32.es_ar = 0xc93,
+        .cpu_regs.x86_32.ss_ar = 0xc93,
+        .cpu_regs.x86_32.tr_ar = 0x8b,
+    };
+
+    domain_vcpus_create(d);
+
+    rc = arch_set_info_hvm_guest(v, &cpu_ctx);
+    if ( rc )
+    {
+        printk("Unable to setup Dom0 BSP context: %d\n", rc);
+        return rc;
+    }
+
+    update_domain_wallclock_time(d);
+
+    v->is_initialised = 1;
+    clear_bit(_VPF_down, &v->pause_flags);
+
+    return 0;
+}
+
 int __init dom_construct_pvh(struct boot_domain *bd)
 {
     int rc;
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
index 243ca09c045a..a27413edb380 100644
--- a/xen/arch/x86/include/asm/domain-builder.h
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -4,6 +4,8 @@
 struct boot_domain;
 struct elf_dom_parms;
 
+int hvm_setup_cpus(struct domain *d, paddr_t entry, paddr_t start_info);
+
 unsigned long dom_paging_pages(
     const struct boot_domain *d, unsigned long nr_pages);
 
-- 
2.30.2

[RFCv2 17/38] x86/boot: rename pvh acpi setup function

[Daniel P. Smith]

The function pvh_setup_acpi() is dom0 specific, renaming it to
dom0_pvh_setup_acpi(). Now export the function so that it may be called by the
domain builder.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c         | 4 ++--
 xen/arch/x86/include/asm/dom0_build.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 7b9a2cccabce..9ea8acbb5a02 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -1125,7 +1125,7 @@ static int __init pvh_setup_acpi_xsdt(struct domain *d, paddr_t madt_addr,
     return rc;
 }
 
-static int __init pvh_setup_acpi(struct domain *d, paddr_t start_info)
+int __init dom0_pvh_setup_acpi(struct domain *d, paddr_t start_info)
 {
     unsigned long pfn, nr_pages;
     paddr_t madt_paddr, xsdt_paddr, rsdp_paddr;
@@ -1283,7 +1283,7 @@ int __init dom0_construct_pvh(struct boot_domain *bd)
         return rc;
     }
 
-    rc = pvh_setup_acpi(d, start_info);
+    rc = dom0_pvh_setup_acpi(bd->d, start_info);
     if ( rc )
     {
         printk("Failed to setup Dom0 ACPI tables: %d\n", rc);
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index e5debd5adf9f..36f563bd9d5b 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -20,6 +20,8 @@ void dom0_pv_restrict_pages(
 
 void dom0_pvh_setup_e820(struct domain *d, unsigned long nr_pages);
 
+int dom0_pvh_setup_acpi(struct domain *d, paddr_t start_info);
+
 int dom0_construct_pv(struct boot_domain *bd);
 int dom0_construct_pvh(struct boot_domain *bd);
 
-- 
2.30.2

[RFCv2 18/38] x86/hyperlaunch: add domu memory map construction

[Daniel P. Smith]

Introduce hvm_setup_e820() that will build the e820 memory map for a general
domU. To populate the ACPI entry, ACPI table size helpers are introduced. A
conditional is added to the domain builder to select between calling
hvm_setup_e820() and dom0_pvh_setup_e820() depending on if it is building dom0
or a domU.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom_build.c | 149 ++++++++++++++++++++++++++++++++++-
 1 file changed, 148 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index c182847147b0..e724d578c5d4 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -17,6 +17,7 @@
 
 #include <acpi/actables.h>
 
+#include <public/hvm/e820.h>
 #include <public/hvm/hvm_vcpu.h>
 
 #include <asm/bootinfo.h>
@@ -44,12 +45,158 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
     }
 }
 
+static unsigned long __init hvm_size_acpi_madt(struct domain *d)
+{
+    unsigned long size = sizeof(struct acpi_table_madt);
+
+    size += sizeof(struct acpi_madt_local_apic) * d->max_vcpus;
+
+    return size;
+}
+
+static unsigned long __init hvm_size_acpi_xsdt(struct domain *d)
+{
+    unsigned long size = sizeof(struct acpi_table_xsdt);
+    /* Only adding the MADT table to the XSDT. */
+    unsigned int num_tables = 1;
+
+    /*
+     * No need to add or subtract anything because struct acpi_table_xsdt
+     * includes one array slot already.
+     */
+    size += num_tables * sizeof(uint64_t);
+
+    return size;
+}
+
+static unsigned long __init hvm_size_acpi_region(struct domain *d)
+{
+    unsigned long size = sizeof(struct acpi_table_rsdp);
+
+    size += hvm_size_acpi_xsdt(d);
+    size += hvm_size_acpi_madt(d);
+
+    return ROUNDUP(size, PAGE_SIZE);
+}
+
+/* From xenguest lib */
+#define END_SPECIAL_REGION   0xff000U
+#define NR_SPECIAL_PAGES     8
+#define START_SPECIAL_REGION (END_SPECIAL_REGION - NR_SPECIAL_PAGES)
+
+#define SPECIALPAGE_PAGING   0
+#define SPECIALPAGE_ACCESS   1
+#define SPECIALPAGE_SHARING  2
+#define SPECIALPAGE_BUFIOREQ 3
+#define SPECIALPAGE_XENSTORE 4
+#define SPECIALPAGE_IOREQ    5
+#define SPECIALPAGE_IDENT_PT 6
+#define SPECIALPAGE_CONSOLE  7
+#define special_pfn(x)       (START_SPECIAL_REGION + (x))
+
+/*
+ * Allocation scheme, derived from xenlight/xenguest:
+ *
+ *                                  |  <4G MMIO Hole  |
+ * [ Low Mem ][ RDM Mem ][ >1M Mem ][ ACPI ][ Special ][ High Mem ]
+ *
+ */
+static void __init hvm_setup_e820(struct domain *d, unsigned long nr_pages)
+{
+    const uint32_t lowmem_reserved_base = 0x9e000;
+    const uint32_t rdm_base = 0xa0000, rdm_size = 0x60;
+    unsigned long low_pages, ext_pages, mmio_pages, acpi_pages, high_pages = 0;
+    unsigned long max_ext_pages = (HVM_BELOW_4G_MMIO_START - MB(1)) >> PAGE_SHIFT,
+                  page_count = 0;
+    unsigned nr = 0, e820_entries = 5;
+
+    /* low pages: below 1MB */
+    low_pages = lowmem_reserved_base >> PAGE_SHIFT;
+    if ( low_pages > nr_pages )
+        panic("Insufficient memory for HVM/PVH domain (%pd)\n", d);
+
+    acpi_pages = hvm_size_acpi_region(d) >> PAGE_SHIFT;
+    mmio_pages = acpi_pages + NR_SPECIAL_PAGES;
+
+    /* ext pages: from 1MB to mmio hole */
+    ext_pages = nr_pages - (low_pages + mmio_pages);
+    if ( ext_pages > max_ext_pages )
+        ext_pages = max_ext_pages;
+
+    /* high pages: above 4GB */
+    if ( nr_pages > (low_pages + mmio_pages + ext_pages) )
+        high_pages = nr_pages - (low_pages + mmio_pages + ext_pages);
+
+    /* If we should have a highmem range, add one more e820 entry */
+    if ( high_pages )
+        e820_entries++;
+
+    ASSERT(e820_entries < E820MAX);
+
+    d->arch.e820 = xzalloc_array(struct e820entry, e820_entries);
+    if ( !d->arch.e820 )
+        panic("Unable to allocate memory for boot domain e820 map\n");
+
+    /* usable: Low memory */
+    d->arch.e820[nr].addr = 0x000000;
+    d->arch.e820[nr].size = low_pages << PAGE_SHIFT;
+    d->arch.e820[nr].type = E820_RAM;
+    page_count += d->arch.e820[nr].size >> PAGE_SHIFT;
+    nr++;
+
+    /* reserved: lowmem reserved device memory */
+    d->arch.e820[nr].addr = rdm_base;
+    d->arch.e820[nr].size = rdm_size;
+    d->arch.e820[nr].type = E820_RESERVED;
+    nr++;
+
+    /* usable: extended memory from 1MB */
+    d->arch.e820[nr].addr = 0x100000;
+    d->arch.e820[nr].size = ext_pages << PAGE_SHIFT;
+    d->arch.e820[nr].type = E820_RAM;
+    page_count += d->arch.e820[nr].size >> PAGE_SHIFT;
+    nr++;
+
+    /* reserved: ACPI entry, ACPI_INFO_PHYSICAL_ADDRESS */
+    d->arch.e820[nr].addr = 0xFC000000;
+    d->arch.e820[nr].size = acpi_pages << PAGE_SHIFT;
+    d->arch.e820[nr].type = E820_ACPI;
+    page_count += d->arch.e820[nr].size >> PAGE_SHIFT;
+    nr++;
+
+    /* reserved: HVM special pages, X86_HVM_END_SPECIAL_REGION */
+    d->arch.e820[nr].addr = START_SPECIAL_REGION << PAGE_SHIFT;
+    d->arch.e820[nr].size = NR_SPECIAL_PAGES << PAGE_SHIFT;
+    d->arch.e820[nr].type = E820_RESERVED;
+    page_count += d->arch.e820[nr].size >> PAGE_SHIFT;
+    nr++;
+
+    /* usable: highmem */
+    if ( high_pages )
+    {
+        d->arch.e820[nr].addr = 0x100000000;
+        d->arch.e820[nr].size = high_pages << PAGE_SHIFT;
+        d->arch.e820[nr].type = E820_RAM;
+        page_count += d->arch.e820[nr].size >> PAGE_SHIFT;
+        nr++;
+    }
+
+    d->arch.nr_e820 = nr;
+
+    ASSERT(nr == e820_entries);
+    ASSERT(nr_pages == page_count);
+}
+
 static void __init pvh_init_p2m(struct boot_domain *bd)
 {
     unsigned long nr_pages = dom_compute_nr_pages(bd, NULL);
     bool preempted;
 
-    dom0_pvh_setup_e820(bd->d, nr_pages);
+    if ( bd->capabilities & DOMAIN_CAPS_HARDWARE )
+        dom0_pvh_setup_e820(bd->d, nr_pages);
+    else
+        hvm_setup_e820(bd->d, nr_pages);
+
     do {
         preempted = false;
         paging_set_allocation(bd->d, dom_paging_pages(bd, nr_pages),
-- 
2.30.2

[RFCv2 19/38] x86/hyperlaunch: move populating p2m under domain builder

[Daniel P. Smith]

Introduce hvm_populate_p2m() for populating domU p2m maps. Rename
pvh_populate_p2m() to dom0_pvh_populate_p2m() and export it. With these
adjustments, move the calls to populate the p2m maps under domain builder.

Signed-off-by: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 xen/arch/x86/hvm/dom0_build.c             | 14 ++------
 xen/arch/x86/hvm/dom_build.c              | 39 +++++++++++++++++++++++
 xen/arch/x86/include/asm/dom0_build.h     |  1 +
 xen/arch/x86/include/asm/domain-builder.h |  2 ++
 4 files changed, 45 insertions(+), 11 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 9ea8acbb5a02..36d2e7e13948 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -89,9 +89,8 @@ static int __init modify_identity_mmio(struct domain *d, unsigned long pfn,
 }
 
 /* Populate a HVM memory range using the biggest possible order. */
-static int __init pvh_populate_memory_range(struct domain *d,
-                                            unsigned long start,
-                                            unsigned long nr_pages)
+int __init pvh_populate_memory_range(
+    struct domain *d, unsigned long start, unsigned long nr_pages)
 {
     static const struct {
         unsigned long align;
@@ -400,7 +399,7 @@ void __init dom0_pvh_setup_e820(struct domain *d, unsigned long nr_pages)
     ASSERT(cur_pages == nr_pages);
 }
 
-static int __init pvh_populate_p2m(struct domain *d)
+int __init dom0_pvh_populate_p2m(struct domain *d)
 {
     struct vcpu *v = d->vcpu[0];
     unsigned int i;
@@ -1262,13 +1261,6 @@ int __init dom0_construct_pvh(struct boot_domain *bd)
     struct domain *d = bd->d;
     int rc;
 
-    rc = pvh_populate_p2m(d);
-    if ( rc )
-    {
-        printk("Failed to setup Dom0 physical memory map\n");
-        return rc;
-    }
-
     rc = pvh_load_kernel(bd, &entry, &start_info);
     if ( rc )
     {
diff --git a/xen/arch/x86/hvm/dom_build.c b/xen/arch/x86/hvm/dom_build.c
index e724d578c5d4..4bbd5d2b8e24 100644
--- a/xen/arch/x86/hvm/dom_build.c
+++ b/xen/arch/x86/hvm/dom_build.c
@@ -24,6 +24,8 @@
 #include <asm/dom0_build.h>
 #include <asm/domain-builder.h>
 #include <asm/hvm/io.h>
+#include <asm/hvm/support.h>
+#include <asm/p2m.h>
 #include <asm/paging.h>
 #include <asm/pci.h>
 
@@ -248,6 +250,33 @@ int __init hvm_setup_cpus(
     return 0;
 }
 
+static int __init hvm_populate_p2m(struct domain *d)
+{
+    unsigned int i;
+
+    /* Populate memory map. */
+    for ( i = 0; i < d->arch.nr_e820; i++ )
+    {
+        int rc;
+        unsigned long addr, size;
+
+        if ( d->arch.e820[i].type != E820_RAM &&
+             d->arch.e820[i].type != E820_ACPI &&
+             PFN_DOWN(d->arch.e820[i].addr) != START_SPECIAL_REGION )
+                continue;
+
+        addr = PFN_DOWN(d->arch.e820[i].addr);
+        size = PFN_DOWN(d->arch.e820[i].size);
+
+        rc = pvh_populate_memory_range(d, addr, size);
+        if ( rc )
+            return rc;
+
+    }
+
+    return 0;
+}
+
 int __init dom_construct_pvh(struct boot_domain *bd)
 {
     int rc;
@@ -289,6 +318,16 @@ int __init dom_construct_pvh(struct boot_domain *bd)
     if ( is_hardware_domain(bd->d) )
         iommu_hwdom_init(bd->d);
 
+    if ( is_control_domain(bd->d) || is_hardware_domain(bd->d) )
+        rc = dom0_pvh_populate_p2m(bd->d);
+    else
+        rc = hvm_populate_p2m(bd->d);
+    if ( rc )
+    {
+        printk("Failed to setup HVM/PVH %pd physical memory map\n", bd->d);
+        return rc;
+    }
+
     return dom0_construct_pvh(bd);
 }
 
diff --git a/xen/arch/x86/include/asm/dom0_build.h b/xen/arch/x86/include/asm/dom0_build.h
index 36f563bd9d5b..3819b3f4e7a4 100644
--- a/xen/arch/x86/include/asm/dom0_build.h
+++ b/xen/arch/x86/include/asm/dom0_build.h
@@ -21,6 +21,7 @@ void dom0_pv_restrict_pages(
 void dom0_pvh_setup_e820(struct domain *d, unsigned long nr_pages);
 
 int dom0_pvh_setup_acpi(struct domain *d, paddr_t start_info);
+int dom0_pvh_populate_p2m(struct domain *d);
 
 int dom0_construct_pv(struct boot_domain *bd);
 int dom0_construct_pvh(struct boot_domain *bd);
diff --git a/xen/arch/x86/include/asm/domain-builder.h b/xen/arch/x86/include/asm/domain-builder.h
index a27413edb380..2de73117cb7f 100644
--- a/xen/arch/x86/include/asm/domain-builder.h
+++ b/xen/arch/x86/include/asm/domain-builder.h
@@ -5,6 +5,8 @@ struct boot_domain;
 struct elf_dom_parms;
 
 int hvm_setup_cpus(struct domain *d, paddr_t entry, paddr_t start_info);
+int pvh_populate_memory_range(
+    struct domain *d, unsigned long start, unsigned long nr_pages);
 
 unsigned long dom_paging_pages(
     const struct boot_domain *d, unsigned long nr_pages);
-- 
2.30.2