[PATCH v2 0/2] x86/pvh: workaround missing MMIO regions in dom0 p2m

[PATCH v2 0/2] x86/pvh: workaround missing MMIO regions in dom0 p2m

[Roger Pau Monne]

Hello,

The aim of this series is to provide a workaround for better handling
missing MMIO regions in a PVH dom0 p2m.  Xen doesn't know the complete
host memory layout, as it's not able to parse any information from
dynamic ACPI tables.  Hence the p2m built for a PVH dom0 might be
missing some MMIO regions only described in ACPI.

Since a PVH dom0 has no way to request the mapping of such regions (and
adding one would also require dom0 kernel modifications) instead provide
an option for Xen to add those MMIO regions as part of handling p2m
page-faults.  The option is currently off by default.

Thanks, Roger.

Roger Pau Monne (2):
  x86/emul: dump unhandled memory accesses for PVH dom0
  x86/dom0: attempt to fixup p2m page-faults for PVH dom0

 CHANGELOG.md                           | 10 ++++
 docs/misc/xen-command-line.pandoc      | 16 +++++-
 xen/arch/x86/dom0_build.c              |  5 ++
 xen/arch/x86/hvm/emulate.c             | 73 ++++++++++++++++++++++++++
 xen/arch/x86/include/asm/hvm/emulate.h |  3 ++
 5 files changed, 106 insertions(+), 1 deletion(-)

-- 
2.46.0

[PATCH v2 1/2] x86/emul: dump unhandled memory accesses for PVH dom0

[Roger Pau Monne]

A PV dom0 can map any host memory as long as it's allowed by the IO
capability range in d->iomem_caps.  On the other hand, a PVH dom0 has no
way to populate MMIO region onto it's p2m, so it's limited to what Xen
initially populates on the p2m based on the host memory map and the enabled
device BARs.

Introduce a new debug build only printk that reports attempts by dom0 to
access addresses not populated on the p2m, and not handled by any emulator.
This is for information purposes only, but might allow getting an idea of
what MMIO ranges might be missing on the p2m.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
---
 xen/arch/x86/hvm/emulate.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index 0d90cc4598be..08b9493e6d88 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -337,6 +337,9 @@ static int hvmemul_do_io(
         /* If there is no suitable backing DM, just ignore accesses */
         if ( !s )
         {
+            if ( is_mmio && is_hardware_domain(currd) )
+                gdprintk(XENLOG_DEBUG, "unhandled memory %s %#lx size %u\n",
+                         dir ? "read from" : "write to", addr, size);
             rc = hvm_process_io_intercept(&null_handler, &p);
             vio->req.state = STATE_IOREQ_NONE;
         }
-- 
2.46.0

[PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Roger Pau Monne]

When building a PVH dom0 Xen attempts to map all (relevant) MMIO regions
into the p2m for dom0 access.  However the information Xen has about the
host memory map is limited.  Xen doesn't have access to any resources
described in ACPI dynamic tables, and hence the p2m mappings provided might
not be complete.

PV doesn't suffer from this issue because a PV dom0 is capable of mapping
into it's page-tables any address not explicitly banned in d->iomem_caps.

Introduce a new command line options that allows Xen to attempt to fixup
the p2m page-faults, by creating p2m identity maps in response to p2m
page-faults.

This is aimed as a workaround to small ACPI regions Xen doesn't know about.
Note that missing large MMIO regions mapped in this way will lead to
slowness due to the VM exit processing, plus the mappings will always use
small pages.

The ultimate aim is to attempt to bring better parity with a classic PV
dom0.

Note such fixup rely on the CPU doing the access to the unpopulated
address.  If the access is attempted from a device instead there's no
possible way to fixup, as IOMMU page-fault are asynchronous.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Only slightly tested on my local PVH dom0 deployment.
---
Changes since v1:
 - Make the fixup function static.
 - Print message in case mapping already exists.
---
 CHANGELOG.md                           | 10 ++++
 docs/misc/xen-command-line.pandoc      | 16 +++++-
 xen/arch/x86/dom0_build.c              |  5 ++
 xen/arch/x86/hvm/emulate.c             | 74 +++++++++++++++++++++++++-
 xen/arch/x86/include/asm/hvm/emulate.h |  3 ++
 5 files changed, 105 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1de1d1eca17f..e5e6ab3a8902 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,16 @@ Notable changes to Xen will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
+## [4.21.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
+
+### Changed
+
+### Added
+ - On x86:
+   - Option to attempt to fixup p2m page-faults on PVH dom0.
+
+### Removed
+
 ## [4.20.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
 
 ### Changed
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 9bbd00baef91..83bb69cfb852 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -822,7 +822,8 @@ Specify the bit width of the DMA heap.
 
 ### dom0
     = List of [ pv | pvh, shadow=<bool>, verbose=<bool>,
-                cpuid-faulting=<bool>, msr-relaxed=<bool> ] (x86)
+                cpuid-faulting=<bool>, msr-relaxed=<bool>,
+                pf-fixup=<bool> ] (x86)
 
     = List of [ sve=<integer> ] (Arm64)
 
@@ -883,6 +884,19 @@ Controls for how dom0 is constructed on x86 systems.
 
     If using this option is necessary to fix an issue, please report a bug.
 
+*   The `pf-fixup` boolean is only applicable when using a PVH dom0 and
+    defaults to false.
+
+    When running dom0 in PVH mode the dom0 kernel has no way to map MMIO
+    regions into its physical memory map, such mode relies on Xen dom0 builder
+    populating the physical memory map with all MMIO regions that dom0 should
+    access.  However Xen doesn't have a complete picture of the host memory
+    map, due to not being able to process ACPI dynamic tables.
+
+    The `pf-fixup` option allows Xen to attempt to add missing MMIO regions
+    to the dom0 physical memory map in response to page-faults generated by
+    dom0 trying to access unpopulated entries in the memory map.
+
 Enables features on dom0 on Arm systems.
 
 *   The `sve` integer parameter enables Arm SVE usage for Dom0 and sets the
diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 2fe6b449149e..11c20b9ab0a4 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -16,6 +16,7 @@
 #include <asm/dom0_build.h>
 #include <asm/guest.h>
 #include <asm/hpet.h>
+#include <asm/hvm/emulate.h>
 #include <asm/io-ports.h>
 #include <asm/io_apic.h>
 #include <asm/p2m.h>
@@ -286,6 +287,10 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
         opt_dom0_cpuid_faulting = val;
     else if ( (val = parse_boolean("msr-relaxed", s, e)) >= 0 )
         opt_dom0_msr_relaxed = val;
+#ifdef CONFIG_HVM
+    else if ( (val = parse_boolean("pf-fixup", s, e)) >= 0 )
+        opt_dom0_pf_fixup = val;
+#endif
     else
         return -EINVAL;
 
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index 08b9493e6d88..3cd7f2e22f26 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -10,12 +10,15 @@
  */
 
 #include <xen/init.h>
+#include <xen/iocap.h>
 #include <xen/ioreq.h>
 #include <xen/lib.h>
 #include <xen/sched.h>
 #include <xen/paging.h>
 #include <xen/trace.h>
 #include <xen/vm_event.h>
+
+#include <asm/altp2m.h>
 #include <asm/event.h>
 #include <asm/i387.h>
 #include <asm/xstate.h>
@@ -161,6 +164,36 @@ void hvmemul_cancel(struct vcpu *v)
     hvmemul_cache_disable(v);
 }
 
+bool __ro_after_init opt_dom0_pf_fixup;
+static int hwdom_fixup_p2m(paddr_t addr)
+{
+    unsigned long gfn = paddr_to_pfn(addr);
+    struct domain *currd = current->domain;
+    p2m_type_t type;
+    mfn_t mfn;
+    int rc;
+
+    ASSERT(is_hardware_domain(currd));
+    ASSERT(!altp2m_active(currd));
+
+    /*
+     * Fixups are only applied for MMIO holes, and rely on the hardware domain
+     * having identity mappings for non RAM regions (gfn == mfn).
+     */
+    if ( !iomem_access_permitted(currd, gfn, gfn) ||
+         !is_memory_hole(_mfn(gfn), _mfn(gfn)) )
+        return -EPERM;
+
+    mfn = get_gfn(currd, gfn, &type);
+    if ( !mfn_eq(mfn, INVALID_MFN) || !p2m_is_hole(type) )
+        rc = mfn_eq(mfn, _mfn(gfn)) ? -EEXIST : -ENOTEMPTY;
+    else
+        rc = set_mmio_p2m_entry(currd, _gfn(gfn), _mfn(gfn), 0);
+    put_gfn(currd, gfn);
+
+    return rc;
+}
+
 static int hvmemul_do_io(
     bool is_mmio, paddr_t addr, unsigned long *reps, unsigned int size,
     uint8_t dir, bool df, bool data_is_addr, uintptr_t data)
@@ -338,8 +371,45 @@ static int hvmemul_do_io(
         if ( !s )
         {
             if ( is_mmio && is_hardware_domain(currd) )
-                gdprintk(XENLOG_DEBUG, "unhandled memory %s %#lx size %u\n",
-                         dir ? "read from" : "write to", addr, size);
+            {
+                /*
+                 * PVH dom0 is likely missing MMIO mappings on the p2m, due to
+                 * the incomplete information Xen has about the memory layout.
+                 *
+                 * Either print a message to note dom0 attempted to access an
+                 * unpopulated GPA, or try to fixup the p2m by creating an
+                 * identity mapping for the faulting GPA.
+                 */
+                if ( opt_dom0_pf_fixup )
+                {
+                    int inner_rc = hwdom_fixup_p2m(addr);
+
+                    if ( !inner_rc || inner_rc == -EEXIST )
+                    {
+                        if ( !inner_rc )
+                            gdprintk(XENLOG_DEBUG,
+                                     "fixup p2m mapping for page %lx added\n",
+                                     paddr_to_pfn(addr));
+                        else
+                            gprintk(XENLOG_INFO,
+                                    "fixup p2m mapping for page %lx already present\n",
+                                    paddr_to_pfn(addr));
+
+                        rc = X86EMUL_RETRY;
+                        vio->req.state = STATE_IOREQ_NONE;
+                        break;
+                    }
+
+                    gprintk(XENLOG_WARNING,
+                            "unable to fixup memory %s %#lx size %u: %d\n",
+                            dir ? "read from" : "write to", addr, size,
+                            inner_rc);
+                }
+                else
+                    gdprintk(XENLOG_DEBUG,
+                             "unhandled memory %s %#lx size %u\n",
+                             dir ? "read from" : "write to", addr, size);
+            }
             rc = hvm_process_io_intercept(&null_handler, &p);
             vio->req.state = STATE_IOREQ_NONE;
         }
diff --git a/xen/arch/x86/include/asm/hvm/emulate.h b/xen/arch/x86/include/asm/hvm/emulate.h
index 760ce5e77cce..d17c025a1d45 100644
--- a/xen/arch/x86/include/asm/hvm/emulate.h
+++ b/xen/arch/x86/include/asm/hvm/emulate.h
@@ -148,6 +148,9 @@ static inline void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
 void hvm_dump_emulation_state(const char *loglvl, const char *prefix,
                               struct hvm_emulate_ctxt *hvmemul_ctxt, int rc);
 
+/* For PVH dom0: signal whether to attempt fixup of p2m page-faults. */
+extern bool opt_dom0_pf_fixup;
+
 #endif /* __ASM_X86_HVM_EMULATE_H__ */
 
 /*
-- 
2.46.0

Re: [PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Roger Pau Monné]

On Tue, Feb 18, 2025 at 03:35:04PM +0100, Roger Pau Monne wrote:
> When building a PVH dom0 Xen attempts to map all (relevant) MMIO regions
> into the p2m for dom0 access.  However the information Xen has about the
> host memory map is limited.  Xen doesn't have access to any resources
> described in ACPI dynamic tables, and hence the p2m mappings provided might
> not be complete.
> 
> PV doesn't suffer from this issue because a PV dom0 is capable of mapping
> into it's page-tables any address not explicitly banned in d->iomem_caps.
> 
> Introduce a new command line options that allows Xen to attempt to fixup
> the p2m page-faults, by creating p2m identity maps in response to p2m
> page-faults.
> 
> This is aimed as a workaround to small ACPI regions Xen doesn't know about.
> Note that missing large MMIO regions mapped in this way will lead to
> slowness due to the VM exit processing, plus the mappings will always use
> small pages.
> 
> The ultimate aim is to attempt to bring better parity with a classic PV
> dom0.
> 
> Note such fixup rely on the CPU doing the access to the unpopulated
> address.  If the access is attempted from a device instead there's no
> possible way to fixup, as IOMMU page-fault are asynchronous.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Only slightly tested on my local PVH dom0 deployment.
> ---
> Changes since v1:
>  - Make the fixup function static.
>  - Print message in case mapping already exists.
> ---
>  CHANGELOG.md                           | 10 ++++
>  docs/misc/xen-command-line.pandoc      | 16 +++++-
>  xen/arch/x86/dom0_build.c              |  5 ++
>  xen/arch/x86/hvm/emulate.c             | 74 +++++++++++++++++++++++++-
>  xen/arch/x86/include/asm/hvm/emulate.h |  3 ++
>  5 files changed, 105 insertions(+), 3 deletions(-)
> 
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 1de1d1eca17f..e5e6ab3a8902 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -4,6 +4,16 @@ Notable changes to Xen will be documented in this file.
>  
>  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
>  
> +## [4.21.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
> +
> +### Changed
> +
> +### Added
> + - On x86:
> +   - Option to attempt to fixup p2m page-faults on PVH dom0.
> +
> +### Removed
> +
>  ## [4.20.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
>  
>  ### Changed
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index 9bbd00baef91..83bb69cfb852 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -822,7 +822,8 @@ Specify the bit width of the DMA heap.
>  
>  ### dom0
>      = List of [ pv | pvh, shadow=<bool>, verbose=<bool>,
> -                cpuid-faulting=<bool>, msr-relaxed=<bool> ] (x86)
> +                cpuid-faulting=<bool>, msr-relaxed=<bool>,
> +                pf-fixup=<bool> ] (x86)
>  
>      = List of [ sve=<integer> ] (Arm64)
>  
> @@ -883,6 +884,19 @@ Controls for how dom0 is constructed on x86 systems.
>  
>      If using this option is necessary to fix an issue, please report a bug.
>  
> +*   The `pf-fixup` boolean is only applicable when using a PVH dom0 and
> +    defaults to false.

I'm considering whether the default should instead be true, so that
PVH is closer to what which regions a classic PV dom0 gets access to.

Thanks, Roger.

Re: [PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Jan Beulich]

On 18.02.2025 15:35, Roger Pau Monne wrote:
> When building a PVH dom0 Xen attempts to map all (relevant) MMIO regions
> into the p2m for dom0 access.  However the information Xen has about the
> host memory map is limited.  Xen doesn't have access to any resources
> described in ACPI dynamic tables, and hence the p2m mappings provided might
> not be complete.
> 
> PV doesn't suffer from this issue because a PV dom0 is capable of mapping
> into it's page-tables any address not explicitly banned in d->iomem_caps.
> 
> Introduce a new command line options that allows Xen to attempt to fixup
> the p2m page-faults, by creating p2m identity maps in response to p2m
> page-faults.
> 
> This is aimed as a workaround to small ACPI regions Xen doesn't know about.
> Note that missing large MMIO regions mapped in this way will lead to
> slowness due to the VM exit processing, plus the mappings will always use
> small pages.
> 
> The ultimate aim is to attempt to bring better parity with a classic PV
> dom0.
> 
> Note such fixup rely on the CPU doing the access to the unpopulated
> address.  If the access is attempted from a device instead there's no
> possible way to fixup, as IOMMU page-fault are asynchronous.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>

Re: [PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Jan Beulich]

On 18.02.2025 16:44, Roger Pau Monné wrote:
> On Tue, Feb 18, 2025 at 03:35:04PM +0100, Roger Pau Monne wrote:
>> --- a/docs/misc/xen-command-line.pandoc
>> +++ b/docs/misc/xen-command-line.pandoc
>> @@ -822,7 +822,8 @@ Specify the bit width of the DMA heap.
>>  
>>  ### dom0
>>      = List of [ pv | pvh, shadow=<bool>, verbose=<bool>,
>> -                cpuid-faulting=<bool>, msr-relaxed=<bool> ] (x86)
>> +                cpuid-faulting=<bool>, msr-relaxed=<bool>,
>> +                pf-fixup=<bool> ] (x86)
>>  
>>      = List of [ sve=<integer> ] (Arm64)
>>  
>> @@ -883,6 +884,19 @@ Controls for how dom0 is constructed on x86 systems.
>>  
>>      If using this option is necessary to fix an issue, please report a bug.
>>  
>> +*   The `pf-fixup` boolean is only applicable when using a PVH dom0 and
>> +    defaults to false.
> 
> I'm considering whether the default should instead be true, so that
> PVH is closer to what which regions a classic PV dom0 gets access to.

I was wondering about this too, but then thought that we may want to do
this in a separate step, once we've had some more coverage.

Jan

Re: [PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Roger Pau Monné]

On Tue, Feb 18, 2025 at 04:58:20PM +0100, Jan Beulich wrote:
> On 18.02.2025 16:44, Roger Pau Monné wrote:
> > On Tue, Feb 18, 2025 at 03:35:04PM +0100, Roger Pau Monne wrote:
> >> --- a/docs/misc/xen-command-line.pandoc
> >> +++ b/docs/misc/xen-command-line.pandoc
> >> @@ -822,7 +822,8 @@ Specify the bit width of the DMA heap.
> >>  
> >>  ### dom0
> >>      = List of [ pv | pvh, shadow=<bool>, verbose=<bool>,
> >> -                cpuid-faulting=<bool>, msr-relaxed=<bool> ] (x86)
> >> +                cpuid-faulting=<bool>, msr-relaxed=<bool>,
> >> +                pf-fixup=<bool> ] (x86)
> >>  
> >>      = List of [ sve=<integer> ] (Arm64)
> >>  
> >> @@ -883,6 +884,19 @@ Controls for how dom0 is constructed on x86 systems.
> >>  
> >>      If using this option is necessary to fix an issue, please report a bug.
> >>  
> >> +*   The `pf-fixup` boolean is only applicable when using a PVH dom0 and
> >> +    defaults to false.
> > 
> > I'm considering whether the default should instead be true, so that
> > PVH is closer to what which regions a classic PV dom0 gets access to.
> 
> I was wondering about this too, but then thought that we may want to do
> this in a separate step, once we've had some more coverage.

Ack, let's put it in first and consider changing the default later.

Thanks, Roger.

Re: [PATCH v2 2/2] x86/dom0: attempt to fixup p2m page-faults for PVH dom0

[Oleksii Kurochko]

On 2/18/25 3:35 PM, Roger Pau Monne wrote:
> When building a PVH dom0 Xen attempts to map all (relevant) MMIO regions
> into the p2m for dom0 access.  However the information Xen has about the
> host memory map is limited.  Xen doesn't have access to any resources
> described in ACPI dynamic tables, and hence the p2m mappings provided might
> not be complete.
>
> PV doesn't suffer from this issue because a PV dom0 is capable of mapping
> into it's page-tables any address not explicitly banned in d->iomem_caps.
>
> Introduce a new command line options that allows Xen to attempt to fixup
> the p2m page-faults, by creating p2m identity maps in response to p2m
> page-faults.
>
> This is aimed as a workaround to small ACPI regions Xen doesn't know about.
> Note that missing large MMIO regions mapped in this way will lead to
> slowness due to the VM exit processing, plus the mappings will always use
> small pages.
>
> The ultimate aim is to attempt to bring better parity with a classic PV
> dom0.
>
> Note such fixup rely on the CPU doing the access to the unpopulated
> address.  If the access is attempted from a device instead there's no
> possible way to fixup, as IOMMU page-fault are asynchronous.
>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Only slightly tested on my local PVH dom0 deployment.
> ---
> Changes since v1:
>   - Make the fixup function static.
>   - Print message in case mapping already exists.
> ---
>   CHANGELOG.md                           | 10 ++++

Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>

~ Oleksii

>   docs/misc/xen-command-line.pandoc      | 16 +++++-
>   xen/arch/x86/dom0_build.c              |  5 ++
>   xen/arch/x86/hvm/emulate.c             | 74 +++++++++++++++++++++++++-
>   xen/arch/x86/include/asm/hvm/emulate.h |  3 ++
>   5 files changed, 105 insertions(+), 3 deletions(-)
>
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 1de1d1eca17f..e5e6ab3a8902 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -4,6 +4,16 @@ Notable changes to Xen will be documented in this file.
>   
>   The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
>   
> +## [4.21.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
> +
> +### Changed
> +
> +### Added
> + - On x86:
> +   - Option to attempt to fixup p2m page-faults on PVH dom0.
> +
> +### Removed
> +
>   ## [4.20.0 UNRELEASED](https://xenbits.xenproject.org/gitweb/?p=xen.git;a=shortlog;h=staging) - TBD
>   
>   ### Changed
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index 9bbd00baef91..83bb69cfb852 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -822,7 +822,8 @@ Specify the bit width of the DMA heap.
>   
>   ### dom0
>       = List of [ pv | pvh, shadow=<bool>, verbose=<bool>,
> -                cpuid-faulting=<bool>, msr-relaxed=<bool> ] (x86)
> +                cpuid-faulting=<bool>, msr-relaxed=<bool>,
> +                pf-fixup=<bool> ] (x86)
>   
>       = List of [ sve=<integer> ] (Arm64)
>   
> @@ -883,6 +884,19 @@ Controls for how dom0 is constructed on x86 systems.
>   
>       If using this option is necessary to fix an issue, please report a bug.
>   
> +*   The `pf-fixup` boolean is only applicable when using a PVH dom0 and
> +    defaults to false.
> +
> +    When running dom0 in PVH mode the dom0 kernel has no way to map MMIO
> +    regions into its physical memory map, such mode relies on Xen dom0 builder
> +    populating the physical memory map with all MMIO regions that dom0 should
> +    access.  However Xen doesn't have a complete picture of the host memory
> +    map, due to not being able to process ACPI dynamic tables.
> +
> +    The `pf-fixup` option allows Xen to attempt to add missing MMIO regions
> +    to the dom0 physical memory map in response to page-faults generated by
> +    dom0 trying to access unpopulated entries in the memory map.
> +
>   Enables features on dom0 on Arm systems.
>   
>   *   The `sve` integer parameter enables Arm SVE usage for Dom0 and sets the
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index 2fe6b449149e..11c20b9ab0a4 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -16,6 +16,7 @@
>   #include <asm/dom0_build.h>
>   #include <asm/guest.h>
>   #include <asm/hpet.h>
> +#include <asm/hvm/emulate.h>
>   #include <asm/io-ports.h>
>   #include <asm/io_apic.h>
>   #include <asm/p2m.h>
> @@ -286,6 +287,10 @@ int __init parse_arch_dom0_param(const char *s, const char *e)
>           opt_dom0_cpuid_faulting = val;
>       else if ( (val = parse_boolean("msr-relaxed", s, e)) >= 0 )
>           opt_dom0_msr_relaxed = val;
> +#ifdef CONFIG_HVM
> +    else if ( (val = parse_boolean("pf-fixup", s, e)) >= 0 )
> +        opt_dom0_pf_fixup = val;
> +#endif
>       else
>           return -EINVAL;
>   
> diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
> index 08b9493e6d88..3cd7f2e22f26 100644
> --- a/xen/arch/x86/hvm/emulate.c
> +++ b/xen/arch/x86/hvm/emulate.c
> @@ -10,12 +10,15 @@
>    */
>   
>   #include <xen/init.h>
> +#include <xen/iocap.h>
>   #include <xen/ioreq.h>
>   #include <xen/lib.h>
>   #include <xen/sched.h>
>   #include <xen/paging.h>
>   #include <xen/trace.h>
>   #include <xen/vm_event.h>
> +
> +#include <asm/altp2m.h>
>   #include <asm/event.h>
>   #include <asm/i387.h>
>   #include <asm/xstate.h>
> @@ -161,6 +164,36 @@ void hvmemul_cancel(struct vcpu *v)
>       hvmemul_cache_disable(v);
>   }
>   
> +bool __ro_after_init opt_dom0_pf_fixup;
> +static int hwdom_fixup_p2m(paddr_t addr)
> +{
> +    unsigned long gfn = paddr_to_pfn(addr);
> +    struct domain *currd = current->domain;
> +    p2m_type_t type;
> +    mfn_t mfn;
> +    int rc;
> +
> +    ASSERT(is_hardware_domain(currd));
> +    ASSERT(!altp2m_active(currd));
> +
> +    /*
> +     * Fixups are only applied for MMIO holes, and rely on the hardware domain
> +     * having identity mappings for non RAM regions (gfn == mfn).
> +     */
> +    if ( !iomem_access_permitted(currd, gfn, gfn) ||
> +         !is_memory_hole(_mfn(gfn), _mfn(gfn)) )
> +        return -EPERM;
> +
> +    mfn = get_gfn(currd, gfn, &type);
> +    if ( !mfn_eq(mfn, INVALID_MFN) || !p2m_is_hole(type) )
> +        rc = mfn_eq(mfn, _mfn(gfn)) ? -EEXIST : -ENOTEMPTY;
> +    else
> +        rc = set_mmio_p2m_entry(currd, _gfn(gfn), _mfn(gfn), 0);
> +    put_gfn(currd, gfn);
> +
> +    return rc;
> +}
> +
>   static int hvmemul_do_io(
>       bool is_mmio, paddr_t addr, unsigned long *reps, unsigned int size,
>       uint8_t dir, bool df, bool data_is_addr, uintptr_t data)
> @@ -338,8 +371,45 @@ static int hvmemul_do_io(
>           if ( !s )
>           {
>               if ( is_mmio && is_hardware_domain(currd) )
> -                gdprintk(XENLOG_DEBUG, "unhandled memory %s %#lx size %u\n",
> -                         dir ? "read from" : "write to", addr, size);
> +            {
> +                /*
> +                 * PVH dom0 is likely missing MMIO mappings on the p2m, due to
> +                 * the incomplete information Xen has about the memory layout.
> +                 *
> +                 * Either print a message to note dom0 attempted to access an
> +                 * unpopulated GPA, or try to fixup the p2m by creating an
> +                 * identity mapping for the faulting GPA.
> +                 */
> +                if ( opt_dom0_pf_fixup )
> +                {
> +                    int inner_rc = hwdom_fixup_p2m(addr);
> +
> +                    if ( !inner_rc || inner_rc == -EEXIST )
> +                    {
> +                        if ( !inner_rc )
> +                            gdprintk(XENLOG_DEBUG,
> +                                     "fixup p2m mapping for page %lx added\n",
> +                                     paddr_to_pfn(addr));
> +                        else
> +                            gprintk(XENLOG_INFO,
> +                                    "fixup p2m mapping for page %lx already present\n",
> +                                    paddr_to_pfn(addr));
> +
> +                        rc = X86EMUL_RETRY;
> +                        vio->req.state = STATE_IOREQ_NONE;
> +                        break;
> +                    }
> +
> +                    gprintk(XENLOG_WARNING,
> +                            "unable to fixup memory %s %#lx size %u: %d\n",
> +                            dir ? "read from" : "write to", addr, size,
> +                            inner_rc);
> +                }
> +                else
> +                    gdprintk(XENLOG_DEBUG,
> +                             "unhandled memory %s %#lx size %u\n",
> +                             dir ? "read from" : "write to", addr, size);
> +            }
>               rc = hvm_process_io_intercept(&null_handler, &p);
>               vio->req.state = STATE_IOREQ_NONE;
>           }
> diff --git a/xen/arch/x86/include/asm/hvm/emulate.h b/xen/arch/x86/include/asm/hvm/emulate.h
> index 760ce5e77cce..d17c025a1d45 100644
> --- a/xen/arch/x86/include/asm/hvm/emulate.h
> +++ b/xen/arch/x86/include/asm/hvm/emulate.h
> @@ -148,6 +148,9 @@ static inline void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
>   void hvm_dump_emulation_state(const char *loglvl, const char *prefix,
>                                 struct hvm_emulate_ctxt *hvmemul_ctxt, int rc);
>   
> +/* For PVH dom0: signal whether to attempt fixup of p2m page-faults. */
> +extern bool opt_dom0_pf_fixup;
> +
>   #endif /* __ASM_X86_HVM_EMULATE_H__ */
>   
>   /*