[PATCH for-4.19 v2 0/3] xen/x86: support foreign mappings for HVM/PVH

[PATCH for-4.19 v2 0/3] xen/x86: support foreign mappings for HVM/PVH

[Roger Pau Monne]

Hello,

The following series attempts to solve a shortcoming of HVM/PVH guests
with the lack of support for foreign mappings.  Such lack of support
prevents using PVH based guests as stubdomains for example.

Add support in a way similar to how it's done on Arm, by iterating over
the p2m based on the maximum gfn.

Patch 2 is not strictly needed.  Moving the enablement of altp2m from an
HVM param to a create domctl flag avoids any possible race with the HVM
param changing after it's been evaluated.  Note the param can only be
set by the control domain, and libxl currently sets it at domain
create.  Also altp2m enablement is different from activation, as
activation does happen during runtime of the domain.

Thanks, Roger.

Roger Pau Monne (3):
  xen/x86: account number of foreign mappings in the p2m
  xen/x86: enable altp2m at create domain domctl
  xen/x86: remove foreign mappings from the p2m on teardown

 CHANGELOG.md                      |  1 +
 tools/libs/light/libxl_x86.c      | 43 ++++++++++++-------
 xen/arch/x86/domain.c             | 31 +++++++++++++-
 xen/arch/x86/hvm/hvm.c            | 15 ++++++-
 xen/arch/x86/include/asm/p2m.h    | 32 +++++++++------
 xen/arch/x86/mm/p2m-basic.c       | 17 ++++++++
 xen/arch/x86/mm/p2m.c             | 68 +++++++++++++++++++++++++++++--
 xen/include/public/arch-x86/xen.h | 18 ++++++++
 xen/include/public/hvm/params.h   |  9 +---
 9 files changed, 195 insertions(+), 39 deletions(-)

-- 
2.44.0

[PATCH for-4.19 v2 1/3] xen/x86: account number of foreign mappings in the p2m

[Roger Pau Monne]

Such information will be needed in order to remove foreign mappings during
teardown for HVM guests.

Right now the introduced counter is not consumed.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - Drop max_gfn accounting.
---
 xen/arch/x86/include/asm/p2m.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/xen/arch/x86/include/asm/p2m.h b/xen/arch/x86/include/asm/p2m.h
index 111badf89a6e..107b9f260848 100644
--- a/xen/arch/x86/include/asm/p2m.h
+++ b/xen/arch/x86/include/asm/p2m.h
@@ -380,6 +380,9 @@ struct p2m_domain {
         unsigned int flags;
         unsigned long entry_count;
     } ioreq;
+
+    /* Number of foreign mappings. */
+    unsigned long      nr_foreign;
 #endif /* CONFIG_HVM */
 };
 
@@ -1049,6 +1052,8 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
         if ( !page_get_owner_and_reference(mfn_to_page(nfn)) )
             return -EBUSY;
 
+        p2m->nr_foreign++;
+
         break;
 
     default:
@@ -1069,6 +1074,7 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
             return -EINVAL;
         }
         put_page(mfn_to_page(ofn));
+        p2m->nr_foreign--;
         break;
 
     default:
-- 
2.44.0

[PATCH for-4.19 v2 2/3] xen/x86: enable altp2m at create domain domctl

[Roger Pau Monne]

Enabling it using an HVM param is fragile, and complicates the logic when
deciding whether options that interact with altp2m can also be enabled.

Leave the HVM param value for consumption by the guest, but prevent it from
being set.  Enabling is now done using the misc_flags field in
xen_arch_domainconfig.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - New in this version.
---
 tools/libs/light/libxl_x86.c      | 43 +++++++++++++++++++++----------
 xen/arch/x86/domain.c             | 25 +++++++++++++++++-
 xen/arch/x86/hvm/hvm.c            | 15 ++++++++++-
 xen/include/public/arch-x86/xen.h | 18 +++++++++++++
 xen/include/public/hvm/params.h   |  9 ++-----
 5 files changed, 87 insertions(+), 23 deletions(-)

diff --git a/tools/libs/light/libxl_x86.c b/tools/libs/light/libxl_x86.c
index a50ec37eb3eb..86ee8132536c 100644
--- a/tools/libs/light/libxl_x86.c
+++ b/tools/libs/light/libxl_x86.c
@@ -6,8 +6,21 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
                                       libxl_domain_config *d_config,
                                       struct xen_domctl_createdomain *config)
 {
+    unsigned int altp2m = d_config->b_info.altp2m;
+
     switch(d_config->c_info.type) {
     case LIBXL_DOMAIN_TYPE_HVM:
+        /*
+         * The config parameter "altp2m" replaces the parameter "altp2mhvm".
+         * For legacy reasons, both parameters are accepted on x86 HVM guests.
+         *
+         * If the legacy field info->u.hvm.altp2m is set, activate altp2m.
+         * Otherwise set altp2m based on the field info->altp2m.
+         */
+        if (altp2m == LIBXL_ALTP2M_MODE_DISABLED &&
+            libxl_defbool_val(d_config->b_info.u.hvm.altp2m))
+            altp2m = libxl_defbool_val(d_config->b_info.u.hvm.altp2m);
+
         config->arch.emulation_flags = (XEN_X86_EMU_ALL & ~XEN_X86_EMU_VPCI);
         if (!libxl_defbool_val(d_config->b_info.u.hvm.pirq))
             config->arch.emulation_flags &= ~XEN_X86_EMU_USE_PIRQ;
@@ -26,6 +39,22 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
     if (libxl_defbool_val(d_config->b_info.arch_x86.msr_relaxed))
         config->arch.misc_flags |= XEN_X86_MSR_RELAXED;
 
+    if (d_config->c_info.type != LIBXL_DOMAIN_TYPE_PV) {
+        switch (altp2m) {
+        case LIBXL_ALTP2M_MODE_MIXED:
+            config->arch.misc_flags |= XEN_X86_ALTP2M_MIXED;
+            break;
+
+        case LIBXL_ALTP2M_MODE_EXTERNAL:
+            config->arch.misc_flags |= XEN_X86_ALTP2M_EXT;
+            break;
+
+        case LIBXL_ALTP2M_MODE_LIMITED:
+            config->arch.misc_flags |= XEN_X86_ALTP2M_LIMITED;
+            break;
+        }
+    }
+
     return 0;
 }
 
@@ -407,19 +436,9 @@ static int hvm_set_conf_params(libxl__gc *gc, uint32_t domid,
     libxl_ctx *ctx = libxl__gc_owner(gc);
     xc_interface *xch = ctx->xch;
     int ret = ERROR_FAIL;
-    unsigned int altp2m = info->altp2m;
 
     switch(info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
-        /* The config parameter "altp2m" replaces the parameter "altp2mhvm". For
-         * legacy reasons, both parameters are accepted on x86 HVM guests.
-         *
-         * If the legacy field info->u.hvm.altp2m is set, activate altp2m.
-         * Otherwise set altp2m based on the field info->altp2m. */
-        if (info->altp2m == LIBXL_ALTP2M_MODE_DISABLED &&
-            libxl_defbool_val(info->u.hvm.altp2m))
-            altp2m = libxl_defbool_val(info->u.hvm.altp2m);
-
         if (xc_hvm_param_set(xch, domid, HVM_PARAM_HPET_ENABLED,
                              libxl_defbool_val(info->u.hvm.hpet))) {
             LOG(ERROR, "Couldn't set HVM_PARAM_HPET_ENABLED");
@@ -444,10 +463,6 @@ static int hvm_set_conf_params(libxl__gc *gc, uint32_t domid,
             LOG(ERROR, "Couldn't set HVM_PARAM_TIMER_MODE");
             goto out;
         }
-        if (xc_hvm_param_set(xch, domid, HVM_PARAM_ALTP2M, altp2m)) {
-            LOG(ERROR, "Couldn't set HVM_PARAM_ALTP2M");
-            goto out;
-        }
         break;
 
     default:
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 20e83cf38bbd..dff790060605 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -637,6 +637,9 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
     bool hap = config->flags & XEN_DOMCTL_CDF_hap;
     bool nested_virt = config->flags & XEN_DOMCTL_CDF_nested_virt;
     unsigned int max_vcpus;
+    unsigned int altp2m = config->arch.misc_flags & (XEN_X86_ALTP2M_MIXED |
+                                                     XEN_X86_ALTP2M_EXT |
+                                                     XEN_X86_ALTP2M_LIMITED);
 
     if ( hvm ? !hvm_enabled : !IS_ENABLED(CONFIG_PV) )
     {
@@ -708,13 +711,33 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         }
     }
 
-    if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
+    if ( config->arch.misc_flags & ~XEN_X86_MISC_FLAGS_ALL )
     {
         dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
                 config->arch.misc_flags);
         return -EINVAL;
     }
 
+    if ( altp2m && (altp2m & (altp2m - 1)) )
+    {
+        dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
+                config->flags);
+        return -EINVAL;
+    }
+
+    if ( altp2m && nested_virt )
+    {
+        dprintk(XENLOG_INFO,
+                "Nested virt and altp2m are mutually incompatible\n");
+        return -EINVAL;
+    }
+
+    if ( altp2m && !hap )
+    {
+        dprintk(XENLOG_INFO, "altp2m requires HAP\n");
+        return -EINVAL;
+    }
+
     return 0;
 }
 
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 0ce45b177cf4..f3e1b9364588 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -659,6 +659,14 @@ int hvm_domain_initialise(struct domain *d,
 
     d->arch.hvm.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
 
+    /* Set altp2m based on domctl flags. */
+    if ( config->arch.misc_flags & XEN_X86_ALTP2M_MIXED )
+        d->arch.hvm.params[HVM_PARAM_ALTP2M] = XEN_ALTP2M_mixed;
+    else if ( config->arch.misc_flags & XEN_X86_ALTP2M_EXT )
+        d->arch.hvm.params[HVM_PARAM_ALTP2M] = XEN_ALTP2M_external;
+    else if ( config->arch.misc_flags & XEN_X86_ALTP2M_LIMITED )
+        d->arch.hvm.params[HVM_PARAM_ALTP2M] = XEN_ALTP2M_limited;
+
     vpic_init(d);
 
     rc = vioapic_init(d);
@@ -4163,6 +4171,12 @@ static int hvm_allow_set_param(struct domain *d,
     case HVM_PARAM_CONSOLE_EVTCHN:
     case HVM_PARAM_X87_FIP_WIDTH:
         break;
+
+    /* The following parameters are read-only. */
+    case HVM_PARAM_ALTP2M:
+        rc = -EEXIST;
+        break;
+
     /* The following parameters are deprecated. */
     case HVM_PARAM_PAE_ENABLED:
     case HVM_PARAM_DM_DOMAIN:
@@ -4204,7 +4218,6 @@ static int hvm_allow_set_param(struct domain *d,
     case HVM_PARAM_BUFIOREQ_PFN:
     case HVM_PARAM_IOREQ_SERVER_PFN:
     case HVM_PARAM_NR_IOREQ_SERVER_PAGES:
-    case HVM_PARAM_ALTP2M:
     case HVM_PARAM_MCA_CAP:
         if ( value != 0 && new_value != value )
             rc = -EEXIST;
diff --git a/xen/include/public/arch-x86/xen.h b/xen/include/public/arch-x86/xen.h
index a9a87d9b50de..bc444e7744a5 100644
--- a/xen/include/public/arch-x86/xen.h
+++ b/xen/include/public/arch-x86/xen.h
@@ -300,6 +300,24 @@ struct xen_arch_domainconfig {
  * doesn't allow the guest to read or write to the underlying MSR.
  */
 #define XEN_X86_MSR_RELAXED (1u << 0)
+
+/* altp2m options.  Only one can be set.
+ *
+ * Note that 'mixed' mode has not been evaluated for safety from a
+ * security perspective.  Before using this mode in a
+ * security-critical environment, each subop should be evaluated for
+ * safety, with unsafe subops blacklisted in XSM.
+ *
+ * Enable altp2m mixed mode.
+ */
+#define XEN_X86_ALTP2M_MIXED   (1U << 1)
+/* Enable altp2m external mode. */
+#define XEN_X86_ALTP2M_EXT     (1U << 2)
+/* Enable altp2m limited mode. */
+#define XEN_X86_ALTP2M_LIMITED (1U << 3)
+
+#define XEN_X86_MISC_FLAGS_ALL (XEN_X86_MSR_RELAXED | XEN_X86_ALTP2M_MIXED | \
+                                XEN_X86_ALTP2M_EXT | XEN_X86_ALTP2M_LIMITED)
     uint32_t misc_flags;
 };
 
diff --git a/xen/include/public/hvm/params.h b/xen/include/public/hvm/params.h
index a22b4ed45d2e..99c40b4287f1 100644
--- a/xen/include/public/hvm/params.h
+++ b/xen/include/public/hvm/params.h
@@ -246,16 +246,11 @@
 #define HVM_PARAM_VM_GENERATION_ID_ADDR 34
 
 /*
- * Set mode for altp2m:
- *  disabled: don't activate altp2m (default)
+ * Get mode for altp2m:
+ *  disabled: altp2m not active (default)
  *  mixed: allow access to all altp2m ops for both in-guest and external tools
  *  external: allow access to external privileged tools only
  *  limited: guest only has limited access (ie. control VMFUNC and #VE)
- *
- * Note that 'mixed' mode has not been evaluated for safety from a
- * security perspective.  Before using this mode in a
- * security-critical environment, each subop should be evaluated for
- * safety, with unsafe subops blacklisted in XSM.
  */
 #define HVM_PARAM_ALTP2M       35
 #define XEN_ALTP2M_disabled      0
-- 
2.44.0

[PATCH for-4.19 v2 3/3] xen/x86: remove foreign mappings from the p2m on teardown

[Roger Pau Monne]

Iterate over the p2m up to the maximum recorded gfn and remove any foreign
mappings, in order to drop the underlying page references and thus don't keep
extra page references if a domain is destroyed while still having foreign
mappings on it's p2m.

The logic is similar to the one used on Arm.

Note that foreign mappings cannot be created by guests that have altp2m or
nested HVM enabled, as p2ms different than the host one are not currently
scrubbed when destroyed in order to drop references to any foreign maps.

It's unclear whether the right solution is to take an extra reference when
foreign maps are added to p2ms different than the host one, or just rely on the
host p2m already having a reference.  The mapping being removed from the host
p2m should cause it to be dropped on all domain p2ms.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - Use existing p2m max_mapped_pfn field.
 - Prevent creating foreign mappings by guests that have altp2m or nestedhvm
   enabled.
---
 CHANGELOG.md                   |  1 +
 xen/arch/x86/domain.c          |  8 +++-
 xen/arch/x86/include/asm/p2m.h | 26 +++++++------
 xen/arch/x86/mm/p2m-basic.c    | 17 +++++++++
 xen/arch/x86/mm/p2m.c          | 68 ++++++++++++++++++++++++++++++++--
 5 files changed, 103 insertions(+), 17 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8041cfb7d243..09bdb9b97578 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
    - HVM PIRQs are disabled by default.
    - Reduce IOMMU setup time for hardware domain.
  - xl/libxl configures vkb=[] for HVM domains with priority over vkb_device.
+ - Allow HVM/PVH domains to map foreign pages.
 
 ### Added
  - On x86:
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index dff790060605..1cb3ccddab00 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -718,7 +718,7 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
-    if ( altp2m && (altp2m & (altp2m - 1)) )
+    if ( altp2m & (altp2m - 1) )
     {
         dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
                 config->flags);
@@ -2387,6 +2387,7 @@ int domain_relinquish_resources(struct domain *d)
         enum {
             PROG_iommu_pagetables = 1,
             PROG_shared,
+            PROG_mappings,
             PROG_paging,
             PROG_vcpu_pagetables,
             PROG_xen,
@@ -2435,6 +2436,11 @@ int domain_relinquish_resources(struct domain *d)
         }
 #endif
 
+    PROGRESS(mappings):
+        ret = relinquish_p2m_mapping(d);
+        if ( ret )
+            return ret;
+
     PROGRESS(paging):
 
         /* Tear down paging-assistance stuff. */
diff --git a/xen/arch/x86/include/asm/p2m.h b/xen/arch/x86/include/asm/p2m.h
index 107b9f260848..c1478ffc3647 100644
--- a/xen/arch/x86/include/asm/p2m.h
+++ b/xen/arch/x86/include/asm/p2m.h
@@ -383,6 +383,8 @@ struct p2m_domain {
 
     /* Number of foreign mappings. */
     unsigned long      nr_foreign;
+    /* Cursor for iterating over the p2m on teardown. */
+    unsigned long      teardown_gfn;
 #endif /* CONFIG_HVM */
 };
 
@@ -395,16 +397,7 @@ struct p2m_domain {
 #endif
 #include <xen/p2m-common.h>
 
-static inline bool arch_acquire_resource_check(struct domain *d)
-{
-    /*
-     * FIXME: Until foreign pages inserted into the P2M are properly
-     * reference counted, it is unsafe to allow mapping of
-     * resource pages unless the caller is the hardware domain
-     * (see set_foreign_p2m_entry()).
-     */
-    return !paging_mode_translate(d) || is_hardware_domain(d);
-}
+bool arch_acquire_resource_check(const struct domain *d);
 
 /*
  * Updates vCPU's n2pm to match its np2m_base in VMCx12 and returns that np2m.
@@ -720,6 +713,10 @@ p2m_pod_offline_or_broken_hit(struct page_info *p);
 void
 p2m_pod_offline_or_broken_replace(struct page_info *p);
 
+/* Perform cleanup of p2m mappings ahead of teardown. */
+int
+relinquish_p2m_mapping(struct domain *d);
+
 #else
 
 static inline bool
@@ -748,6 +745,11 @@ static inline void p2m_pod_offline_or_broken_replace(struct page_info *p)
     ASSERT_UNREACHABLE();
 }
 
+static inline int relinquish_p2m_mapping(struct domain *d)
+{
+    return 0;
+}
+
 #endif
 
 
@@ -1043,7 +1045,7 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
         break;
 
     case p2m_map_foreign:
-        if ( !mfn_valid(nfn) )
+        if ( !mfn_valid(nfn) || p2m != p2m_get_hostp2m(p2m->domain) )
         {
             ASSERT_UNREACHABLE();
             return -EINVAL;
@@ -1068,7 +1070,7 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
         break;
 
     case p2m_map_foreign:
-        if ( !mfn_valid(ofn) )
+        if ( !mfn_valid(ofn) || p2m != p2m_get_hostp2m(p2m->domain) )
         {
             ASSERT_UNREACHABLE();
             return -EINVAL;
diff --git a/xen/arch/x86/mm/p2m-basic.c b/xen/arch/x86/mm/p2m-basic.c
index 8599bd15c61a..d374e1e824f1 100644
--- a/xen/arch/x86/mm/p2m-basic.c
+++ b/xen/arch/x86/mm/p2m-basic.c
@@ -13,6 +13,8 @@
 
 #include <xen/event.h>
 #include <xen/types.h>
+#include <asm/altp2m.h>
+#include <asm/hvm/nestedhvm.h>
 #include <asm/p2m.h>
 #include "mm-locks.h"
 #include "p2m.h"
@@ -207,6 +209,21 @@ void p2m_final_teardown(struct domain *d)
     p2m_teardown_hostp2m(d);
 }
 
+bool arch_acquire_resource_check(const struct domain *d)
+{
+    /*
+     * altp2m is not supported as we would otherwise also need to walk the
+     * altp2m tables and drop any foreign map entries in order to drop the page
+     * reference.
+     *
+     * The same applies to nestedhvm nested p2m tables, as the type from the L0
+     * p2m is replicated into the L1 p2m, and there's no filtering that
+     * prevents foreign mappings from being created in nestedp2m.
+     */
+    return d->arch.hvm.params[HVM_PARAM_ALTP2M] == XEN_ALTP2M_disabled &&
+           !nestedhvm_enabled(d);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/x86/mm/p2m.c b/xen/arch/x86/mm/p2m.c
index ce742c12e0de..23c6c42af275 100644
--- a/xen/arch/x86/mm/p2m.c
+++ b/xen/arch/x86/mm/p2m.c
@@ -2333,10 +2333,6 @@ static int p2m_add_foreign(struct domain *tdom, unsigned long fgfn,
     int rc;
     struct domain *fdom;
 
-    /*
-     * hvm fixme: until support is added to p2m teardown code to cleanup any
-     * foreign entries, limit this to hardware domain only.
-     */
     if ( !arch_acquire_resource_check(tdom) )
         return -EPERM;
 
@@ -2693,6 +2689,70 @@ int p2m_set_altp2m_view_visibility(struct domain *d, unsigned int altp2m_idx,
     return rc;
 }
 
+/*
+ * Remove foreign mappings from the p2m, as that drops the page reference taken
+ * when mapped.
+ */
+int relinquish_p2m_mapping(struct domain *d)
+{
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+    unsigned long gfn, count = 0;
+    int rc = 0;
+
+    if ( !paging_mode_translate(d) )
+        return 0;
+
+    BUG_ON(!d->is_dying);
+
+    p2m_lock(p2m);
+
+    gfn = p2m->teardown_gfn;
+
+    /* Iterate over the whole p2m on debug builds to ensure correctness. */
+    while ( gfn <= p2m->max_mapped_pfn &&
+            (IS_ENABLED(CONFIG_DEBUG) || p2m->nr_foreign) )
+    {
+        unsigned int order;
+        p2m_type_t t;
+        p2m_access_t a;
+
+        _get_gfn_type_access(p2m, _gfn(gfn), &t, &a, 0, &order, 0);
+        ASSERT(IS_ALIGNED(gfn, 1UL << order));
+
+        if ( t == p2m_map_foreign )
+        {
+            ASSERT(p2m->nr_foreign);
+            ASSERT(order == 0);
+
+            rc = p2m_set_entry(p2m, _gfn(gfn), INVALID_MFN, order, p2m_invalid,
+                               p2m->default_access);
+            if ( rc )
+            {
+                printk(XENLOG_ERR
+                       "%pd: failed to unmap foreign page %" PRI_gfn " order %u error %d\n",
+                       d, gfn, order, rc);
+                ASSERT_UNREACHABLE();
+                break;
+            }
+        }
+
+        gfn += 1UL << order;
+
+        if ( !(++count & 0xff) && hypercall_preempt_check() )
+        {
+            rc = -ERESTART;
+            break;
+        }
+    }
+
+    ASSERT(gfn <= p2m->max_mapped_pfn || !p2m->nr_foreign);
+    p2m->teardown_gfn = gfn;
+
+    p2m_unlock(p2m);
+
+    return rc;
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.44.0

Re: [PATCH for-4.19 v2 2/3] xen/x86: enable altp2m at create domain domctl

[Andrew Cooper]

On 08/05/2024 12:23 pm, Roger Pau Monne wrote:
> Enabling it using an HVM param is fragile, and complicates the logic when
> deciding whether options that interact with altp2m can also be enabled.
>
> Leave the HVM param value for consumption by the guest, but prevent it from
> being set.  Enabling is now done using the misc_flags field in
> xen_arch_domainconfig.
>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Changes since v1:
>  - New in this version.

Ha.  So this is actually work that Petr has been wanting to do.

Petr has a series hoping to make it into 4.19 (x86: Make MAX_ALTP2M
configurable), which just missed out on this side of things.

altp2m is not architecture specific at all, and there's even support for
ARM out on the mailing list.  Therefore, the altp2m mode wants to be
common, just like the new MAX_ALTP2M setting already is.

Both fields can reasonably share uint32_t, but could you work with Petr
to make both halfs of this land cleanly.


As to the HVMPARAM, I'd really quite like to delete it.  It was always a
bodge, and there's a full set of HVMOP_altp2m_* for a guest to use.

> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> index 20e83cf38bbd..dff790060605 100644
> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -708,13 +711,33 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
>          }
>      }
>  
> -    if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
> +    if ( config->arch.misc_flags & ~XEN_X86_MISC_FLAGS_ALL )
>      {
>          dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
>                  config->arch.misc_flags);
>          return -EINVAL;
>      }
>  
> +    if ( altp2m && (altp2m & (altp2m - 1)) )
> +    {
> +        dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
> +                config->flags);
> +        return -EINVAL;

I think this would be clearer to follow by having a 2 bit field called
altp2m_mode and check for <= 2.

> +    }
> +
> +    if ( altp2m && nested_virt )
> +    {
> +        dprintk(XENLOG_INFO,
> +                "Nested virt and altp2m are mutually incompatible\n");

There's nothing inherently incompatible.  I think it's more that noone
had any interest in trying to make it work in combination with nested p2ms.

I'd phrase it as "not supported", rather than incompatible.

> +        return -EINVAL;
> +    }
> +
> +    if ( altp2m && !hap )
> +    {
> +        dprintk(XENLOG_INFO, "altp2m requires HAP\n");
> +        return -EINVAL;
> +    }

altp2m ought to work fine with shadow.  It's only if you want VMFUNC/#VE
acceleration that you depend on EPT.

Again, I'd phrase this as "not supported".

~Andrew

Re: [PATCH for-4.19 v2 2/3] xen/x86: enable altp2m at create domain domctl

[Petr Beneš]

On Wed, May 8, 2024 at 9:38 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> Both fields can reasonably share uint32_t, but could you work with Petr
> to make both halfs of this land cleanly.

Hi, I think creating a new anonymous struct "altp2m" within `struct
domain` would be a good fit. uint16_t for my MAX_ALTP2M replacement
field will be enough, so the other uint16_t could be used for various
altp2m modes/flags.

P.

Re: [PATCH for-4.19 v2 2/3] xen/x86: enable altp2m at create domain domctl

[Roger Pau Monné]

On Wed, May 08, 2024 at 08:38:07PM +0100, Andrew Cooper wrote:
> On 08/05/2024 12:23 pm, Roger Pau Monne wrote:
> > Enabling it using an HVM param is fragile, and complicates the logic when
> > deciding whether options that interact with altp2m can also be enabled.
> >
> > Leave the HVM param value for consumption by the guest, but prevent it from
> > being set.  Enabling is now done using the misc_flags field in
> > xen_arch_domainconfig.
> >
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > ---
> > Changes since v1:
> >  - New in this version.
> 
> Ha.  So this is actually work that Petr has been wanting to do.
> 
> Petr has a series hoping to make it into 4.19 (x86: Make MAX_ALTP2M
> configurable), which just missed out on this side of things.
> 
> altp2m is not architecture specific at all, and there's even support for
> ARM out on the mailing list.  Therefore, the altp2m mode wants to be
> common, just like the new MAX_ALTP2M setting already is.

Initially I had it as a set of XEN_DOMCTL_CDF_* flags, but it wasn't
clear to me whether the modes could be shared between arches.

> Both fields can reasonably share uint32_t, but could you work with Petr
> to make both halfs of this land cleanly.

I'm happy for Petr to pick this patch as part of the series if he
feels like.

I assume the plan would be to add an XEN_DOMCTL_CDF_altp2m flag, and
then a new field to signal the mode.

> 
> As to the HVMPARAM, I'd really quite like to delete it.  It was always a
> bodge, and there's a full set of HVMOP_altp2m_* for a guest to use.

I've assumed we must keep HVM_PARAM_ALTP2M for backwards
compatibility.

> > diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> > index 20e83cf38bbd..dff790060605 100644
> > --- a/xen/arch/x86/domain.c
> > +++ b/xen/arch/x86/domain.c
> > @@ -708,13 +711,33 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
> >          }
> >      }
> >  
> > -    if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
> > +    if ( config->arch.misc_flags & ~XEN_X86_MISC_FLAGS_ALL )
> >      {
> >          dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
> >                  config->arch.misc_flags);
> >          return -EINVAL;
> >      }
> >  
> > +    if ( altp2m && (altp2m & (altp2m - 1)) )
> > +    {
> > +        dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
> > +                config->flags);
> > +        return -EINVAL;
> 
> I think this would be clearer to follow by having a 2 bit field called
> altp2m_mode and check for <= 2.

Don't we need 3 bits, for mixed, external and limited modes?

We could do with 2 bits if we signal altp2m enabled in a different
field, and then introduce a field to just contain the mode.

FWIW, the check should be `if ( altp2m & (altp2m - 1) )`.  I had
updated this, but seems like I missed to re-generate the patches.

> > +    }
> > +
> > +    if ( altp2m && nested_virt )
> > +    {
> > +        dprintk(XENLOG_INFO,
> > +                "Nested virt and altp2m are mutually incompatible\n");
> 
> There's nothing inherently incompatible.  I think it's more that noone
> had any interest in trying to make it work in combination with nested p2ms.
> 
> I'd phrase it as "not supported", rather than incompatible.

"Nested virt and altp2m are not supported together\n"

> > +        return -EINVAL;
> > +    }
> > +
> > +    if ( altp2m && !hap )
> > +    {
> > +        dprintk(XENLOG_INFO, "altp2m requires HAP\n");
> > +        return -EINVAL;
> > +    }
> 
> altp2m ought to work fine with shadow.  It's only if you want VMFUNC/#VE
> acceleration that you depend on EPT.
> 
> Again, I'd phrase this as "not supported".

"altp2m is only supported with HAP\n"

To avoid the double negation of "not supported without HAP" wording.

Thanks, Roger.

Re: [PATCH for-4.19 v2 3/3] xen/x86: remove foreign mappings from the p2m on teardown

[Roger Pau Monné]

On Wed, May 08, 2024 at 01:23:23PM +0200, Roger Pau Monne wrote:
> Iterate over the p2m up to the maximum recorded gfn and remove any foreign
> mappings, in order to drop the underlying page references and thus don't keep
> extra page references if a domain is destroyed while still having foreign
> mappings on it's p2m.
> 
> The logic is similar to the one used on Arm.
> 
> Note that foreign mappings cannot be created by guests that have altp2m or
> nested HVM enabled, as p2ms different than the host one are not currently
> scrubbed when destroyed in order to drop references to any foreign maps.
> 
> It's unclear whether the right solution is to take an extra reference when
> foreign maps are added to p2ms different than the host one, or just rely on the
> host p2m already having a reference.  The mapping being removed from the host
> p2m should cause it to be dropped on all domain p2ms.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Changes since v1:
>  - Use existing p2m max_mapped_pfn field.
>  - Prevent creating foreign mappings by guests that have altp2m or nestedhvm
>    enabled.
> ---
>  CHANGELOG.md                   |  1 +
>  xen/arch/x86/domain.c          |  8 +++-
>  xen/arch/x86/include/asm/p2m.h | 26 +++++++------
>  xen/arch/x86/mm/p2m-basic.c    | 17 +++++++++
>  xen/arch/x86/mm/p2m.c          | 68 ++++++++++++++++++++++++++++++++--
>  5 files changed, 103 insertions(+), 17 deletions(-)
> 
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 8041cfb7d243..09bdb9b97578 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
>     - HVM PIRQs are disabled by default.
>     - Reduce IOMMU setup time for hardware domain.
>   - xl/libxl configures vkb=[] for HVM domains with priority over vkb_device.
> + - Allow HVM/PVH domains to map foreign pages.
>  
>  ### Added
>   - On x86:
> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
> index dff790060605..1cb3ccddab00 100644
> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -718,7 +718,7 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
>          return -EINVAL;
>      }
>  
> -    if ( altp2m && (altp2m & (altp2m - 1)) )
> +    if ( altp2m & (altp2m - 1) )
>      {
>          dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
>                  config->flags);
> @@ -2387,6 +2387,7 @@ int domain_relinquish_resources(struct domain *d)
>          enum {
>              PROG_iommu_pagetables = 1,
>              PROG_shared,
> +            PROG_mappings,
>              PROG_paging,
>              PROG_vcpu_pagetables,
>              PROG_xen,
> @@ -2435,6 +2436,11 @@ int domain_relinquish_resources(struct domain *d)
>          }
>  #endif
>  
> +    PROGRESS(mappings):
> +        ret = relinquish_p2m_mapping(d);
> +        if ( ret )
> +            return ret;
> +
>      PROGRESS(paging):
>  
>          /* Tear down paging-assistance stuff. */
> diff --git a/xen/arch/x86/include/asm/p2m.h b/xen/arch/x86/include/asm/p2m.h
> index 107b9f260848..c1478ffc3647 100644
> --- a/xen/arch/x86/include/asm/p2m.h
> +++ b/xen/arch/x86/include/asm/p2m.h
> @@ -383,6 +383,8 @@ struct p2m_domain {
>  
>      /* Number of foreign mappings. */
>      unsigned long      nr_foreign;
> +    /* Cursor for iterating over the p2m on teardown. */
> +    unsigned long      teardown_gfn;
>  #endif /* CONFIG_HVM */
>  };
>  
> @@ -395,16 +397,7 @@ struct p2m_domain {
>  #endif
>  #include <xen/p2m-common.h>
>  
> -static inline bool arch_acquire_resource_check(struct domain *d)
> -{
> -    /*
> -     * FIXME: Until foreign pages inserted into the P2M are properly
> -     * reference counted, it is unsafe to allow mapping of
> -     * resource pages unless the caller is the hardware domain
> -     * (see set_foreign_p2m_entry()).
> -     */
> -    return !paging_mode_translate(d) || is_hardware_domain(d);

This must be:

    return is_pv_domain(d) ||
           (d->arch.hvm.params[HVM_PARAM_ALTP2M] == XEN_ALTP2M_disabled &&
            !nestedhvm_enabled(d));

Sorry.

Re: [PATCH for-4.19 v2 2/3] xen/x86: enable altp2m at create domain domctl

[Jan Beulich]

On 09.05.2024 10:23, Roger Pau Monné wrote:
> On Wed, May 08, 2024 at 08:38:07PM +0100, Andrew Cooper wrote:
>> On 08/05/2024 12:23 pm, Roger Pau Monne wrote:
>>> Enabling it using an HVM param is fragile, and complicates the logic when
>>> deciding whether options that interact with altp2m can also be enabled.
>>>
>>> Leave the HVM param value for consumption by the guest, but prevent it from
>>> being set.  Enabling is now done using the misc_flags field in
>>> xen_arch_domainconfig.
>>>
>>> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
>>> ---
>>> Changes since v1:
>>>  - New in this version.
>>
>> Ha.  So this is actually work that Petr has been wanting to do.
>>
>> Petr has a series hoping to make it into 4.19 (x86: Make MAX_ALTP2M
>> configurable), which just missed out on this side of things.
>>
>> altp2m is not architecture specific at all, and there's even support for
>> ARM out on the mailing list.  Therefore, the altp2m mode wants to be
>> common, just like the new MAX_ALTP2M setting already is.
> 
> Initially I had it as a set of XEN_DOMCTL_CDF_* flags, but it wasn't
> clear to me whether the modes could be shared between arches.
> 
>> Both fields can reasonably share uint32_t, but could you work with Petr
>> to make both halfs of this land cleanly.
> 
> I'm happy for Petr to pick this patch as part of the series if he
> feels like.
> 
> I assume the plan would be to add an XEN_DOMCTL_CDF_altp2m flag, and
> then a new field to signal the mode.
> 
>>
>> As to the HVMPARAM, I'd really quite like to delete it.  It was always a
>> bodge, and there's a full set of HVMOP_altp2m_* for a guest to use.
> 
> I've assumed we must keep HVM_PARAM_ALTP2M for backwards
> compatibility.
> 
>>> diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
>>> index 20e83cf38bbd..dff790060605 100644
>>> --- a/xen/arch/x86/domain.c
>>> +++ b/xen/arch/x86/domain.c
>>> @@ -708,13 +711,33 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
>>>          }
>>>      }
>>>  
>>> -    if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
>>> +    if ( config->arch.misc_flags & ~XEN_X86_MISC_FLAGS_ALL )
>>>      {
>>>          dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
>>>                  config->arch.misc_flags);
>>>          return -EINVAL;
>>>      }
>>>  
>>> +    if ( altp2m && (altp2m & (altp2m - 1)) )
>>> +    {
>>> +        dprintk(XENLOG_INFO, "Multiple altp2m options selected in flags: %#x\n",
>>> +                config->flags);
>>> +        return -EINVAL;
>>
>> I think this would be clearer to follow by having a 2 bit field called
>> altp2m_mode and check for <= 2.
> 
> Don't we need 3 bits, for mixed, external and limited modes?
> 
> We could do with 2 bits if we signal altp2m enabled in a different
> field, and then introduce a field to just contain the mode.

I think what Andrew meant is

#define XEN_X86_ALTP2M_MIXED   (1U << 1)
/* Enable altp2m external mode. */
#define XEN_X86_ALTP2M_EXT     (2U << 1)
/* Enable altp2m limited mode. */
#define XEN_X86_ALTP2M_LIMITED (3U << 1)

(leaving aside the x86-only vs common aspect). That would also eliminate
the ability to request multiple (conflicting) modes.

Jan

Re: [PATCH for-4.19 v2 3/3] xen/x86: remove foreign mappings from the p2m on teardown

[Jan Beulich]

On 08.05.2024 13:23, Roger Pau Monne wrote:
> @@ -1043,7 +1045,7 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
>          break;
>  
>      case p2m_map_foreign:
> -        if ( !mfn_valid(nfn) )
> +        if ( !mfn_valid(nfn) || p2m != p2m_get_hostp2m(p2m->domain) )
>          {
>              ASSERT_UNREACHABLE();
>              return -EINVAL;
> @@ -1068,7 +1070,7 @@ static inline int p2m_entry_modify(struct p2m_domain *p2m, p2m_type_t nt,
>          break;
>  
>      case p2m_map_foreign:
> -        if ( !mfn_valid(ofn) )
> +        if ( !mfn_valid(ofn) || p2m != p2m_get_hostp2m(p2m->domain) )
>          {
>              ASSERT_UNREACHABLE();
>              return -EINVAL;

I wonder whether these two hunks wouldn't more logically be part of patch 1
or 2 (probably not 1, as at that point the assertion isn't quite correct
yet). Then again it may be as (seemingly) unrelated there as it is here.
Thus
Reviewed-by: Jan Beulich <jbeulich@suse.com>

Jan

Re: [PATCH for-4.19 v2 1/3] xen/x86: account number of foreign mappings in the p2m

[Jan Beulich]

On 08.05.2024 13:23, Roger Pau Monne wrote:
> Such information will be needed in order to remove foreign mappings during
> teardown for HVM guests.
> 
> Right now the introduced counter is not consumed.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>
Yet I think it shouldn't be committed (much) ahead of patch 3.

Jan