[PATCH 0/3] x86/HVM: batch the flushes of grant unmaps

[PATCH 0/3] x86/HVM: batch the flushes of grant unmaps

[Luca Di Maio]

Hi all,

GNTTABOP_unmap_grant_ref already batches the TLB flushes of a PV
caller: gnttab_unmap_grant_ref() unmaps up to 32 grants, flushes the
TLBs once with gnttab_flush_tlb(), and only then releases the pages.
gnttab_flush_tlb() does nothing for a translated caller, whose unmaps
go through the p2m, and the p2m code flushes the guest TLBs and the
IOTLB for every page instead. A PVH dom0 running netback pays both
flushes for every page a domU transmits.

This series gives translated callers the same batching, for both
flushes, before the pages are released. The IOTLB half has no PV
counterpart: a PV dom0's IOMMU maps all of RAM by default, so its
grant unmaps don't touch it. Patch 1 is preparatory, and patches 2 and
3 have the details.

iperf3 between a PVH dom0 running netback and two PVH domUs:

  machine:  c7i.metal-24xl, Xeon Platinum 8488C, 96 threads,
            VT-d with shared EPT tables
  dom0:     96 pinned vCPUs, 4 GiB, Debian 13, Linux 6.12
  domUs:    8 vCPUs, 1 GiB, Linux 6.18, on a bridge
  Xen:      staging at 9a6cc49b44, built with GCC 14.2
  runs:     10 s, Gbit/s received, higher is better, 60 per row
            from 4 boots of each build, interleaved
  shown:    mean of the runs +/- their relative standard deviation

                   base          patched       %change
  domU to dom0
    1 stream       0.64 +/-1%    3.07 +/-3%      +379%
    2 streams      0.81 +/-6%    3.09 +/-2%      +282%
    4 streams      0.84 +/-4%    2.74 +/-6%      +227%
    8 streams      0.83 +/-2%    2.40 +/-5%      +190%
    16 streams     0.81 +/-2%    2.22 +/-3%      +174%
  domU to domU
    1 stream       0.62 +/-6%    2.26 +/-3%      +263%
    2 streams      0.73 +/-7%    2.31 +/-10%     +218%
    4 streams      0.75 +/-4%    1.86 +/-11%     +149%
    8 streams      0.73 +/-4%    1.62 +/-10%     +122%
    16 streams     0.71 +/-3%    1.59 +/-6%      +124%
  dom0 to domU
    1 stream       4.19 +/-6%    4.12 +/-6%      -1.7%
    2 streams      4.87 +/-9%    4.71 +/-10%     -3.4%
    4 streams      8.73 +/-21%   8.52 +/-23%     -2.5%
    8 streams     12.53 +/-17%  12.57 +/-17%     +0.3%
    16 streams    16.85 +/-12%  16.72 +/-11%     -0.7%

The table comes from one of 13 sessions on this instance type, which
ran 7,779 runs over 57 boots in total.

A domU transmits 2.2 to 4.8 times as much, and 2.2 to 4.0 times as
much per busy dom0 vCPU, where base never exceeds 0.84 Gbit/s whatever
the number of streams.

From dom0 to a domU, netback fills the domU's buffers with grant
copies and maps few of its packets, so the series barely runs there
and the differences stay within the margin of error.

Between boots of the same build, the single-stream mean varied no
more than the run-to-run spread predicts. A debug build ran the same
benchmark without assertion failures or IOMMU faults.

Thanks,
L

Luca Di Maio (3):
  IOMMU: record the flushes skipped under iommu_dont_flush_iotlb
  x86/HVM: defer IOTLB flushes across a grant unmap batch
  x86/HVM: defer p2m TLB flushes across a grant unmap batch

 xen/arch/arm/include/asm/grant_table.h     |  3 ++
 xen/arch/x86/hvm/grant_table.c             | 53 +++++++++++++++++++++-
 xen/arch/x86/include/asm/grant_table.h     | 13 ++++++
 xen/arch/x86/include/asm/hvm/grant_table.h |  5 ++
 xen/arch/x86/include/asm/hvm/vcpu.h        | 10 ++++
 xen/arch/x86/mm/p2m-ept.c                  |  4 ++
 xen/common/grant_table.c                   |  6 +++
 xen/drivers/passthrough/iommu.c            | 12 ++++-
 xen/include/xen/iommu.h                    |  1 +
 9 files changed, 105 insertions(+), 2 deletions(-)


base-commit: 41ddcfe808ec1228289ae60c30768d50a58c98fe
--
2.54.0

[PATCH 1/3] IOMMU: record the flushes skipped under iommu_dont_flush_iotlb

[Luca Di Maio]

While iommu_dont_flush_iotlb is set, iommu_legacy_unmap() and the
shared-EPT path of ept_set_entry() skip their IOTLB flush and drop the
flags it would have used. Accumulate them in the per-CPU
iommu_skipped_flush_flags, so that a caller deferring the flush can
issue it with the same flags. The next patch does so for grant unmaps.

Use a uint8_t, as a fixed width is meant: a byte holds every flag and
fits in the padding after iommu_dont_flush_iotlb, so the per-CPU data
that follows doesn't move.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Luca Di Maio <luca@edera.dev>

---
 xen/arch/x86/mm/p2m-ept.c       |  4 ++++
 xen/drivers/passthrough/iommu.c | 12 +++++++++++-
 xen/include/xen/iommu.h         |  1 +
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 204bdb054a..1a87b25d94 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -981,6 +981,10 @@ out:
                                    (iommu_flags ? IOMMU_FLUSHF_added : 0) |
                                    (vtd_pte_present ? IOMMU_FLUSHF_modified
                                                     : 0));
+        else if ( iommu_use_hap_pt(d) )
+            this_cpu(iommu_skipped_flush_flags) |=
+                (iommu_flags ? IOMMU_FLUSHF_added : 0) |
+                (vtd_pte_present ? IOMMU_FLUSHF_modified : 0);
         else if ( need_iommu_pt_sync(d) )
             rc = iommu_flags ?
                 iommu_legacy_map(d, _dfn(gfn), mfn, 1ul << order, iommu_flags) :
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index 4b22ecfe65..70ee45afb9 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -64,6 +64,11 @@ bool __read_mostly iommu_hap_pt_share = true;
 bool __read_mostly iommu_debug;
 
 DEFINE_PER_CPU(bool, iommu_dont_flush_iotlb);
+/*
+ * A byte holds every IOMMU_FLUSHF_*, and fits in the alignment padding the
+ * bool above leaves, instead of shifting the per-CPU data that follows.
+ */
+DEFINE_PER_CPU(uint8_t, iommu_skipped_flush_flags);
 
 static int __init cf_check parse_iommu_param(const char *s)
 {
@@ -458,7 +463,12 @@ int iommu_legacy_unmap(struct domain *d, dfn_t dfn, unsigned long page_count)
     unsigned int flush_flags = 0;
     int rc = iommu_unmap(d, dfn, page_count, 0, &flush_flags);
 
-    if ( !this_cpu(iommu_dont_flush_iotlb) && !rc )
+    if ( this_cpu(iommu_dont_flush_iotlb) )
+    {
+        BUILD_BUG_ON(IOMMU_FLUSHF_all > UINT8_MAX);
+        this_cpu(iommu_skipped_flush_flags) |= flush_flags;
+    }
+    else if ( !rc )
         rc = iommu_iotlb_flush(d, dfn, page_count, flush_flags);
 
     return rc;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 37c4a1dc82..028f41bc15 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -501,6 +501,7 @@ void iommu_dev_iotlb_flush_timeout(struct domain *d, struct pci_dev *pdev);
  * the caller.
  */
 DECLARE_PER_CPU(bool, iommu_dont_flush_iotlb);
+DECLARE_PER_CPU(uint8_t, iommu_skipped_flush_flags);
 
 bool arch_iommu_use_permitted(const struct domain *d);
 
-- 
2.54.0

[PATCH 2/3] x86/HVM: defer IOTLB flushes across a grant unmap batch

[Luca Di Maio]

gnttab_unmap_grant_ref() flushes the TLBs of a PV caller once per group
of GNTTAB_UNMAP_BATCH_SIZE unmaps. A translated caller instead gets an
IOTLB flush per page from the p2m code, so netback in a PVH dom0 issues
one for every page a domU transmits.

Open a batch for translated callers before each group, and end it next
to gnttab_flush_tlb(), so before unmap_common_complete() releases the
pages. Ending the batch flushes the smallest aligned power-of-two range
holding all the unmapped gfns, as VT-d flushes any other range as a
whole domain.

The batch lives in the vCPU, and iommu_dont_flush_iotlb isn't held
across the guest copies between the unmaps, as those can sleep on a
vm_event ring and change the p2m, e.g. by unsharing a page. If the
flush fails, the pages may still be reachable through the IOMMU, so
the group isn't completed and its pages leak.

PV callers, Arm, and other callers of replace_grant_host_mapping() are
unchanged.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Luca Di Maio <luca@edera.dev>

---
 xen/arch/arm/include/asm/grant_table.h     |  3 ++
 xen/arch/x86/hvm/grant_table.c             | 47 +++++++++++++++++++++-
 xen/arch/x86/include/asm/grant_table.h     | 13 ++++++
 xen/arch/x86/include/asm/hvm/grant_table.h |  5 +++
 xen/arch/x86/include/asm/hvm/vcpu.h        |  9 +++++
 xen/common/grant_table.c                   |  6 +++
 6 files changed, 82 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/include/asm/grant_table.h b/xen/arch/arm/include/asm/grant_table.h
index 7661a95341..66f90083db 100644
--- a/xen/arch/arm/include/asm/grant_table.h
+++ b/xen/arch/arm/include/asm/grant_table.h
@@ -40,6 +40,9 @@ int create_grant_host_mapping(uint64_t gpaddr, mfn_t frame,
 int replace_grant_host_mapping(uint64_t gpaddr, mfn_t frame,
                                uint64_t new_gpaddr, unsigned int flags);
 
+static inline void gnttab_unmap_batch_begin(void) {}
+static inline int gnttab_unmap_batch_end(void) { return 0; }
+
 /*
  * The region used by Xen on the memory will never be mapped in DOM0
  * memory layout. Therefore it can be used for the grant table.
diff --git a/xen/arch/x86/hvm/grant_table.c b/xen/arch/x86/hvm/grant_table.c
index afe449d888..1d7f9b19f9 100644
--- a/xen/arch/x86/hvm/grant_table.c
+++ b/xen/arch/x86/hvm/grant_table.c
@@ -41,6 +41,8 @@ int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
     p2m_type_t type;
     mfn_t old_mfn;
     struct domain *d = current->domain;
+    struct hvm_grant_unmap_batch *batch = &current->arch.hvm.grant_unmap_batch;
+    int rc;
 
     if ( new_addr != 0 || (flags & GNTMAP_contains_pte) )
         return GNTST_general_error;
@@ -54,7 +56,25 @@ int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
                  type, mfn_x(old_mfn), mfn_x(frame));
         return GNTST_general_error;
     }
-    if ( p2m_remove_page(d, _gfn(gfn), frame, PAGE_ORDER_4K) )
+
+    /* Not across the guest copies, which can change the p2m or sleep. */
+    if ( batch->open )
+    {
+        this_cpu(iommu_skipped_flush_flags) = 0;
+        this_cpu(iommu_dont_flush_iotlb) = true;
+    }
+
+    rc = p2m_remove_page(d, _gfn(gfn), frame, PAGE_ORDER_4K);
+
+    if ( batch->open )
+    {
+        this_cpu(iommu_dont_flush_iotlb) = false;
+        batch->iommu_flags |= this_cpu(iommu_skipped_flush_flags);
+        batch->gfn_lo = min(batch->gfn_lo, gfn);
+        batch->gfn_hi = max(batch->gfn_hi, gfn);
+    }
+
+    if ( rc )
     {
         put_gfn(d, gfn);
         return GNTST_general_error;
@@ -64,6 +84,31 @@ int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
     return GNTST_okay;
 }
 
+void grant_p2m_unmap_batch_begin(void)
+{
+    struct hvm_grant_unmap_batch *batch = &current->arch.hvm.grant_unmap_batch;
+
+    ASSERT(!batch->open);
+    *batch = (struct hvm_grant_unmap_batch){ .open = true, .gfn_lo = ~0UL };
+}
+
+int grant_p2m_unmap_batch_end(void)
+{
+    struct domain *d = current->domain;
+    struct hvm_grant_unmap_batch *batch = &current->arch.hvm.grant_unmap_batch;
+    unsigned int order;
+
+    batch->open = false;
+
+    if ( !batch->iommu_flags )
+        return 0;
+
+    /* VT-d flushes any range but an aligned power of two as a whole domain. */
+    order = flsl(batch->gfn_lo ^ batch->gfn_hi);
+    return iommu_iotlb_flush(d, _dfn(ROUNDDOWN(batch->gfn_lo, 1UL << order)),
+                             1UL << order, batch->iommu_flags);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/x86/include/asm/grant_table.h b/xen/arch/x86/include/asm/grant_table.h
index e939c9bacb..3aae47092f 100644
--- a/xen/arch/x86/include/asm/grant_table.h
+++ b/xen/arch/x86/include/asm/grant_table.h
@@ -32,6 +32,19 @@ static inline int replace_grant_host_mapping(uint64_t addr, mfn_t frame,
     return replace_grant_pv_mapping(addr, frame, new_addr, flags);
 }
 
+static inline void gnttab_unmap_batch_begin(void)
+{
+    if ( paging_mode_external(current->domain) )
+        grant_p2m_unmap_batch_begin();
+}
+
+static inline int __must_check gnttab_unmap_batch_end(void)
+{
+    if ( paging_mode_external(current->domain) )
+        return grant_p2m_unmap_batch_end();
+    return 0;
+}
+
 #define gnttab_get_frame_gfn(gt, st, idx) ({                             \
     mfn_t mfn_ = (st) ? gnttab_status_mfn(gt, idx)                       \
                       : gnttab_shared_mfn(gt, idx);                      \
diff --git a/xen/arch/x86/include/asm/hvm/grant_table.h b/xen/arch/x86/include/asm/hvm/grant_table.h
index 01e23f79b8..9877262f51 100644
--- a/xen/arch/x86/include/asm/hvm/grant_table.h
+++ b/xen/arch/x86/include/asm/hvm/grant_table.h
@@ -19,6 +19,8 @@ int create_grant_p2m_mapping(uint64_t addr, mfn_t frame,
                              unsigned int cache_flags);
 int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
                               uint64_t new_addr, unsigned int flags);
+void grant_p2m_unmap_batch_begin(void);
+int grant_p2m_unmap_batch_end(void);
 
 #else
 
@@ -37,6 +39,9 @@ static inline int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
     return GNTST_general_error;
 }
 
+static inline void grant_p2m_unmap_batch_begin(void) {}
+static inline int grant_p2m_unmap_batch_end(void) { return 0; }
+
 #endif
 
 #endif /* __X86_HVM_GRANT_TABLE_H__ */
diff --git a/xen/arch/x86/include/asm/hvm/vcpu.h b/xen/arch/x86/include/asm/hvm/vcpu.h
index 876c230784..12b7e9b707 100644
--- a/xen/arch/x86/include/asm/hvm/vcpu.h
+++ b/xen/arch/x86/include/asm/hvm/vcpu.h
@@ -108,6 +108,13 @@ struct altp2mvcpu {
 
 #define vcpu_altp2m(v) ((v)->arch.hvm.avcpu)
 
+/* Per vCPU, as the guest copies between the unmaps can sleep. */
+struct hvm_grant_unmap_batch {
+    bool          open;
+    unsigned int  iommu_flags;
+    unsigned long gfn_lo, gfn_hi;
+};
+
 struct hvm_vcpu {
     /* Guest control-register and EFER values, just as the guest sees them. */
     unsigned long       guest_cr[5];
@@ -172,6 +179,8 @@ struct hvm_vcpu {
     /* Pending hw/sw interrupt (.vector = -1 means nothing pending). */
     struct x86_event     inject_event;
 
+    struct hvm_grant_unmap_batch grant_unmap_batch;
+
 #ifdef CONFIG_VIRIDIAN
     struct viridian_vcpu *viridian;
 #endif
diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index b15805f3f9..ade42b4cb4 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -1722,6 +1722,7 @@ gnttab_unmap_grant_ref(
         c = min(count, (unsigned int)GNTTAB_UNMAP_BATCH_SIZE);
         partial_done = 0;
 
+        gnttab_unmap_batch_begin();
         for ( i = 0; i < c; i++ )
         {
             if ( unlikely(__copy_from_guest(&op, uop, 1)) )
@@ -1734,6 +1735,9 @@ gnttab_unmap_grant_ref(
         }
 
         gnttab_flush_tlb(current->domain);
+        /* After a failed flush the pages may still be reachable: leak them. */
+        if ( gnttab_unmap_batch_end() )
+            partial_done = 0;
 
         for ( i = 0; i < partial_done; i++ )
             unmap_common_complete(&common[i]);
@@ -1749,6 +1753,8 @@ gnttab_unmap_grant_ref(
 
 fault:
     gnttab_flush_tlb(current->domain);
+    if ( gnttab_unmap_batch_end() )
+        partial_done = 0;
 
     for ( i = 0; i < partial_done; i++ )
         unmap_common_complete(&common[i]);
-- 
2.54.0

[PATCH 3/3] x86/HVM: defer p2m TLB flushes across a grant unmap batch

[Luca Di Maio]

A grant unmap by a translated caller also flushes the guest TLBs:
p2m_remove_page() sets need_flush of the host p2m, and put_gfn()
flushes when it drops the last level of the p2m lock. Each flush sends
an IPI to every other CPU in d->dirty_cpumask and waits for all of
them, so a PVH dom0 with busy vCPUs pays a round of IPIs for every page
it unmaps.

Defer that flush to the end of the batch, still before the pages are
released. Under the gfn lock of get_gfn_query() only the removal can
set need_flush, as the query neither allocates nor unshares, so no
other change's flush is deferred. p2m_free_ptp() still flushes before
it frees a table.

The final flush reaches the CPUs in d->dirty_cpumask at that time. A
CPU that left the mask in between has flushed already, as
__context_switch() clears it from the mask only after write_ptbase()
has flushed the guest TLBs. With EPT, each change also marks every CPU
for an INVEPT before its next VM entry.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Luca Di Maio <luca@edera.dev>

---
 xen/arch/x86/hvm/grant_table.c      | 6 ++++++
 xen/arch/x86/include/asm/hvm/vcpu.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/xen/arch/x86/hvm/grant_table.c b/xen/arch/x86/hvm/grant_table.c
index 1d7f9b19f9..e340712dd6 100644
--- a/xen/arch/x86/hvm/grant_table.c
+++ b/xen/arch/x86/hvm/grant_table.c
@@ -41,6 +41,7 @@ int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
     p2m_type_t type;
     mfn_t old_mfn;
     struct domain *d = current->domain;
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
     struct hvm_grant_unmap_batch *batch = &current->arch.hvm.grant_unmap_batch;
     int rc;
 
@@ -72,6 +73,8 @@ int replace_grant_p2m_mapping(uint64_t addr, mfn_t frame,
         batch->iommu_flags |= this_cpu(iommu_skipped_flush_flags);
         batch->gfn_lo = min(batch->gfn_lo, gfn);
         batch->gfn_hi = max(batch->gfn_hi, gfn);
+        batch->tlb_flush |= p2m->need_flush;
+        p2m->need_flush = false;
     }
 
     if ( rc )
@@ -95,10 +98,13 @@ void grant_p2m_unmap_batch_begin(void)
 int grant_p2m_unmap_batch_end(void)
 {
     struct domain *d = current->domain;
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
     struct hvm_grant_unmap_batch *batch = &current->arch.hvm.grant_unmap_batch;
     unsigned int order;
 
     batch->open = false;
+    if ( batch->tlb_flush )
+        p2m->tlb_flush(p2m);
 
     if ( !batch->iommu_flags )
         return 0;
diff --git a/xen/arch/x86/include/asm/hvm/vcpu.h b/xen/arch/x86/include/asm/hvm/vcpu.h
index 12b7e9b707..9638e50e91 100644
--- a/xen/arch/x86/include/asm/hvm/vcpu.h
+++ b/xen/arch/x86/include/asm/hvm/vcpu.h
@@ -111,6 +111,7 @@ struct altp2mvcpu {
 /* Per vCPU, as the guest copies between the unmaps can sleep. */
 struct hvm_grant_unmap_batch {
     bool          open;
+    bool          tlb_flush;
     unsigned int  iommu_flags;
     unsigned long gfn_lo, gfn_hi;
 };
-- 
2.54.0