[PATCH v6 0/3] Support multiple ioreq pages

[PATCH v6 0/3] Support multiple ioreq pages

[Julian Vetter]

Hello Jan,
thank you again for your thorough feedback. I have addressed your
comments. I just have one remaining issue. For the temporary MFNs array
I now use a static array, i.e., mfns[IOREQ_NR_PAGES_MAX], with a
IOREQ_NR_PAGES_MAX = DIV_ROUND_UP(HVM_MAX_VCPUS, PAGE_SIZE / sizeof(ioreq_t)).
Is this appropriate? Or should I leave the HVM_MAX_VCPUS out of this and
instead use a fixed constant value? Also for now I haven't taken the
path to permanently store the MFNs as you suggested. I'm not sure, but I
don't think it's really necessary. Since this array will rarely exceed
the size of 2 or 3.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
Julian Vetter (3):
  ioreq: switch ioreq page allocation to vmap
  ioreq: Indent ioreq_server_alloc_mfn() body one level deeper
  x86/ioreq: Extend ioreq server to support multiple ioreq pages

 xen/arch/x86/hvm/ioreq.c |  63 ++++++++++++++++---
 xen/common/ioreq.c       | 127 ++++++++++++++++++++++++++-------------
 xen/include/xen/ioreq.h  |  13 +++-
 3 files changed, 151 insertions(+), 52 deletions(-)

-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[PATCH v6 1/3] ioreq: switch ioreq page allocation to vmap

[Julian Vetter]

Switch the Xen-side ioreq page mapping from prepare_ring_for_helper() /
map_domain_page_global() to explicit vmap(), to ensure vmap_to_page()
can recover the struct page_info * uniformly during teardown.

This is a prerequisite for multi-page ioreq support: the non-buf ioreq
region will need to span multiple pages for domains with more vCPUs than
fit in a single page, and vmap() is the natural interface for contiguous
multi-page Xen VA mappings.

In non-debug builds map_domain_page_global() uses the directmap for low
MFNs rather than vmap(), so this change has a small overhead in the
common case. Debug builds already used vmap() indirectly.

With both paths using vmap(), vmap_to_page() can recover the struct
page_info * uniformly, so drop the 'page' field from struct ioreq_page
and update all callers accordingly.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
Changes in v6:
- Updated commit message to clearly specify why these changes are made
- Added comment to say that this is {prepare,destroy}_ring_for_helper()
  just using vmap_to_page() + v{map,unmap}()
- Kept proper ordering in ioreq_server_free_mfn(), first clearing the va
  pointer before unmapping
---
 xen/arch/x86/hvm/ioreq.c | 55 +++++++++++++++++++++++++++++++++-------
 xen/common/ioreq.c       | 34 +++++++++++++------------
 xen/include/xen/ioreq.h  |  1 -
 3 files changed, 64 insertions(+), 26 deletions(-)

diff --git a/xen/arch/x86/hvm/ioreq.c b/xen/arch/x86/hvm/ioreq.c
index a5fa97e149..3cabec141c 100644
--- a/xen/arch/x86/hvm/ioreq.c
+++ b/xen/arch/x86/hvm/ioreq.c
@@ -15,6 +15,7 @@
 #include <xen/sched.h>
 #include <xen/softirq.h>
 #include <xen/trace.h>
+#include <xen/vmap.h>
 #include <xen/vpci.h>
 
 #include <asm/hvm/emulate.h>
@@ -128,8 +129,13 @@ static void hvm_unmap_ioreq_gfn(struct ioreq_server *s, bool buf)
     if ( gfn_eq(iorp->gfn, INVALID_GFN) )
         return;
 
-    destroy_ring_for_helper(&iorp->va, iorp->page);
-    iorp->page = NULL;
+    /* Equivalent to destroy_ring_for_helper(), using vmap_to_page(). */
+    if ( iorp->va )
+    {
+        put_page_and_type(vmap_to_page(iorp->va));
+        vunmap(iorp->va);
+        iorp->va = NULL;
+    }
 
     hvm_free_ioreq_gfn(s, iorp->gfn);
     iorp->gfn = INVALID_GFN;
@@ -139,9 +145,12 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
 {
     struct domain *d = s->target;
     struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
+    struct page_info *page;
+    p2m_type_t p2mt;
+    mfn_t mfn;
     int rc;
 
-    if ( iorp->page )
+    if ( iorp->va )
     {
         /*
          * If a page has already been allocated (which will happen on
@@ -162,12 +171,40 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
     if ( gfn_eq(iorp->gfn, INVALID_GFN) )
         return -ENOMEM;
 
-    rc = prepare_ring_for_helper(d, gfn_x(iorp->gfn), &iorp->page,
-                                 &iorp->va);
-
+    /*
+     * Equivalent to prepare_ring_for_helper() using vmap(). Using vmap()
+     * rather than map_domain_page_global() ensures vmap_to_page() can
+     * recover the struct page_info * uniformly at teardown, which is
+     * needed to support multi-page ioreq mappings (see nr_ioreq_pages()).
+     */
+    rc = check_get_page_from_gfn(d, iorp->gfn, false, &p2mt, &page);
     if ( rc )
-        hvm_unmap_ioreq_gfn(s, buf);
+    {
+        if ( rc == -EAGAIN )
+            rc = -ENOENT;
+        goto fail;
+    }
+
+    if ( !get_page_type(page, PGT_writable_page) )
+    {
+        put_page(page);
+        rc = -EINVAL;
+        goto fail;
+    }
+
+    mfn = page_to_mfn(page);
+    iorp->va = vmap(&mfn, 1);
+    if ( !iorp->va )
+    {
+        put_page_and_type(page);
+        rc = -ENOMEM;
+        goto fail;
+    }
+
+    return 0;
 
+ fail:
+    hvm_unmap_ioreq_gfn(s, buf);
     return rc;
 }
 
@@ -179,7 +216,7 @@ static void hvm_remove_ioreq_gfn(struct ioreq_server *s, bool buf)
     if ( gfn_eq(iorp->gfn, INVALID_GFN) )
         return;
 
-    if ( p2m_remove_page(d, iorp->gfn, page_to_mfn(iorp->page), 0) )
+    if ( p2m_remove_page(d, iorp->gfn, vmap_to_mfn(iorp->va), 0) )
         domain_crash(d);
     clear_page(iorp->va);
 }
@@ -195,7 +232,7 @@ static int hvm_add_ioreq_gfn(struct ioreq_server *s, bool buf)
 
     clear_page(iorp->va);
 
-    rc = p2m_add_page(d, iorp->gfn, page_to_mfn(iorp->page), 0, p2m_ram_rw);
+    rc = p2m_add_page(d, iorp->gfn, vmap_to_mfn(iorp->va), 0, p2m_ram_rw);
     if ( rc == 0 )
         paging_mark_pfn_dirty(d, _pfn(gfn_x(iorp->gfn)));
 
diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c
index f5fd30ce12..d8d02167b4 100644
--- a/xen/common/ioreq.c
+++ b/xen/common/ioreq.c
@@ -17,11 +17,11 @@
  */
 
 #include <xen/domain.h>
-#include <xen/domain_page.h>
 #include <xen/event.h>
 #include <xen/init.h>
 #include <xen/ioreq.h>
 #include <xen/irq.h>
+#include <xen/vmap.h>
 #include <xen/lib.h>
 #include <xen/paging.h>
 #include <xen/sched.h>
@@ -262,8 +262,9 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
 {
     struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
     struct page_info *page;
+    mfn_t mfn;
 
-    if ( iorp->page )
+    if ( iorp->va )
     {
         /*
          * If a guest frame has already been mapped (which may happen
@@ -291,11 +292,11 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
         return -ENODATA;
     }
 
-    iorp->va = __map_domain_page_global(page);
+    mfn = page_to_mfn(page);
+    iorp->va = vmap(&mfn, 1);
     if ( !iorp->va )
         goto fail;
 
-    iorp->page = page;
     clear_page(iorp->va);
     return 0;
 
@@ -309,15 +310,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
 static void ioreq_server_free_mfn(struct ioreq_server *s, bool buf)
 {
     struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
-    struct page_info *page = iorp->page;
+    struct page_info *page;
+    void *va;
 
-    if ( !page )
+    if ( !iorp->va )
         return;
 
-    iorp->page = NULL;
-
-    unmap_domain_page_global(iorp->va);
+    va = iorp->va;
+    page = vmap_to_page(va);
     iorp->va = NULL;
+    vunmap(va);
 
     put_page_alloc_ref(page);
     put_page_and_type(page);
@@ -333,7 +335,8 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
 
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
-        if ( (s->ioreq.page == page) || (s->bufioreq.page == page) )
+        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
+             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
         {
             found = true;
             break;
@@ -627,10 +630,9 @@ static void ioreq_server_deinit(struct ioreq_server *s)
      * NOTE: It is safe to call both arch_ioreq_server_unmap_pages() and
      *       ioreq_server_free_pages() in that order.
      *       This is because the former will do nothing if the pages
-     *       are not mapped, leaving the page to be freed by the latter.
-     *       However if the pages are mapped then the former will set
-     *       the page_info pointer to NULL, meaning the latter will do
-     *       nothing.
+     *       are not mapped, leaving the pages to be freed by the latter.
+     *       However if the pages are mapped then the former will clear
+     *       iorp->va, meaning the latter will do nothing.
      */
     arch_ioreq_server_unmap_pages(s);
     ioreq_server_free_pages(s);
@@ -819,12 +821,12 @@ int ioreq_server_get_frame(struct domain *d, ioservid_t id,
         if ( !HANDLE_BUFIOREQ(s) )
             goto out;
 
-        *mfn = page_to_mfn(s->bufioreq.page);
+        *mfn = vmap_to_mfn(s->bufioreq.va);
         rc = 0;
         break;
 
     case XENMEM_resource_ioreq_server_frame_ioreq(0):
-        *mfn = page_to_mfn(s->ioreq.page);
+        *mfn = vmap_to_mfn(s->ioreq.va);
         rc = 0;
         break;
 
diff --git a/xen/include/xen/ioreq.h b/xen/include/xen/ioreq.h
index e86f0869fa..d63fa4729e 100644
--- a/xen/include/xen/ioreq.h
+++ b/xen/include/xen/ioreq.h
@@ -25,7 +25,6 @@
 
 struct ioreq_page {
     gfn_t gfn;
-    struct page_info *page;
     void *va;
 };
 
-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v6 1/3] ioreq: switch ioreq page allocation to vmap

[Jan Beulich]

On 20.04.2026 11:38, Julian Vetter wrote:
> Switch the Xen-side ioreq page mapping from prepare_ring_for_helper() /
> map_domain_page_global() to explicit vmap(), to ensure vmap_to_page()
> can recover the struct page_info * uniformly during teardown.

What's after the comma isn't really the main reason for this patch, is it?
Describing this aspect ...

> This is a prerequisite for multi-page ioreq support: the non-buf ioreq
> region will need to span multiple pages for domains with more vCPUs than
> fit in a single page, and vmap() is the natural interface for contiguous
> multi-page Xen VA mappings.
> 
> In non-debug builds map_domain_page_global() uses the directmap for low
> MFNs rather than vmap(), so this change has a small overhead in the
> common case. Debug builds already used vmap() indirectly.
> 
> With both paths using vmap(), vmap_to_page() can recover the struct
> page_info * uniformly, so drop the 'page' field from struct ioreq_page
> and update all callers accordingly.

... here (at the bottom) is fully sufficient.

> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
> ---
> Changes in v6:
> - Updated commit message to clearly specify why these changes are made
> - Added comment to say that this is {prepare,destroy}_ring_for_helper()
>   just using vmap_to_page() + v{map,unmap}()
> - Kept proper ordering in ioreq_server_free_mfn(), first clearing the va
>   pointer before unmapping

Yet then you didn't extend the same consideration ...

> @@ -128,8 +129,13 @@ static void hvm_unmap_ioreq_gfn(struct ioreq_server *s, bool buf)
>      if ( gfn_eq(iorp->gfn, INVALID_GFN) )
>          return;
>  
> -    destroy_ring_for_helper(&iorp->va, iorp->page);
> -    iorp->page = NULL;
> +    /* Equivalent to destroy_ring_for_helper(), using vmap_to_page(). */
> +    if ( iorp->va )
> +    {
> +        put_page_and_type(vmap_to_page(iorp->va));
> +        vunmap(iorp->va);
> +        iorp->va = NULL;
> +    }

... to here. (Really we should perhaps introduce VUNMAP(), much like we
have XFREE(), XVFREE(), etc.)

Further the ordering doesn't match destroy_ring_for_helper(), which unmaps
first and only then drops the page refs.

> @@ -162,12 +171,40 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
>      if ( gfn_eq(iorp->gfn, INVALID_GFN) )
>          return -ENOMEM;
>  
> -    rc = prepare_ring_for_helper(d, gfn_x(iorp->gfn), &iorp->page,
> -                                 &iorp->va);
> -
> +    /*
> +     * Equivalent to prepare_ring_for_helper() using vmap(). Using vmap()
> +     * rather than map_domain_page_global() ensures vmap_to_page() can
> +     * recover the struct page_info * uniformly at teardown, which is
> +     * needed to support multi-page ioreq mappings (see nr_ioreq_pages()).
> +     */

"is needed" is too strong, I think - surely there would be a way to handle
that without vmap_to_page(), by tracking all struct page_info * separately.

> @@ -309,15 +310,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>  static void ioreq_server_free_mfn(struct ioreq_server *s, bool buf)
>  {
>      struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
> -    struct page_info *page = iorp->page;
> +    struct page_info *page;
> +    void *va;
>  
> -    if ( !page )
> +    if ( !iorp->va )
>          return;
>  
> -    iorp->page = NULL;
> -
> -    unmap_domain_page_global(iorp->va);
> +    va = iorp->va;

Please can this be the initializer of the variable, for the if() above to
then use that local var?

> @@ -333,7 +335,8 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
>  
>      FOR_EACH_IOREQ_SERVER(d, id, s)
>      {
> -        if ( (s->ioreq.page == page) || (s->bufioreq.page == page) )
> +        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
> +             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
>          {
>              found = true;
>              break;

You mention in the description that some extra overhead is introduced. The
(generally) two page walks done here are particularly concerning. Since we
have a valid struct page_info * available here, I wonder if we shouldn't
aid this lookup by recording the VA in one of struct page_info's fields.
Afaics vmap() doesn't use any of the fields, so it should be relatively
easy to determine a field to use for this purpose. The more involved part
would then be to make sure the field (in other struct page_info instances)
is also properly different from any VA vmap() may return.

Jan

Re: [PATCH v6 1/3] ioreq: switch ioreq page allocation to vmap

[Julian Vetter]

[-- Attachment #1: Type: text/plain, Size: 6163 bytes --]



On 8/18/26 3:06 PM, Jan Beulich wrote:
> On 20.04.2026 11:38, Julian Vetter wrote:
>> Switch the Xen-side ioreq page mapping from prepare_ring_for_helper() /
>> map_domain_page_global() to explicit vmap(), to ensure vmap_to_page()
>> can recover the struct page_info * uniformly during teardown.
> 
> What's after the comma isn't really the main reason for this patch, is it?
> Describing this aspect ...
> 
>> This is a prerequisite for multi-page ioreq support: the non-buf ioreq
>> region will need to span multiple pages for domains with more vCPUs than
>> fit in a single page, and vmap() is the natural interface for contiguous
>> multi-page Xen VA mappings.
>>
>> In non-debug builds map_domain_page_global() uses the directmap for low
>> MFNs rather than vmap(), so this change has a small overhead in the
>> common case. Debug builds already used vmap() indirectly.
>>
>> With both paths using vmap(), vmap_to_page() can recover the struct
>> page_info * uniformly, so drop the 'page' field from struct ioreq_page
>> and update all callers accordingly.
> 
> ... here (at the bottom) is fully sufficient.
> 
>> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
>> ---
>> Changes in v6:
>> - Updated commit message to clearly specify why these changes are made
>> - Added comment to say that this is {prepare,destroy}_ring_for_helper()
>>    just using vmap_to_page() + v{map,unmap}()
>> - Kept proper ordering in ioreq_server_free_mfn(), first clearing the va
>>    pointer before unmapping
> 
> Yet then you didn't extend the same consideration ...
> 
>> @@ -128,8 +129,13 @@ static void hvm_unmap_ioreq_gfn(struct ioreq_server *s, bool buf)
>>       if ( gfn_eq(iorp->gfn, INVALID_GFN) )
>>           return;
>>   
>> -    destroy_ring_for_helper(&iorp->va, iorp->page);
>> -    iorp->page = NULL;
>> +    /* Equivalent to destroy_ring_for_helper(), using vmap_to_page(). */
>> +    if ( iorp->va )
>> +    {
>> +        put_page_and_type(vmap_to_page(iorp->va));
>> +        vunmap(iorp->va);
>> +        iorp->va = NULL;
>> +    }
> 
> ... to here. (Really we should perhaps introduce VUNMAP(), much like we
> have XFREE(), XVFREE(), etc.)
> 
> Further the ordering doesn't match destroy_ring_for_helper(), which unmaps
> first and only then drops the page refs.
> 
>> @@ -162,12 +171,40 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
>>       if ( gfn_eq(iorp->gfn, INVALID_GFN) )
>>           return -ENOMEM;
>>   
>> -    rc = prepare_ring_for_helper(d, gfn_x(iorp->gfn), &iorp->page,
>> -                                 &iorp->va);
>> -
>> +    /*
>> +     * Equivalent to prepare_ring_for_helper() using vmap(). Using vmap()
>> +     * rather than map_domain_page_global() ensures vmap_to_page() can
>> +     * recover the struct page_info * uniformly at teardown, which is
>> +     * needed to support multi-page ioreq mappings (see nr_ioreq_pages()).
>> +     */
> 
> "is needed" is too strong, I think - surely there would be a way to handle
> that without vmap_to_page(), by tracking all struct page_info * separately.
> 
>> @@ -309,15 +310,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>>   static void ioreq_server_free_mfn(struct ioreq_server *s, bool buf)
>>   {
>>       struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
>> -    struct page_info *page = iorp->page;
>> +    struct page_info *page;
>> +    void *va;
>>   
>> -    if ( !page )
>> +    if ( !iorp->va )
>>           return;
>>   
>> -    iorp->page = NULL;
>> -
>> -    unmap_domain_page_global(iorp->va);
>> +    va = iorp->va;
> 
> Please can this be the initializer of the variable, for the if() above to
> then use that local var?
> 
>> @@ -333,7 +335,8 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
>>   
>>       FOR_EACH_IOREQ_SERVER(d, id, s)
>>       {
>> -        if ( (s->ioreq.page == page) || (s->bufioreq.page == page) )
>> +        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
>> +             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
>>           {
>>               found = true;
>>               break;
> 
> You mention in the description that some extra overhead is introduced. The
> (generally) two page walks done here are particularly concerning. Since we
> have a valid struct page_info * available here, I wonder if we shouldn't
> aid this lookup by recording the VA in one of struct page_info's fields.
> Afaics vmap() doesn't use any of the fields, so it should be relatively
> easy to determine a field to use for this purpose. The more involved part
> would then be to make sure the field (in other struct page_info instances)
> is also properly different from any VA vmap() may return.

Hello Jan,

Thank you again for your feedback! I will wait then for Anthony's 
decision regarding whether the multi-page ioreq support and the ioreq_t 
growth should be combined into a single effort, before I proceed further 
with a v7.

I just wanted to clarify one thing regarding the overhead I mentioned in 
the first patch's commit message ("this change has a small overhead in 
the common case"). Here, I was referring to vmap()/vunmap() replacing 
map_domain_page_global(). Where map_domain_page_global() has a directmap 
fast path. I didn't mean the overhead of the added vmap_to_page(). I 
should maybe clarify this better in my next iteration's commit message.

On your suggestion to cache the VA in struct page_info to speed up the 
vmap_to_page() lookups in is_ioreq_server_page(): I looked through the 
tree, and that function currently has only one caller 
sh_remove_all_mappings() (in xen/arch/x86/mm/shadow/common.c) and is 
only reached in a failure case. Given that, and given how widely shared 
and size-critical struct page_info is, I'm wondering whether it's really 
worth touching it for the gain of not having to do the 2 lookups. What 
do you think?

Thanks,
Julian

>
> Jan



--
 | Vates

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v6 1/3] ioreq: switch ioreq page allocation to vmap

[Jan Beulich]

On 25.08.2026 16:19, Julian Vetter wrote:
> On 8/18/26 3:06 PM, Jan Beulich wrote:
>> On 20.04.2026 11:38, Julian Vetter wrote:
>>> @@ -333,7 +335,8 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
>>>   
>>>       FOR_EACH_IOREQ_SERVER(d, id, s)
>>>       {
>>> -        if ( (s->ioreq.page == page) || (s->bufioreq.page == page) )
>>> +        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
>>> +             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
>>>           {
>>>               found = true;
>>>               break;
>>
>> You mention in the description that some extra overhead is introduced. The
>> (generally) two page walks done here are particularly concerning. Since we
>> have a valid struct page_info * available here, I wonder if we shouldn't
>> aid this lookup by recording the VA in one of struct page_info's fields.
>> Afaics vmap() doesn't use any of the fields, so it should be relatively
>> easy to determine a field to use for this purpose. The more involved part
>> would then be to make sure the field (in other struct page_info instances)
>> is also properly different from any VA vmap() may return.
> 
> Hello Jan,
> 
> Thank you again for your feedback! I will wait then for Anthony's 
> decision regarding whether the multi-page ioreq support and the ioreq_t 
> growth should be combined into a single effort, before I proceed further 
> with a v7.
> 
> I just wanted to clarify one thing regarding the overhead I mentioned in 
> the first patch's commit message ("this change has a small overhead in 
> the common case"). Here, I was referring to vmap()/vunmap() replacing 
> map_domain_page_global(). Where map_domain_page_global() has a directmap 
> fast path. I didn't mean the overhead of the added vmap_to_page(). I 
> should maybe clarify this better in my next iteration's commit message.
> 
> On your suggestion to cache the VA in struct page_info to speed up the 
> vmap_to_page() lookups in is_ioreq_server_page(): I looked through the 
> tree, and that function currently has only one caller 
> sh_remove_all_mappings() (in xen/arch/x86/mm/shadow/common.c) and is 
> only reached in a failure case. Given that, and given how widely shared 
> and size-critical struct page_info is, I'm wondering whether it's really 
> worth touching it for the gain of not having to do the 2 lookups. What 
> do you think?

Hmm, indeed. Yet how would we prevent new uses from being hit? At least
a comment may want adding somewhere (where it's not too easy to overlook).

That said, why the mention of "size-critical" when I said "determine a
field", not "add a field"? (Really in different context I've suggested
the same as a possibility to George, for his ASI work.)

Jan

Re: [PATCH v6 1/3] ioreq: switch ioreq page allocation to vmap

[George Dunlap]

On Tue, Aug 25, 2026 at 3:20 PM Julian Vetter <julian.vetter@vates.tech> wrote:
> Thank you again for your feedback! I will wait then for Anthony's
> decision regarding whether the multi-page ioreq support and the ioreq_t
> growth should be combined into a single effort, before I proceed further
> with a v7.

BTW, if we're considering modifying the ioreq server protocol, I have
a few requests to consider from an ASI perspective.

Basically, at the moment, ioreq vcpu rings are grouped by backend; so
each vcpu shares a ring with all other vcpus; meaning that, in theory,
one vcpu could read the payload of another vcpu's IO operations.  This
isn't critical, but the more isolation the better.

What would be more convenient from an ASI perspective would be to have
per-vcpu data shared on separate pages.  One design would be to have a
single ioreq page per vcpu, with all ioreq server rings on the single
page.  What would perhaps be nicer long-term is to figure out a way
for all per-vcpu shared structures to share a single page (or set of
pages).

Not sure how this fits with the ioreq_t growth or multi-page ioreq
support, but thought it would be worthwhile to toss out there for
consideration:  At least to make ioreq page isolation less difficult,
and perhaps to make it easier.

 -George

[PATCH v6 2/3] ioreq: Indent ioreq_server_alloc_mfn() body one level deeper

[Julian Vetter]

No functional change. It adds a wrapping block to prepare for the loop
that the subsequent patch introduces to handle multiple ioreq pages.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
Changes in v6:
- Dropped the indentation change for ioreq_server_free_mfn, because the
  modifications in the next patch don't really merit the change anymore
---
 xen/common/ioreq.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c
index d8d02167b4..bae9b99c99 100644
--- a/xen/common/ioreq.c
+++ b/xen/common/ioreq.c
@@ -277,22 +277,24 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
         return 0;
     }
 
-    page = alloc_domheap_page(s->target, MEMF_no_refcount);
+    {
+        page = alloc_domheap_page(s->target, MEMF_no_refcount);
 
-    if ( !page )
-        return -ENOMEM;
+        if ( !page )
+            return -ENOMEM;
 
-    if ( !get_page_and_type(page, s->target, PGT_writable_page) )
-    {
-        /*
-         * The domain can't possibly know about this page yet, so failure
-         * here is a clear indication of something fishy going on.
-         */
-        domain_crash(s->emulator);
-        return -ENODATA;
-    }
+        if ( !get_page_and_type(page, s->target, PGT_writable_page) )
+        {
+            /*
+             * The domain can't possibly know about this page yet, so failure
+             * here is a clear indication of something fishy going on.
+             */
+            domain_crash(s->emulator);
+            return -ENODATA;
+        }
 
-    mfn = page_to_mfn(page);
+        mfn = page_to_mfn(page);
+    }
     iorp->va = vmap(&mfn, 1);
     if ( !iorp->va )
         goto fail;
-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v6 2/3] ioreq: Indent ioreq_server_alloc_mfn() body one level deeper

[Jan Beulich]

On 20.04.2026 11:38, Julian Vetter wrote:
> --- a/xen/common/ioreq.c
> +++ b/xen/common/ioreq.c
> @@ -277,22 +277,24 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>          return 0;
>      }
>  
> -    page = alloc_domheap_page(s->target, MEMF_no_refcount);
> +    {
> +        page = alloc_domheap_page(s->target, MEMF_no_refcount);
>  
> -    if ( !page )
> -        return -ENOMEM;
> +        if ( !page )
> +            return -ENOMEM;
>  
> -    if ( !get_page_and_type(page, s->target, PGT_writable_page) )
> -    {
> -        /*
> -         * The domain can't possibly know about this page yet, so failure
> -         * here is a clear indication of something fishy going on.
> -         */
> -        domain_crash(s->emulator);
> -        return -ENODATA;
> -    }
> +        if ( !get_page_and_type(page, s->target, PGT_writable_page) )
> +        {
> +            /*
> +             * The domain can't possibly know about this page yet, so failure
> +             * here is a clear indication of something fishy going on.
> +             */
> +            domain_crash(s->emulator);
> +            return -ENODATA;
> +        }
>  
> -    mfn = page_to_mfn(page);
> +        mfn = page_to_mfn(page);
> +    }
>      iorp->va = vmap(&mfn, 1);
>      if ( !iorp->va )
>          goto fail;

Please would you then also add another blank line after the new closing curly
brace? (Based on the corresponding ioreq_server_free_mfn() change having been
dropped, I'd like to wait with ack-ing until the 3rd patch is in final shape.)

Jan

[PATCH v6 3/3] x86/ioreq: Extend ioreq server to support multiple ioreq pages

[Julian Vetter]

As the number of vCPUs grows, a single ioreq page of 128 slots may not
be sufficient. Add support for allocating and mapping multiple ioreq
pages so that the ioreq region can scale with d->max_vcpus.

Introduce nr_ioreq_pages() to compute the number of pages required for
a given domain, and IOREQ_NR_PAGES_MAX as a compile-time upper bound
(based on HVM_MAX_VCPUS).

ioreq_server_alloc_mfn() is updated to allocate nr_ioreq_pages() pages
and map them contiguously via vmap().

is_ioreq_server_page() iterates over all ioreq pages when checking
page ownership. ioreq_server_get_frame() allows callers to retrieve any
ioreq page by index via the XENMEM_acquire_resource interface.

On x86, the legacy GFN mapping path (hvm_map_ioreq_gfn) is limited to
a single ioreq page; device models requiring more ioreq slots must use
the resource mapping interface (XENMEM_acquire_resource).

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
Changes in v6:
- Adapted the comment to not mention the guest, but the device model
- Replaced the dynamic allocation for the mfns array by a static array
- Fixed error handling in ioreq_server_alloc_mfn, using an extra
  nr_alloc variable to track the already allocated pages
- Dropped unnecessary void casts
---
 xen/arch/x86/hvm/ioreq.c |  8 ++++
 xen/common/ioreq.c       | 93 ++++++++++++++++++++++++++++------------
 xen/include/xen/ioreq.h  | 12 ++++++
 3 files changed, 86 insertions(+), 27 deletions(-)

diff --git a/xen/arch/x86/hvm/ioreq.c b/xen/arch/x86/hvm/ioreq.c
index 3cabec141c..ee679bdf5a 100644
--- a/xen/arch/x86/hvm/ioreq.c
+++ b/xen/arch/x86/hvm/ioreq.c
@@ -166,6 +166,14 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
     if ( d->is_dying )
         return -EINVAL;
 
+    /*
+     * The legacy GFN path supports only a single ioreq page. Device models
+     * requiring more ioreq slots must use the resource mapping interface
+     * (XENMEM_acquire_resource).
+     */
+    if ( !buf && nr_ioreq_pages(d) > 1 )
+        return -EOPNOTSUPP;
+
     iorp->gfn = hvm_alloc_ioreq_gfn(s);
 
     if ( gfn_eq(iorp->gfn, INVALID_GFN) )
diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c
index bae9b99c99..3a08e77597 100644
--- a/xen/common/ioreq.c
+++ b/xen/common/ioreq.c
@@ -261,8 +261,11 @@ bool vcpu_ioreq_handle_completion(struct vcpu *v)
 static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
 {
     struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
-    struct page_info *page;
-    mfn_t mfn;
+    unsigned int i, nr_alloc = 0, nr_pages = buf ? 1 : nr_ioreq_pages(s->target);
+    mfn_t mfns[IOREQ_NR_PAGES_MAX] = {};
+    int rc;
+
+    ASSERT(nr_pages <= IOREQ_NR_PAGES_MAX);
 
     if ( iorp->va )
     {
@@ -277,11 +280,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
         return 0;
     }
 
+    for ( i = 0; i < nr_pages; i++ )
     {
-        page = alloc_domheap_page(s->target, MEMF_no_refcount);
+        struct page_info *page = alloc_domheap_page(s->target,
+                                                    MEMF_no_refcount);
 
         if ( !page )
-            return -ENOMEM;
+        {
+            rc = -ENOMEM;
+            goto fail;
+        }
 
         if ( !get_page_and_type(page, s->target, PGT_writable_page) )
         {
@@ -290,41 +298,59 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
              * here is a clear indication of something fishy going on.
              */
             domain_crash(s->emulator);
-            return -ENODATA;
+            rc = -ENODATA;
+            goto fail;
         }
 
-        mfn = page_to_mfn(page);
+        mfns[nr_alloc++] = page_to_mfn(page);
     }
-    iorp->va = vmap(&mfn, 1);
+
+    iorp->va = vmap(mfns, nr_pages);
     if ( !iorp->va )
+    {
+        rc = -ENOMEM;
         goto fail;
+    }
 
-    clear_page(iorp->va);
+    memset(iorp->va, 0, nr_pages * PAGE_SIZE);
     return 0;
 
  fail:
-    put_page_alloc_ref(page);
-    put_page_and_type(page);
+    while ( nr_alloc-- )
+    {
+        struct page_info *page = mfn_to_page(mfns[nr_alloc]);
+
+        put_page_alloc_ref(page);
+        put_page_and_type(page);
+    }
 
-    return -ENOMEM;
+    return rc;
 }
 
 static void ioreq_server_free_mfn(struct ioreq_server *s, bool buf)
 {
     struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
-    struct page_info *page;
+    unsigned int i, nr_pages = buf ? 1 : nr_ioreq_pages(s->target);
+    struct page_info *pages[IOREQ_NR_PAGES_MAX];
     void *va;
 
     if ( !iorp->va )
         return;
 
+    ASSERT(nr_pages <= IOREQ_NR_PAGES_MAX);
+
+    for ( i = 0; i < nr_pages; i++ )
+        pages[i] = vmap_to_page(iorp->va + i * PAGE_SIZE);
+
     va = iorp->va;
-    page = vmap_to_page(va);
     iorp->va = NULL;
     vunmap(va);
 
-    put_page_alloc_ref(page);
-    put_page_and_type(page);
+    for ( i = 0; i < nr_pages; i++ )
+    {
+        put_page_alloc_ref(pages[i]);
+        put_page_and_type(pages[i]);
+    }
 }
 
 bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
@@ -337,12 +363,25 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
 
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
-        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
-             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
+        unsigned int i;
+
+        if ( s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page )
         {
             found = true;
             break;
         }
+
+        for ( i = 0; i < nr_ioreq_pages(d) && s->ioreq.va; i++ )
+        {
+            if ( vmap_to_page(s->ioreq.va + i * PAGE_SIZE) == page )
+            {
+                found = true;
+                break;
+            }
+        }
+
+        if ( found )
+            break;
     }
 
     rspin_unlock(&d->ioreq_server.lock);
@@ -816,26 +855,26 @@ int ioreq_server_get_frame(struct domain *d, ioservid_t id,
     if ( rc )
         goto out;
 
-    switch ( idx )
+    if ( idx == XENMEM_resource_ioreq_server_frame_bufioreq )
     {
-    case XENMEM_resource_ioreq_server_frame_bufioreq:
         rc = -ENOENT;
         if ( !HANDLE_BUFIOREQ(s) )
             goto out;
 
         *mfn = vmap_to_mfn(s->bufioreq.va);
         rc = 0;
-        break;
+    }
+    else if ( idx >= XENMEM_resource_ioreq_server_frame_ioreq(0) &&
+              idx < XENMEM_resource_ioreq_server_frame_ioreq(nr_ioreq_pages(d)) )
+    {
+        unsigned int page_idx = idx - XENMEM_resource_ioreq_server_frame_ioreq(0);
 
-    case XENMEM_resource_ioreq_server_frame_ioreq(0):
-        *mfn = vmap_to_mfn(s->ioreq.va);
+        ASSERT(page_idx < nr_ioreq_pages(d));
+        *mfn = vmap_to_mfn(s->ioreq.va + page_idx * PAGE_SIZE);
         rc = 0;
-        break;
-
-    default:
-        rc = -EINVAL;
-        break;
     }
+    else
+        rc = -EINVAL;
 
  out:
     rspin_unlock(&d->ioreq_server.lock);
diff --git a/xen/include/xen/ioreq.h b/xen/include/xen/ioreq.h
index d63fa4729e..d2a08c2371 100644
--- a/xen/include/xen/ioreq.h
+++ b/xen/include/xen/ioreq.h
@@ -35,6 +35,18 @@ struct ioreq_vcpu {
     bool             pending;
 };
 
+/*
+ * Maximum number of ioreq pages, based on the maximum number
+ * of vCPUs and the number of ioreq slots per page.
+ */
+#define IOREQ_NR_PAGES_MAX \
+    DIV_ROUND_UP(HVM_MAX_VCPUS, PAGE_SIZE / sizeof(ioreq_t))
+
+static inline unsigned int nr_ioreq_pages(const struct domain *d)
+{
+    return DIV_ROUND_UP(d->max_vcpus, PAGE_SIZE / sizeof(ioreq_t));
+}
+
 #define NR_IO_RANGE_TYPES (XEN_DMOP_IO_RANGE_PCI + 1)
 #define MAX_NR_IO_RANGES  256
 
-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v6 3/3] x86/ioreq: Extend ioreq server to support multiple ioreq pages

[Teddy Astie]

Le 20/04/2026 à 11:41, Julian Vetter a écrit :
> As the number of vCPUs grows, a single ioreq page of 128 slots may not
> be sufficient. Add support for allocating and mapping multiple ioreq
> pages so that the ioreq region can scale with d->max_vcpus.
>
> Introduce nr_ioreq_pages() to compute the number of pages required for
> a given domain, and IOREQ_NR_PAGES_MAX as a compile-time upper bound
> (based on HVM_MAX_VCPUS).
>
> ioreq_server_alloc_mfn() is updated to allocate nr_ioreq_pages() pages
> and map them contiguously via vmap().
>
> is_ioreq_server_page() iterates over all ioreq pages when checking
> page ownership. ioreq_server_get_frame() allows callers to retrieve any
> ioreq page by index via the XENMEM_acquire_resource interface.
>
> On x86, the legacy GFN mapping path (hvm_map_ioreq_gfn) is limited to
> a single ioreq page; device models requiring more ioreq slots must use
> the resource mapping interface (XENMEM_acquire_resource).
>
> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
> ---
> Changes in v6:
> - Adapted the comment to not mention the guest, but the device model
> - Replaced the dynamic allocation for the mfns array by a static array
> - Fixed error handling in ioreq_server_alloc_mfn, using an extra
>    nr_alloc variable to track the already allocated pages
> - Dropped unnecessary void casts
> ---
>   xen/arch/x86/hvm/ioreq.c |  8 ++++
>   xen/common/ioreq.c       | 93 ++++++++++++++++++++++++++++------------
>   xen/include/xen/ioreq.h  | 12 ++++++
>   3 files changed, 86 insertions(+), 27 deletions(-)
>
> diff --git a/xen/arch/x86/hvm/ioreq.c b/xen/arch/x86/hvm/ioreq.c
> index 3cabec141c..ee679bdf5a 100644
> --- a/xen/arch/x86/hvm/ioreq.c
> +++ b/xen/arch/x86/hvm/ioreq.c
> @@ -166,6 +166,14 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
>       if ( d->is_dying )
>           return -EINVAL;
>
> +    /*
> +     * The legacy GFN path supports only a single ioreq page. Device models
> +     * requiring more ioreq slots must use the resource mapping interface
> +     * (XENMEM_acquire_resource).
> +     */
> +    if ( !buf && nr_ioreq_pages(d) > 1 )
> +        return -EOPNOTSUPP;
> +
>       iorp->gfn = hvm_alloc_ioreq_gfn(s);
>
>       if ( gfn_eq(iorp->gfn, INVALID_GFN) )
> diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c
> index bae9b99c99..3a08e77597 100644
> --- a/xen/common/ioreq.c
> +++ b/xen/common/ioreq.c
> @@ -261,8 +261,11 @@ bool vcpu_ioreq_handle_completion(struct vcpu *v)
>   static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>   {
>       struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
> -    struct page_info *page;
> -    mfn_t mfn;
> +    unsigned int i, nr_alloc = 0, nr_pages = buf ? 1 : nr_ioreq_pages(s->target);
> +    mfn_t mfns[IOREQ_NR_PAGES_MAX] = {};
> +    int rc;
> +
> +    ASSERT(nr_pages <= IOREQ_NR_PAGES_MAX);
>
>       if ( iorp->va )
>       {
> @@ -277,11 +280,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>           return 0;
>       }
>
> +    for ( i = 0; i < nr_pages; i++ )
>       {
> -        page = alloc_domheap_page(s->target, MEMF_no_refcount);
> +        struct page_info *page = alloc_domheap_page(s->target,
> +                                                    MEMF_no_refcount);
>
>           if ( !page )
> -            return -ENOMEM;
> +        {
> +            rc = -ENOMEM;
> +            goto fail;
> +        }
>
>           if ( !get_page_and_type(page, s->target, PGT_writable_page) )
>           {
> @@ -290,41 +298,59 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>                * here is a clear indication of something fishy going on.
>                */
>               domain_crash(s->emulator);
> -            return -ENODATA;
> +            rc = -ENODATA;
> +            goto fail;
>           }
>
> -        mfn = page_to_mfn(page);
> +        mfns[nr_alloc++] = page_to_mfn(page);
>       }
> -    iorp->va = vmap(&mfn, 1);
> +
> +    iorp->va = vmap(mfns, nr_pages);
>       if ( !iorp->va )
> +    {
> +        rc = -ENOMEM;
>           goto fail;
> +    }
>
> -    clear_page(iorp->va);
> +    memset(iorp->va, 0, nr_pages * PAGE_SIZE);
>       return 0;
>
>    fail:
> -    put_page_alloc_ref(page);
> -    put_page_and_type(page);
> +    while ( nr_alloc-- )
> +    {
> +        struct page_info *page = mfn_to_page(mfns[nr_alloc]);
> +
> +        put_page_alloc_ref(page);
> +        put_page_and_type(page);
> +    }
>
> -    return -ENOMEM;
> +    return rc;
>   }
>
>   static void ioreq_server_free_mfn(struct ioreq_server *s, bool buf)
>   {
>       struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
> -    struct page_info *page;
> +    unsigned int i, nr_pages = buf ? 1 : nr_ioreq_pages(s->target);
> +    struct page_info *pages[IOREQ_NR_PAGES_MAX];
>       void *va;
>
>       if ( !iorp->va )
>           return;
>
> +    ASSERT(nr_pages <= IOREQ_NR_PAGES_MAX);
> +
> +    for ( i = 0; i < nr_pages; i++ )
> +        pages[i] = vmap_to_page(iorp->va + i * PAGE_SIZE);
> +
>       va = iorp->va;
> -    page = vmap_to_page(va);
>       iorp->va = NULL;
>       vunmap(va);
>
> -    put_page_alloc_ref(page);
> -    put_page_and_type(page);
> +    for ( i = 0; i < nr_pages; i++ )
> +    {
> +        put_page_alloc_ref(pages[i]);
> +        put_page_and_type(pages[i]);
> +    }
>   }
>
>   bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
> @@ -337,12 +363,25 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
>
>       FOR_EACH_IOREQ_SERVER(d, id, s)
>       {
> -        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
> -             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
> +        unsigned int i;
> +
> +        if ( s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page )
>           {
>               found = true;
>               break;
>           }
> +
> +        for ( i = 0; i < nr_ioreq_pages(d) && s->ioreq.va; i++ )
> +        {
> +            if ( vmap_to_page(s->ioreq.va + i * PAGE_SIZE) == page )
> +            {
> +                found = true;
> +                break;
> +            }
> +        }
> +
> +        if ( found )
> +            break;
>       }
>
>       rspin_unlock(&d->ioreq_server.lock);
> @@ -816,26 +855,26 @@ int ioreq_server_get_frame(struct domain *d, ioservid_t id,
>       if ( rc )
>           goto out;
>
> -    switch ( idx )
> +    if ( idx == XENMEM_resource_ioreq_server_frame_bufioreq )
>       {
> -    case XENMEM_resource_ioreq_server_frame_bufioreq:
>           rc = -ENOENT;
>           if ( !HANDLE_BUFIOREQ(s) )
>               goto out;
>
>           *mfn = vmap_to_mfn(s->bufioreq.va);
>           rc = 0;
> -        break;
> +    }
> +    else if ( idx >= XENMEM_resource_ioreq_server_frame_ioreq(0) &&
> +              idx < XENMEM_resource_ioreq_server_frame_ioreq(nr_ioreq_pages(d)) )
> +    {
> +        unsigned int page_idx = idx - XENMEM_resource_ioreq_server_frame_ioreq(0);
>
> -    case XENMEM_resource_ioreq_server_frame_ioreq(0):
> -        *mfn = vmap_to_mfn(s->ioreq.va);
> +        ASSERT(page_idx < nr_ioreq_pages(d));
> +        *mfn = vmap_to_mfn(s->ioreq.va + page_idx * PAGE_SIZE);
>           rc = 0;
> -        break;
> -
> -    default:
> -        rc = -EINVAL;
> -        break;
>       }
> +    else
> +        rc = -EINVAL;
>
>    out:
>       rspin_unlock(&d->ioreq_server.lock);
> diff --git a/xen/include/xen/ioreq.h b/xen/include/xen/ioreq.h
> index d63fa4729e..d2a08c2371 100644
> --- a/xen/include/xen/ioreq.h
> +++ b/xen/include/xen/ioreq.h
> @@ -35,6 +35,18 @@ struct ioreq_vcpu {
>       bool             pending;
>   };
>
> +/*
> + * Maximum number of ioreq pages, based on the maximum number
> + * of vCPUs and the number of ioreq slots per page.
> + */
> +#define IOREQ_NR_PAGES_MAX \
> +    DIV_ROUND_UP(HVM_MAX_VCPUS, PAGE_SIZE / sizeof(ioreq_t))
> +
> +static inline unsigned int nr_ioreq_pages(const struct domain *d)
> +{
> +    return DIV_ROUND_UP(d->max_vcpus, PAGE_SIZE / sizeof(ioreq_t));
> +}
> +
>   #define NR_IO_RANGE_TYPES (XEN_DMOP_IO_RANGE_PCI + 1)
>   #define MAX_NR_IO_RANGES  256
>

Is there anything that would prevent the use of alloc_domheap_pages() to
allocate a configuous set of pages at once; and vmap_contig() to map it
in one go.
That also prevents the ioreq pages from being scattered around.

We would lose a few pages by aligning the size into a order, but that
probably better than the alternatives.

That way, we would only have to keep the base gfn (or first page_info)
and size of the allocation, and don't have to use a array of mfn nor
have to reverse the vmap to track it.

Teddy


--
Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v6 3/3] x86/ioreq: Extend ioreq server to support multiple ioreq pages

[Jan Beulich]

On 20.04.2026 14:49, Teddy Astie wrote:
> Is there anything that would prevent the use of alloc_domheap_pages() to 
> allocate a configuous set of pages at once; and vmap_contig() to map it 
> in one go.
> That also prevents the ioreq pages from being scattered around.
> 
> We would lose a few pages by aligning the size into a order, but that 
> probably better than the alternatives.
> 
> That way, we would only have to keep the base gfn (or first page_info) 
> and size of the allocation, and don't have to use a array of mfn nor 
> have to reverse the vmap to track it.

Well, higher-order allocations can easily fail when there's ample memory
available. Hence why a goal ought to be to avoid such allocations at
runtime. That's why we (now) have vmalloc() and xvmalloc().

Jan

Re: [PATCH v6 3/3] x86/ioreq: Extend ioreq server to support multiple ioreq pages

[Jan Beulich]

On 20.04.2026 11:38, Julian Vetter wrote:
> As the number of vCPUs grows, a single ioreq page of 128 slots may not
> be sufficient. Add support for allocating and mapping multiple ioreq
> pages so that the ioreq region can scale with d->max_vcpus.
> 
> Introduce nr_ioreq_pages() to compute the number of pages required for
> a given domain, and IOREQ_NR_PAGES_MAX as a compile-time upper bound
> (based on HVM_MAX_VCPUS).
> 
> ioreq_server_alloc_mfn() is updated to allocate nr_ioreq_pages() pages
> and map them contiguously via vmap().
> 
> is_ioreq_server_page() iterates over all ioreq pages when checking
> page ownership. ioreq_server_get_frame() allows callers to retrieve any
> ioreq page by index via the XENMEM_acquire_resource interface.
> 
> On x86, the legacy GFN mapping path (hvm_map_ioreq_gfn) is limited to
> a single ioreq page; device models requiring more ioreq slots must use
> the resource mapping interface (XENMEM_acquire_resource).
> 
> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
> ---
> Changes in v6:
> - Adapted the comment to not mention the guest, but the device model
> - Replaced the dynamic allocation for the mfns array by a static array
> - Fixed error handling in ioreq_server_alloc_mfn, using an extra
>   nr_alloc variable to track the already allocated pages
> - Dropped unnecessary void casts
> ---
>  xen/arch/x86/hvm/ioreq.c |  8 ++++
>  xen/common/ioreq.c       | 93 ++++++++++++++++++++++++++++------------
>  xen/include/xen/ioreq.h  | 12 ++++++
>  3 files changed, 86 insertions(+), 27 deletions(-)
> 
> diff --git a/xen/arch/x86/hvm/ioreq.c b/xen/arch/x86/hvm/ioreq.c
> index 3cabec141c..ee679bdf5a 100644
> --- a/xen/arch/x86/hvm/ioreq.c
> +++ b/xen/arch/x86/hvm/ioreq.c
> @@ -166,6 +166,14 @@ static int hvm_map_ioreq_gfn(struct ioreq_server *s, bool buf)
>      if ( d->is_dying )
>          return -EINVAL;
>  
> +    /*
> +     * The legacy GFN path supports only a single ioreq page. Device models
> +     * requiring more ioreq slots must use the resource mapping interface
> +     * (XENMEM_acquire_resource).
> +     */
> +    if ( !buf && nr_ioreq_pages(d) > 1 )
> +        return -EOPNOTSUPP;
> +
>      iorp->gfn = hvm_alloc_ioreq_gfn(s);
>  
>      if ( gfn_eq(iorp->gfn, INVALID_GFN) )
> diff --git a/xen/common/ioreq.c b/xen/common/ioreq.c
> index bae9b99c99..3a08e77597 100644
> --- a/xen/common/ioreq.c
> +++ b/xen/common/ioreq.c
> @@ -261,8 +261,11 @@ bool vcpu_ioreq_handle_completion(struct vcpu *v)
>  static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>  {
>      struct ioreq_page *iorp = buf ? &s->bufioreq : &s->ioreq;
> -    struct page_info *page;
> -    mfn_t mfn;
> +    unsigned int i, nr_alloc = 0, nr_pages = buf ? 1 : nr_ioreq_pages(s->target);
> +    mfn_t mfns[IOREQ_NR_PAGES_MAX] = {};

This is okay to have with the present HVM_MAX_VCPUS, but we need to put in
place something to bound stack usage here. Once the upper bound on the
number of vCPU-s has grown enough, some other mechanism will need to be put
in place, preferably still without runtime allocation. (And no, this isn't
a static array, i.e. the revlog entry isn't quite correct.)

For the moment I'd suggest BUILD_BUG_ON(ARRAY_SIZE(mfns) > 32), with a
clarifying comment.

> +    int rc;
> +
> +    ASSERT(nr_pages <= IOREQ_NR_PAGES_MAX);

Why would this be relevant to check (only) here? Imo this either wants
dropping, or moving into nr_ioreq_pages().

> @@ -277,11 +280,16 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>          return 0;
>      }
>  
> +    for ( i = 0; i < nr_pages; i++ )
>      {
> -        page = alloc_domheap_page(s->target, MEMF_no_refcount);
> +        struct page_info *page = alloc_domheap_page(s->target,
> +                                                    MEMF_no_refcount);

This movement of the decl would better also be part of patch 2.

> @@ -290,41 +298,59 @@ static int ioreq_server_alloc_mfn(struct ioreq_server *s, bool buf)
>               * here is a clear indication of something fishy going on.
>               */
>              domain_crash(s->emulator);
> -            return -ENODATA;
> +            rc = -ENODATA;
> +            goto fail;
>          }
>  
> -        mfn = page_to_mfn(page);
> +        mfns[nr_alloc++] = page_to_mfn(page);
>      }
> -    iorp->va = vmap(&mfn, 1);
> +
> +    iorp->va = vmap(mfns, nr_pages);
>      if ( !iorp->va )
> +    {
> +        rc = -ENOMEM;
>          goto fail;
> +    }
>  
> -    clear_page(iorp->va);
> +    memset(iorp->va, 0, nr_pages * PAGE_SIZE);

clear_page() is a bit more efficient, so I wonder whether - especially for
small nr_pages - we aren't needlessly losing performance here. Question is
whether there's a reasonable to establish boundary at which memset()
(largely) catches up.

> @@ -337,12 +363,25 @@ bool is_ioreq_server_page(struct domain *d, const struct page_info *page)
>  
>      FOR_EACH_IOREQ_SERVER(d, id, s)
>      {
> -        if ( (s->ioreq.va && vmap_to_page(s->ioreq.va) == page) ||
> -             (s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page) )
> +        unsigned int i;
> +
> +        if ( s->bufioreq.va && vmap_to_page(s->bufioreq.va) == page )
>          {
>              found = true;
>              break;
>          }
> +
> +        for ( i = 0; i < nr_ioreq_pages(d) && s->ioreq.va; i++ )

s->ioreq.va is loop invariant, without the compiler being in the position
to know. The condition therefore wants moving out, preferably as

        if ( !s->ioreq.va )
            continue;

to avoid indentation growing too much.

> +        {
> +            if ( vmap_to_page(s->ioreq.va + i * PAGE_SIZE) == page )
> +            {
> +                found = true;
> +                break;
> +            }
> +        }

It may help readability here if for()'s curly braces were dropped.

> +        if ( found )
> +            break;
>      }
>  
>      rspin_unlock(&d->ioreq_server.lock);
> @@ -816,26 +855,26 @@ int ioreq_server_get_frame(struct domain *d, ioservid_t id,
>      if ( rc )
>          goto out;
>  
> -    switch ( idx )
> +    if ( idx == XENMEM_resource_ioreq_server_frame_bufioreq )
>      {
> -    case XENMEM_resource_ioreq_server_frame_bufioreq:
>          rc = -ENOENT;
>          if ( !HANDLE_BUFIOREQ(s) )
>              goto out;
>  
>          *mfn = vmap_to_mfn(s->bufioreq.va);
>          rc = 0;
> -        break;
> +    }
> +    else if ( idx >= XENMEM_resource_ioreq_server_frame_ioreq(0) &&
> +              idx < XENMEM_resource_ioreq_server_frame_ioreq(nr_ioreq_pages(d)) )
> +    {
> +        unsigned int page_idx = idx - XENMEM_resource_ioreq_server_frame_ioreq(0);
>  
> -    case XENMEM_resource_ioreq_server_frame_ioreq(0):
> -        *mfn = vmap_to_mfn(s->ioreq.va);
> +        ASSERT(page_idx < nr_ioreq_pages(d));

Isn't this redundant with the range check on idx?

> --- a/xen/include/xen/ioreq.h
> +++ b/xen/include/xen/ioreq.h
> @@ -35,6 +35,18 @@ struct ioreq_vcpu {
>      bool             pending;
>  };
>  
> +/*
> + * Maximum number of ioreq pages, based on the maximum number
> + * of vCPUs and the number of ioreq slots per page.
> + */
> +#define IOREQ_NR_PAGES_MAX \
> +    DIV_ROUND_UP(HVM_MAX_VCPUS, PAGE_SIZE / sizeof(ioreq_t))
> +
> +static inline unsigned int nr_ioreq_pages(const struct domain *d)
> +{
> +    return DIV_ROUND_UP(d->max_vcpus, PAGE_SIZE / sizeof(ioreq_t));
> +}

To reduce redundancy, how about

static inline unsigned int nr_ioreq_pages(const struct domain *d)
{
    return DIV_ROUND_UP(d ? d->max_vcpus : HVM_MAX_VCPUS,
                        PAGE_SIZE / sizeof(ioreq_t));
}

#define IOREQ_NR_PAGES_MAX nr_ioreq_pages(NULL)

?

Jan

Re: [PATCH v6 0/3] Support multiple ioreq pages

[Jan Beulich]

On 20.04.2026 11:38, Julian Vetter wrote:
> Hello Jan,
> thank you again for your thorough feedback. I have addressed your
> comments. I just have one remaining issue. For the temporary MFNs array
> I now use a static array, i.e., mfns[IOREQ_NR_PAGES_MAX], with a
> IOREQ_NR_PAGES_MAX = DIV_ROUND_UP(HVM_MAX_VCPUS, PAGE_SIZE / sizeof(ioreq_t)).
> Is this appropriate?

Whether using a static array is appropriate I can't tell after merely
having read this cover letter.

> Or should I leave the HVM_MAX_VCPUS out of this and
> instead use a fixed constant value? Also for now I haven't taken the
> path to permanently store the MFNs as you suggested. I'm not sure, but I
> don't think it's really necessary. Since this array will rarely exceed
> the size of 2 or 3.

How do you know? The array will need to be large enough to cope with
anything the hypervisor was configured for.

Until we actually raise the vCPU-s limit, using HVM_MAX_VCPUS for the
size calculation looks okay to me. All uses of that constant will need
looking at / adjusting anyway during that subsequent effort.

Jan

Re: [PATCH v6 0/3] Support multiple ioreq pages

[Jan Beulich]

On 20.04.2026 11:38, Julian Vetter wrote:
> Julian Vetter (3):
>   ioreq: switch ioreq page allocation to vmap
>   ioreq: Indent ioreq_server_alloc_mfn() body one level deeper
>   x86/ioreq: Extend ioreq server to support multiple ioreq pages
> 
>  xen/arch/x86/hvm/ioreq.c |  63 ++++++++++++++++---
>  xen/common/ioreq.c       | 127 ++++++++++++++++++++++++++-------------
>  xen/include/xen/ioreq.h  |  13 +++-
>  3 files changed, 151 insertions(+), 52 deletions(-)

For (future) reference, in case it wasn't said earlier:

To be able to test this, at least the last patch here will want to wait
until the apic_id == vcpu_id * 2 issue was addressed. Andrew said he'd pick
up Alejandro's work there, thus - once finished - permitting up to 255
vCPU-s (i.e. requiring 2 IOREQ pages).

Once (really: before) we grow the number of vCPU-s for HVM, we need to
revisit the amount of VA space set aside for vmap(). For many years we've
been adding new uses of vmap() without making sure its reserved range is
still adequately sized.

Since multi-page functionality added here will also need qemu changes, and
since we did determine (elsewhere) that ioreq_t needs to grow as well, it
remains to be decided whether the two changes wouldn't better be done
together, to keep the qemu backwards compatibility logic somewhat limited
in size / complexity. Anthony (in particular) - thoughts?

There may be more aspects which I forget.

Jan