[RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Val Packett]

Starting a virtio backend in a PV domain would panic the kernel in
alloc_ioreq, trying to dereference vma->vm_private_data as a pages
pointer when in reality it stayed as PRIV_VMA_LOCKED.

Fix by allocating a pages array in mmap_resource in the PV case,
filling it with page info converted from the pfn array. This allows
ioreq to function successfully with a backend provided by a PV dom0.

Signed-off-by: Val Packett <val@invisiblethingslab.com>
---
I've been porting the xen-vhost-frontend[1] to Qubes, which runs on amd64
and we (still) use PV for dom0. The x86 part didn't give me much trouble,
but the first thing I found was this crash due to using a PV domain to host
the backend. alloc_ioreq was dereferencing the '1' constant and panicking
the dom0 kernel.

I figured out that I can make a pages array in the expected format from the
pfn array where the actual memory mapping happens for the PV case, and with
the fix, the ioreq part works: the vhost frontend replies to the probing
sequence and the guest recognizes which virtio device is being provided.

I still have another thing to debug: the MMIO accesses from the inner driver
(e.g. virtio_rng) don't get through to the vhost provider (ioeventfd does
not get notified), and manually kicking the eventfd from the frontend
seems to crash... Xen itself?? (no Linux panic on console, just a freeze and
quick reboot - will try to set up a serial console now)

But I figured I'd post this as an RFC already, since the other bug may be
unrelated and the ioreq area itself does work now. I'd like to hear some
feedback on this from people who actually know Xen :)

[1]: https://github.com/vireshk/xen-vhost-frontend

Thanks,
~val
---
 drivers/xen/privcmd.c | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index f52a457b302d..c9b4dae7e520 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -834,8 +834,23 @@ static long privcmd_ioctl_mmap_resource(struct file *file,
 				if (rc < 0)
 					break;
 			}
-		} else
+		} else {
+			unsigned int i;
+			unsigned int numpgs = kdata.num / XEN_PFN_PER_PAGE;
+			struct page **pages;
 			rc = 0;
+
+			pages = kvcalloc(numpgs, sizeof(pages[0]), GFP_KERNEL);
+			if (pages == NULL) {
+				rc = -ENOMEM;
+				goto out;
+			}
+
+			for (i = 0; i < numpgs; i++) {
+				pages[i] = xen_pfn_to_page(pfns[i * XEN_PFN_PER_PAGE]);
+			}
+			vma->vm_private_data = pages;
+		}
 	}
 
 out:
@@ -1589,15 +1604,18 @@ static void privcmd_close(struct vm_area_struct *vma)
 	int numgfns = (vma->vm_end - vma->vm_start) >> XEN_PAGE_SHIFT;
 	int rc;
 
-	if (xen_pv_domain() || !numpgs || !pages)
+	if (!numpgs || !pages)
 		return;
 
-	rc = xen_unmap_domain_gfn_range(vma, numgfns, pages);
-	if (rc == 0)
-		xen_free_unpopulated_pages(numpgs, pages);
-	else
-		pr_crit("unable to unmap MFN range: leaking %d pages. rc=%d\n",
-			numpgs, rc);
+	if (!xen_pv_domain()) {
+		rc = xen_unmap_domain_gfn_range(vma, numgfns, pages);
+		if (rc == 0)
+			xen_free_unpopulated_pages(numpgs, pages);
+		else
+			pr_crit("unable to unmap MFN range: leaking %d pages. rc=%d\n",
+				numpgs, rc);
+	}
+
 	kvfree(pages);
 }
 
-- 
2.51.0

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 493 bytes --]

I'm pretty sure Xen doesn't support ioeventfd at all yet.  There are
two options:

1. Port the backend to use an ioreq server instead of ioeventfd.
2. Modify Linux (and possibly Xen) to support ioeventfd.

Which virtio device are you trying to implement?  virtio-GPU will need
a lot of additional work and it would probably be best to discuss
this privately (over a call or similar).  Others are likely to be
significantly easier.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 637 bytes --]

On 10/18/25 20:42, Demi Marie Obenour wrote:
> I'm pretty sure Xen doesn't support ioeventfd at all yet.  There are
> two options:
> 
> 1. Port the backend to use an ioreq server instead of ioeventfd.
> 2. Modify Linux (and possibly Xen) to support ioeventfd.
> 
> Which virtio device are you trying to implement?  virtio-GPU will need
> a lot of additional work and it would probably be best to discuss
> this privately (over a call or similar).  Others are likely to be
> significantly easier.

Whoops, I got confused and didn't see you were using a Xen-native
frontend.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 2354 bytes --]

On 15.10.25 21:57, Val Packett wrote:
> Starting a virtio backend in a PV domain would panic the kernel in
> alloc_ioreq, trying to dereference vma->vm_private_data as a pages
> pointer when in reality it stayed as PRIV_VMA_LOCKED.
> 
> Fix by allocating a pages array in mmap_resource in the PV case,
> filling it with page info converted from the pfn array. This allows
> ioreq to function successfully with a backend provided by a PV dom0.
> 
> Signed-off-by: Val Packett <val@invisiblethingslab.com>
> ---
> I've been porting the xen-vhost-frontend[1] to Qubes, which runs on amd64
> and we (still) use PV for dom0. The x86 part didn't give me much trouble,
> but the first thing I found was this crash due to using a PV domain to host
> the backend. alloc_ioreq was dereferencing the '1' constant and panicking
> the dom0 kernel.
> 
> I figured out that I can make a pages array in the expected format from the
> pfn array where the actual memory mapping happens for the PV case, and with
> the fix, the ioreq part works: the vhost frontend replies to the probing
> sequence and the guest recognizes which virtio device is being provided.
> 
> I still have another thing to debug: the MMIO accesses from the inner driver
> (e.g. virtio_rng) don't get through to the vhost provider (ioeventfd does
> not get notified), and manually kicking the eventfd from the frontend
> seems to crash... Xen itself?? (no Linux panic on console, just a freeze and
> quick reboot - will try to set up a serial console now)

IMHO for making the MMIO accesses work you'd need to implement ioreq-server
support for PV-domains in the hypervisor. This will be a major endeavor, so
before taking your Linux kernel patch I'd like to see this covered.

> But I figured I'd post this as an RFC already, since the other bug may be
> unrelated and the ioreq area itself does work now. I'd like to hear some
> feedback on this from people who actually know Xen :)

My main problem with your patch is that it is adding a memory allocation
for a very rare use case impacting all current users of that functionality.

You could avoid that by using a different ioctl which could be selected by
specifying a new flag when calling xenforeignmemory_open() (have a look
into the Xen sources under tools/libs/foreignmemory/core.c).


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 3003 bytes --]

On 11/4/25 07:15, Jürgen Groß wrote:
> On 15.10.25 21:57, Val Packett wrote:
>> Starting a virtio backend in a PV domain would panic the kernel in
>> alloc_ioreq, trying to dereference vma->vm_private_data as a pages
>> pointer when in reality it stayed as PRIV_VMA_LOCKED.
>>
>> Fix by allocating a pages array in mmap_resource in the PV case,
>> filling it with page info converted from the pfn array. This allows
>> ioreq to function successfully with a backend provided by a PV dom0.
>>
>> Signed-off-by: Val Packett <val@invisiblethingslab.com>
>> ---
>> I've been porting the xen-vhost-frontend[1] to Qubes, which runs on amd64
>> and we (still) use PV for dom0. The x86 part didn't give me much trouble,
>> but the first thing I found was this crash due to using a PV domain to host
>> the backend. alloc_ioreq was dereferencing the '1' constant and panicking
>> the dom0 kernel.
>>
>> I figured out that I can make a pages array in the expected format from the
>> pfn array where the actual memory mapping happens for the PV case, and with
>> the fix, the ioreq part works: the vhost frontend replies to the probing
>> sequence and the guest recognizes which virtio device is being provided.
>>
>> I still have another thing to debug: the MMIO accesses from the inner driver
>> (e.g. virtio_rng) don't get through to the vhost provider (ioeventfd does
>> not get notified), and manually kicking the eventfd from the frontend
>> seems to crash... Xen itself?? (no Linux panic on console, just a freeze and
>> quick reboot - will try to set up a serial console now)
> 
> IMHO for making the MMIO accesses work you'd need to implement ioreq-server
> support for PV-domains in the hypervisor. This will be a major endeavor, so
> before taking your Linux kernel patch I'd like to see this covered.

Would fixing this be a good use of time, or would it be better to
focus on switching to PVH dom0?  I don't know if it makes sense to
spend effort on PV dom0 when dom0 isn't going to be PV indefinitely.

Edera might well be interested in the PV case, as they run in cloud
VMs without nested virtualization.  That's not relevant to Qubes
OS, though.

>> But I figured I'd post this as an RFC already, since the other bug may be
>> unrelated and the ioreq area itself does work now. I'd like to hear some
>> feedback on this from people who actually know Xen :)
> 
> My main problem with your patch is that it is adding a memory allocation
> for a very rare use case impacting all current users of that functionality.
> 
> You could avoid that by using a different ioctl which could be selected by
> specifying a new flag when calling xenforeignmemory_open() (have a look
> into the Xen sources under tools/libs/foreignmemory/core.c).

Should there at least be a check to prevent the kernel from crashing?
I'd expect an unsupported use of the API to return an error, not
cause the kernel to oops.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 1935 bytes --]

On 11/4/25 07:15, Jürgen Groß wrote:
> On 15.10.25 21:57, Val Packett wrote:
>> Starting a virtio backend in a PV domain would panic the kernel in
>> alloc_ioreq, trying to dereference vma->vm_private_data as a pages
>> pointer when in reality it stayed as PRIV_VMA_LOCKED.
>>
>> Fix by allocating a pages array in mmap_resource in the PV case,
>> filling it with page info converted from the pfn array. This allows
>> ioreq to function successfully with a backend provided by a PV dom0.
>>
>> Signed-off-by: Val Packett <val@invisiblethingslab.com>
>> ---
>> I've been porting the xen-vhost-frontend[1] to Qubes, which runs on amd64
>> and we (still) use PV for dom0. The x86 part didn't give me much trouble,
>> but the first thing I found was this crash due to using a PV domain to host
>> the backend. alloc_ioreq was dereferencing the '1' constant and panicking
>> the dom0 kernel.
>>
>> I figured out that I can make a pages array in the expected format from the
>> pfn array where the actual memory mapping happens for the PV case, and with
>> the fix, the ioreq part works: the vhost frontend replies to the probing
>> sequence and the guest recognizes which virtio device is being provided.
>>
>> I still have another thing to debug: the MMIO accesses from the inner driver
>> (e.g. virtio_rng) don't get through to the vhost provider (ioeventfd does
>> not get notified), and manually kicking the eventfd from the frontend
>> seems to crash... Xen itself?? (no Linux panic on console, just a freeze and
>> quick reboot - will try to set up a serial console now)
> 
> IMHO for making the MMIO accesses work you'd need to implement ioreq-server
> support for PV-domains in the hypervisor. This will be a major endeavor, so
> before taking your Linux kernel patch I'd like to see this covered.

Could Xen return an error instead of crashing?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Val Packett]

[-- Attachment #1.1.1: Type: text/plain, Size: 5014 bytes --]


On 11/4/25 9:15 AM, Jürgen Groß wrote:
> On 15.10.25 21:57, Val Packett wrote:
>> Starting a virtio backend in a PV domain would panic the kernel in
>> alloc_ioreq, trying to dereference vma->vm_private_data as a pages
>> pointer when in reality it stayed as PRIV_VMA_LOCKED.
>>
>> Fix by allocating a pages array in mmap_resource in the PV case,
>> filling it with page info converted from the pfn array. This allows
>> ioreq to function successfully with a backend provided by a PV dom0.
>>
>> Signed-off-by: Val Packett <val@invisiblethingslab.com>
>> ---
>> I've been porting the xen-vhost-frontend[1] to Qubes, which runs on 
>> amd64
>> and we (still) use PV for dom0. The x86 part didn't give me much 
>> trouble,
>> but the first thing I found was this crash due to using a PV domain 
>> to host
>> the backend. alloc_ioreq was dereferencing the '1' constant and 
>> panicking
>> the dom0 kernel.
>>
>> I figured out that I can make a pages array in the expected format 
>> from the
>> pfn array where the actual memory mapping happens for the PV case, 
>> and with
>> the fix, the ioreq part works: the vhost frontend replies to the probing
>> sequence and the guest recognizes which virtio device is being provided.
>>
>> I still have another thing to debug: the MMIO accesses from the inner 
>> driver
>> (e.g. virtio_rng) don't get through to the vhost provider (ioeventfd 
>> does
>> not get notified), and manually kicking the eventfd from the frontend
>> seems to crash... Xen itself?? (no Linux panic on console, just a 
>> freeze and
>> quick reboot - will try to set up a serial console now)
>
> IMHO for making the MMIO accesses work you'd need to implement 
> ioreq-server
> support for PV-domains in the hypervisor. This will be a major 
> endeavor, so
> before taking your Linux kernel patch I'd like to see this covered.

Sorry, I wasn't clear enough.. it's *not* that MMIO accesses don't work.

I debugged this a bit more, and it turns out:

1. the reason why "ioeventfd does not get notified" is because accessing 
the virtio page (allocated with this privcmd interface) from the kernel 
was failing. The exchange between the guest driver and the userspace 
ioreq server has been working perfectly, but the *kernel* access (which 
is what needs this `struct page` allocation with the current code) was 
returning nonsense and the check for the virtqueue readiness flag was 
failing.

I have noticed and fixed (locally) a bug in this patch: reusing the 
`pfns` allocation for `errs` in `xen_remap_domain_mfn_array` meant that 
the actual pfn value was overwritten with a zero ("success" error code), 
and that's the `pfn` I was using.

Still, the memory visible in the dom0 kernel at that pfn is not the same 
allocation that's mapped into the process. Instead, it's some random 
other memory. I've added a hexdump for it in the ioeventfd notifier and 
it was returning random stuff from other userspace programs such as "// 
SPDX-License-Identifier" from a text editor (haha). Actually, *once* it 
did just work and I've managed to attach a virtio-rng driver and have it 
fully work.

Clearly I'm just struggling with the way memory mappings work under PV. 
Do I need to specifically create a second mapping for the kernel using 
the same `xen_remap_domain_mfn_array` call?

2. the reason why "manually kicking the eventfd from the frontend seems 
to crash... Xen itself" was actually because that triggered the guest 
interrupt and I was using the ISA interrupts that required the virtual 
(IO)APIC to exist, and it doesn't in PVH domains. For now I switched my 
test setup to HVM to get around that, but I'd need to.. figure out a 
virq/pirq type setup to route XEN_DMOP_set_isa_irq_level calls over 
event channels for PV(H) guests.

>> But I figured I'd post this as an RFC already, since the other bug 
>> may be
>> unrelated and the ioreq area itself does work now. I'd like to hear some
>> feedback on this from people who actually know Xen :)
>
> My main problem with your patch is that it is adding a memory allocation
> for a very rare use case impacting all current users of that 
> functionality.
>
> You could avoid that by using a different ioctl which could be 
> selected by
> specifying a new flag when calling xenforeignmemory_open() (have a look
> into the Xen sources under tools/libs/foreignmemory/core.c).

Right, that could be solved. Having userspace choose based on what kind 
of domain it is sounds a bit painful (you're talking about C libraries 
and I'm using independent Rust ones, so this logic would have to be 
present in multiple places).. But this kernel code could be refactored more.

We don't actually need any `struct page` specifically, 
`ioeventfd_interrupt` only really needs a kernel pointer to the actual 
ioreq memory we're allocating here.

I'm mostly just asking for help to figure out how to get that pointer.


Thanks,
~val


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 1289 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 273 bytes --]

Re: [RFC PATCH] xen: privcmd: fix ioeventfd/ioreq crashing PV domain

[Demi Marie Obenour]

[-- Attachment #1.1.1: Type: text/plain, Size: 4130 bytes --]

On 11/4/25 20:16, Val Packett wrote:
> 
> On 11/4/25 9:15 AM, Jürgen Groß wrote:
>> On 15.10.25 21:57, Val Packett wrote:
>>> Starting a virtio backend in a PV domain would panic the kernel in
>>> alloc_ioreq, trying to dereference vma->vm_private_data as a pages
>>> pointer when in reality it stayed as PRIV_VMA_LOCKED.
>>>
>>> Fix by allocating a pages array in mmap_resource in the PV case,
>>> filling it with page info converted from the pfn array. This allows
>>> ioreq to function successfully with a backend provided by a PV dom0.
>>>
>>> Signed-off-by: Val Packett <val@invisiblethingslab.com>
>>> ---
>>> I've been porting the xen-vhost-frontend[1] to Qubes, which runs on 
>>> amd64
>>> and we (still) use PV for dom0. The x86 part didn't give me much 
>>> trouble,
>>> but the first thing I found was this crash due to using a PV domain 
>>> to host
>>> the backend. alloc_ioreq was dereferencing the '1' constant and 
>>> panicking
>>> the dom0 kernel.
>>>
>>> I figured out that I can make a pages array in the expected format 
>>> from the
>>> pfn array where the actual memory mapping happens for the PV case, 
>>> and with
>>> the fix, the ioreq part works: the vhost frontend replies to the probing
>>> sequence and the guest recognizes which virtio device is being provided.
>>>
>>> I still have another thing to debug: the MMIO accesses from the inner 
>>> driver
>>> (e.g. virtio_rng) don't get through to the vhost provider (ioeventfd 
>>> does
>>> not get notified), and manually kicking the eventfd from the frontend
>>> seems to crash... Xen itself?? (no Linux panic on console, just a 
>>> freeze and
>>> quick reboot - will try to set up a serial console now)
>>
>> IMHO for making the MMIO accesses work you'd need to implement 
>> ioreq-server
>> support for PV-domains in the hypervisor. This will be a major 
>> endeavor, so
>> before taking your Linux kernel patch I'd like to see this covered.
> 
> Sorry, I wasn't clear enough.. it's *not* that MMIO accesses don't work.
> 
> I debugged this a bit more, and it turns out:
> 
> 1. the reason why "ioeventfd does not get notified" is because accessing 
> the virtio page (allocated with this privcmd interface) from the kernel 
> was failing. The exchange between the guest driver and the userspace 
> ioreq server has been working perfectly, but the *kernel* access (which 
> is what needs this `struct page` allocation with the current code) was 
> returning nonsense and the check for the virtqueue readiness flag was 
> failing.
> 
> I have noticed and fixed (locally) a bug in this patch: reusing the 
> `pfns` allocation for `errs` in `xen_remap_domain_mfn_array` meant that 
> the actual pfn value was overwritten with a zero ("success" error code), 
> and that's the `pfn` I was using.
> 
> Still, the memory visible in the dom0 kernel at that pfn is not the same 
> allocation that's mapped into the process. Instead, it's some random 
> other memory. I've added a hexdump for it in the ioeventfd notifier and 
> it was returning random stuff from other userspace programs such as "// 
> SPDX-License-Identifier" from a text editor (haha). Actually, *once* it 
> did just work and I've managed to attach a virtio-rng driver and have it 
> fully work.
> 
> Clearly I'm just struggling with the way memory mappings work under PV. 
> Do I need to specifically create a second mapping for the kernel using 
> the same `xen_remap_domain_mfn_array` call?
> 
> 2. the reason why "manually kicking the eventfd from the frontend seems 
> to crash... Xen itself" was actually because that triggered the guest 
> interrupt and I was using the ISA interrupts that required the virtual 
> (IO)APIC to exist, and it doesn't in PVH domains. For now I switched my 
> test setup to HVM to get around that, but I'd need to.. figure out a 
> virq/pirq type setup to route XEN_DMOP_set_isa_irq_level calls over 
> event channels for PV(H) guests.

Still, this should return an error rather than crashing the hypervisor.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]