[RFC PATCH] vhost: enable IOMMU_NOTIFIER_UNMAP events handling when device-iotlb=on

[RFC PATCH] vhost: enable IOMMU_NOTIFIER_UNMAP events handling when device-iotlb=on

[Viktor Prutyanov]

Even if Device-TLB and PCI ATS is enabled, the guest can reject to use
it. For example, this situation appears when Windows Server 2022 is
running with intel-iommu with device-iotlb=on and virtio-net-pci with
vhost=on. The guest implies that no address translation info cached in
device IOTLB and doesn't send device IOTLB invalidation commands. So,
it leads to irrelevant address translations in vhost-net in the host
kernel. Therefore network frames from the guest in host tap interface
contains wrong payload data.

This patch enables IOTLB unmap events (IOMMU_NOTIFIER_UNMAP) along with
Device-TLB unmap events (IOMMU_NOTIFIER_DEVIOTLB_UNMAP) handling for
proper vhost IOTLB unmapping when the guest isn't aware of Device-TLB.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2001312
Signed-off-by: Viktor Prutyanov <viktor@daynix.com>
---

 Tested on Windows Server 2022 and Fedora guests with
 -device virtio-net-pci,bus=pci.3,netdev=nd0,iommu_platform=on,ats=on
 -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,vhost=on
 -device intel-iommu,intremap=on,eim=on,device-iotlb=on/off

 hw/virtio/vhost.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index a266396576..968ca18fce 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -796,7 +796,7 @@ static void vhost_iommu_region_add(MemoryListener *listener,
     iommu_idx = memory_region_iommu_attrs_to_index(iommu_mr,
                                                    MEMTXATTRS_UNSPECIFIED);
     iommu_notifier_init(&iommu->n, vhost_iommu_unmap_notify,
-                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP,
+                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP | IOMMU_NOTIFIER_UNMAP,
                         section->offset_within_region,
                         int128_get64(end),
                         iommu_idx);
-- 
2.35.1

Re: [RFC PATCH] vhost: enable IOMMU_NOTIFIER_UNMAP events handling when device-iotlb=on

[Viktor Prutyanov]

On Thu, Mar 30, 2023 at 7:49 PM Viktor Prutyanov <viktor@daynix.com> wrote:
>
> Even if Device-TLB and PCI ATS is enabled, the guest can reject to use
> it. For example, this situation appears when Windows Server 2022 is
> running with intel-iommu with device-iotlb=on and virtio-net-pci with
> vhost=on. The guest implies that no address translation info cached in
> device IOTLB and doesn't send device IOTLB invalidation commands. So,
> it leads to irrelevant address translations in vhost-net in the host
> kernel. Therefore network frames from the guest in host tap interface
> contains wrong payload data.
>
> This patch enables IOTLB unmap events (IOMMU_NOTIFIER_UNMAP) along with
> Device-TLB unmap events (IOMMU_NOTIFIER_DEVIOTLB_UNMAP) handling for
> proper vhost IOTLB unmapping when the guest isn't aware of Device-TLB.
>
> Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2001312
> Signed-off-by: Viktor Prutyanov <viktor@daynix.com>
> ---
>
>  Tested on Windows Server 2022 and Fedora guests with
>  -device virtio-net-pci,bus=pci.3,netdev=nd0,iommu_platform=on,ats=on
>  -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,vhost=on
>  -device intel-iommu,intremap=on,eim=on,device-iotlb=on/off
>
>  hw/virtio/vhost.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index a266396576..968ca18fce 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -796,7 +796,7 @@ static void vhost_iommu_region_add(MemoryListener *listener,
>      iommu_idx = memory_region_iommu_attrs_to_index(iommu_mr,
>                                                     MEMTXATTRS_UNSPECIFIED);
>      iommu_notifier_init(&iommu->n, vhost_iommu_unmap_notify,
> -                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP,
> +                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP | IOMMU_NOTIFIER_UNMAP,
>                          section->offset_within_region,
>                          int128_get64(end),
>                          iommu_idx);
> --
> 2.35.1
>

ping

Re: [RFC PATCH] vhost: enable IOMMU_NOTIFIER_UNMAP events handling when device-iotlb=on

[Jason Wang]

On Thu, Apr 13, 2023 at 2:24 PM Viktor Prutyanov <viktor@daynix.com> wrote:
>
> On Thu, Mar 30, 2023 at 7:49 PM Viktor Prutyanov <viktor@daynix.com> wrote:
> >
> > Even if Device-TLB and PCI ATS is enabled, the guest can reject to use
> > it. For example, this situation appears when Windows Server 2022 is
> > running with intel-iommu with device-iotlb=on and virtio-net-pci with
> > vhost=on. The guest implies that no address translation info cached in
> > device IOTLB and doesn't send device IOTLB invalidation commands. So,
> > it leads to irrelevant address translations in vhost-net in the host
> > kernel. Therefore network frames from the guest in host tap interface
> > contains wrong payload data.
> >
> > This patch enables IOTLB unmap events (IOMMU_NOTIFIER_UNMAP) along with
> > Device-TLB unmap events (IOMMU_NOTIFIER_DEVIOTLB_UNMAP) handling for
> > proper vhost IOTLB unmapping when the guest isn't aware of Device-TLB.
> >
> > Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2001312
> > Signed-off-by: Viktor Prutyanov <viktor@daynix.com>
> > ---
> >
> >  Tested on Windows Server 2022 and Fedora guests with
> >  -device virtio-net-pci,bus=pci.3,netdev=nd0,iommu_platform=on,ats=on
> >  -netdev tap,id=nd0,ifname=tap1,script=no,downscript=no,vhost=on
> >  -device intel-iommu,intremap=on,eim=on,device-iotlb=on/off
> >
> >  hw/virtio/vhost.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > index a266396576..968ca18fce 100644
> > --- a/hw/virtio/vhost.c
> > +++ b/hw/virtio/vhost.c
> > @@ -796,7 +796,7 @@ static void vhost_iommu_region_add(MemoryListener *listener,
> >      iommu_idx = memory_region_iommu_attrs_to_index(iommu_mr,
> >                                                     MEMTXATTRS_UNSPECIFIED);
> >      iommu_notifier_init(&iommu->n, vhost_iommu_unmap_notify,
> > -                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP,
> > +                        IOMMU_NOTIFIER_DEVIOTLB_UNMAP | IOMMU_NOTIFIER_UNMAP,
> >                          section->offset_within_region,
> >                          int128_get64(end),
> >                          iommu_idx);

This looks kind of tricky. E.g when ATS is enabled, there could be
unnecessary flues. Or at least it requires some comment to explain.

I think the correct fix is:

1) introduce a transport specific method for querying if it supports
device IOTLB
2) in the pci implementation hook it to some like pice_ats_enabled()
3) so we can decide if device IOTLB is enabled, go for DEVIOTLB_UNMAP
otherwise a simple UNMAP.

Thanks

> > --
> > 2.35.1
> >
>
> ping
>