[Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Thanos Makatos]

When configuring device pass-through via VFIO to a VM, I noticed that QEMU tries to register (DMA_MAP) all memory regions of a guest (not only RAM). That includes firmware regions like "pc.rom". Would a physical device ever need access to those? Am I missing something?

Re: [Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Alex Williamson]

On Fri, 31 May 2019 13:32:29 +0000
Thanos Makatos <thanos.makatos@nutanix.com> wrote:

> When configuring device pass-through via VFIO to a VM, I noticed that
> QEMU tries to register (DMA_MAP) all memory regions of a guest (not
> only RAM). That includes firmware regions like "pc.rom". Would a
> physical device ever need access to those?

Probably not, but are those things not in the address space of the
device on a physical system?

> Am I missing something?

Does this cause a problem?  It's not always easy to identify regions
that should not be mapped to a device, clearly we're not going to
create a whitelist based on the name of the region.  Thanks,

Alex

Re: [Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Thanos Makatos]

> > When configuring device pass-through via VFIO to a VM, I noticed that
> > QEMU tries to register (DMA_MAP) all memory regions of a guest (not
                > > only RAM). That includes firmware regions like "pc.rom". Would a
> > physical device ever need access to those?
>
> Probably not, but are those things not in the address space of the
> device on a physical system?

They are. I'm wondering whether it makes sense in a virtualized environment.

>
> > Am I missing something?
>
> Does this cause a problem?

It does in my use case. We're experimenting with devices backed by another
userspace application. We can configure QEMU to allocate shared memory
(MAP_SHARED) for guest RAM (which we can register in the other process) but not
for anything else.

>  It's not always easy to identify regions
> that should not be mapped to a device, clearly we're not going to
> create a whitelist based on the name of the region.  Thanks,

Indeed. Could we decide whether or not to register an address space with
VFIO in a more intelligent manner? E.g. the following simplistic patch solves
our problem:

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index 4374cc6176..d9d3b1277a 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -430,6 +430,9 @@ static void vfio_listener_region_add(MemoryListener *listener,
     VFIOHostDMAWindow *hostwin;
     bool hostwin_found;

+    if (!section->mr->ram_device)
+        return;
+
     if (vfio_listener_skipped_section(section)) {
         trace_vfio_listener_region_add_skip(
                 section->offset_within_address_space,

Re: [Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Alex Williamson]

On Fri, 31 May 2019 15:28:07 +0000
Thanos Makatos <thanos.makatos@nutanix.com> wrote:

> > > When configuring device pass-through via VFIO to a VM, I noticed that
> > > QEMU tries to register (DMA_MAP) all memory regions of a guest (not
>                 > > only RAM). That includes firmware regions like "pc.rom". Would a
> > > physical device ever need access to those?  
> >
> > Probably not, but are those things not in the address space of the
> > device on a physical system?  
> 
> They are. I'm wondering whether it makes sense in a virtualized environment.
> 
> >  
> > > Am I missing something?  
> >
> > Does this cause a problem?  
> 
> It does in my use case. We're experimenting with devices backed by another
> userspace application. We can configure QEMU to allocate shared memory
> (MAP_SHARED) for guest RAM (which we can register in the other process) but not
> for anything else.
> 
> >  It's not always easy to identify regions
> > that should not be mapped to a device, clearly we're not going to
> > create a whitelist based on the name of the region.  Thanks,  
> 
> Indeed. Could we decide whether or not to register an address space with
> VFIO in a more intelligent manner? E.g. the following simplistic patch solves
> our problem:
> 
> diff --git a/hw/vfio/common.c b/hw/vfio/common.c
> index 4374cc6176..d9d3b1277a 100644
> --- a/hw/vfio/common.c
> +++ b/hw/vfio/common.c
> @@ -430,6 +430,9 @@ static void vfio_listener_region_add(MemoryListener *listener,
>      VFIOHostDMAWindow *hostwin;
>      bool hostwin_found;
> 
> +    if (!section->mr->ram_device)
> +        return;
> +

Nope, this would prevent IOMMU mapping of assigned device MMIO regions
which would prevent peer-to-peer DMA between assigned devices.  Thanks,

Alex

Re: [Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Thanos Makatos]

> > Indeed. Could we decide whether or not to register an address space with
> > VFIO in a more intelligent manner? E.g. the following simplistic patch solves
> > our problem:
> >
> > diff --git a/hw/vfio/common.c b/hw/vfio/common.c
> > index 4374cc6176..d9d3b1277a 100644
> > --- a/hw/vfio/common.c
> > +++ b/hw/vfio/common.c
> > @@ -430,6 +430,9 @@ static void
> vfio_listener_region_add(MemoryListener *listener,
> >      VFIOHostDMAWindow *hostwin;
> >      bool hostwin_found;
> >
> > +    if (!section->mr->ram_device)
> > +        return;
> > +
> 
> Nope, this would prevent IOMMU mapping of assigned device MMIO
> regions
> which would prevent peer-to-peer DMA between assigned devices.  Thanks,

Understood.

Is there a strong reason why QEMU allocates memory for these address spaces without MAP_SHARED? In our use case it would solve our problem if we could make QEMU use MAP_SHARED. I understand that this isn't strictly correct, so would it be acceptable to enable this behavior with a command-line option or an #ifdef?

Re: [Qemu-devel] QEMU tries to register to VFIO memory that is not RAM

[Thanos Makatos]

> > > diff --git a/hw/vfio/common.c b/hw/vfio/common.c
> > > index 4374cc6176..d9d3b1277a 100644
> > > --- a/hw/vfio/common.c
> > > +++ b/hw/vfio/common.c
> > > @@ -430,6 +430,9 @@ static void
> > vfio_listener_region_add(MemoryListener *listener,
> > >      VFIOHostDMAWindow *hostwin;
> > >      bool hostwin_found;
> > >
> > > +    if (!section->mr->ram_device)
> > > +        return;
> > > +
> >
> > Nope, this would prevent IOMMU mapping of assigned device MMIO
> > regions
> > which would prevent peer-to-peer DMA between assigned devices.
> Thanks,
> 
> Understood.
> 
> Is there a strong reason why QEMU allocates memory for these address
> spaces without MAP_SHARED? In our use case it would solve our problem if
> we could make QEMU use MAP_SHARED. I understand that this isn't strictly
> correct, so would it be acceptable to enable this behavior with a command-
> line option or an #ifdef?

Ping!