Questions about the VFIO BAR region

Questions about the VFIO BAR region

[Li Qiang]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hello Alex, Auger and all,

I have a question about the VFIO virtual device BAR.

In vfio_region_setup, it initialize a ‘region->mem’ MR and set its ops to ‘vfio_regions_ops’. 
In ‘vfio_region_mmap’, it maps the physical device’s MMIO to QEMU’s virtual address space 
as a raw MR ‘region->mmaps[i].mem’. 
And also it set the latter MR as a subregion of the first one.

So when the guest accesses the BAR, it will direct go to the physical device’s BAR.
My question is here:
When the qemu will use the ‘vfio_regions_ops’ to read/write the BAR?
Also whey in the last of ‘vfio_region_write/read’ we need to call ‘vbasedev->ops->vfio_eoi(vbasedev);’?


Thanks,
Li Qiang


[-- Attachment #2: Type: text/html, Size: 2925 bytes --]

Re: Questions about the VFIO BAR region

[Alex Williamson]

On Tue, 5 Nov 2019 00:40:39 +0800
Li Qiang <liq3ea@163.com> wrote:

> Hello Alex, Auger and all,
> 
> I have a question about the VFIO virtual device BAR.
> 
> In vfio_region_setup, it initialize a ‘region->mem’ MR and set its ops to ‘vfio_regions_ops’. 
> In ‘vfio_region_mmap’, it maps the physical device’s MMIO to QEMU’s virtual address space 
> as a raw MR ‘region->mmaps[i].mem’. 
> And also it set the latter MR as a subregion of the first one.
> 
> So when the guest accesses the BAR, it will direct go to the physical device’s BAR.
> My question is here:
> When the qemu will use the ‘vfio_regions_ops’ to read/write the BAR?
> Also whey in the last of ‘vfio_region_write/read’ we need to call ‘vbasedev->ops->vfio_eoi(vbasedev);’?

We support:

 a) sparse mmaps where the entire BAR is not covered by an mmap
 b) quirks, which layer on top of the mmaps to provide virtualized
    access
 c) INTx emulation which disables mmaps MRs in order to detect device
    access as a generic mechanism for inferring interrupt
    acknowledgment.

The latter being the reason we call vfio_eoi.  Thanks,

Alex

Re: Questions about the VFIO BAR region

[Li Qiang]

[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]

Alex Williamson <alex.williamson@redhat.com> 于2019年11月5日周二 上午2:49写道：

> On Tue, 5 Nov 2019 00:40:39 +0800
> Li Qiang <liq3ea@163.com> wrote:
>
> > Hello Alex, Auger and all,
> >
> > I have a question about the VFIO virtual device BAR.
> >
> > In vfio_region_setup, it initialize a ‘region->mem’ MR and set its ops
> to ‘vfio_regions_ops’.
> > In ‘vfio_region_mmap’, it maps the physical device’s MMIO to QEMU’s
> virtual address space
> > as a raw MR ‘region->mmaps[i].mem’.
> > And also it set the latter MR as a subregion of the first one.
> >
> > So when the guest accesses the BAR, it will direct go to the physical
> device’s BAR.
> > My question is here:
> > When the qemu will use the ‘vfio_regions_ops’ to read/write the BAR?
> > Also whey in the last of ‘vfio_region_write/read’ we need to call
> ‘vbasedev->ops->vfio_eoi(vbasedev);’?
>
> We support:
>
>  a) sparse mmaps where the entire BAR is not covered by an mmap
>

Got.



>  b) quirks, which layer on top of the mmaps to provide virtualized
>     access
>

Do you mean like in 'vfio_probe_ati_bar4_quirk', register a high priority
subregion of VFIORegion.mem.
So when the guest write the BAR, vfio_regions_ops will be used. Here
'quirks' do you mean such things?

static void vfio_probe_ati_bar4_quirk(VFIOPCIDevice *vdev, int nr)
{
    VFIOQuirk *quirk;
    VFIOConfigWindowQuirk *window;

    ...
    memory_region_init_io(window->addr_mem, OBJECT(vdev),
                          &vfio_generic_window_address_quirk, window,
                          "vfio-ati-bar4-window-address-quirk", 4);
    memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
                                        window->address_offset,
                                        window->addr_mem, 1);
   ...
}



>  c) INTx emulation which disables mmaps MRs in order to detect device
>     access as a generic mechanism for inferring interrupt
>     acknowledgment.
>

In the above two cases, in 'vfio_region_write/read' we always access the
physical device's BAR.
So as far as I can understand, the physical device(sometimes) will trigger
interrupts. And the responsible of clear it
will be by the 'guest'. So I can't understand why there calls
'vbasedev->ops->vfio_eoi'. Could you please give me an
example.


Thanks,
Li Qiang



>
> The latter being the reason we call vfio_eoi.  Thanks,
>
> Alex
>
>

[-- Attachment #2: Type: text/html, Size: 3811 bytes --]

Re: Questions about the VFIO BAR region

[Auger Eric]

Hi Li,

On 11/5/19 2:16 AM, Li Qiang wrote:
> 
> 
> Alex Williamson <alex.williamson@redhat.com
> <mailto:alex.williamson@redhat.com>> 于2019年11月5日周二 上午2:49写道：
> 
>     On Tue, 5 Nov 2019 00:40:39 +0800
>     Li Qiang <liq3ea@163.com <mailto:liq3ea@163.com>> wrote:
> 
>     > Hello Alex, Auger and all,
>     >
>     > I have a question about the VFIO virtual device BAR.
>     >
>     > In vfio_region_setup, it initialize a ‘region->mem’ MR and set its
>     ops to ‘vfio_regions_ops’.
>     > In ‘vfio_region_mmap’, it maps the physical device’s MMIO to
>     QEMU’s virtual address space
>     > as a raw MR ‘region->mmaps[i].mem’.
>     > And also it set the latter MR as a subregion of the first one.
>     >
>     > So when the guest accesses the BAR, it will direct go to the
>     physical device’s BAR.
>     > My question is here:
>     > When the qemu will use the ‘vfio_regions_ops’ to read/write the BAR?
>     > Also whey in the last of ‘vfio_region_write/read’ we need to call
>     ‘vbasedev->ops->vfio_eoi(vbasedev);’?
> 
>     We support:
> 
>      a) sparse mmaps where the entire BAR is not covered by an mmap
> 
> 
> Got.
> 
>  
> 
>      b) quirks, which layer on top of the mmaps to provide virtualized
>         access
> 
> 
> Do you mean like in 'vfio_probe_ati_bar4_quirk', register a high
> priority subregion of VFIORegion.mem.
> So when the guest write the BAR, vfio_regions_ops will be used. Here
> 'quirks' do you mean such things?
> 
> static void vfio_probe_ati_bar4_quirk(VFIOPCIDevice *vdev, int nr)
> {
>     VFIOQuirk *quirk;
>     VFIOConfigWindowQuirk *window;
> 
>     ...
>     memory_region_init_io(window->addr_mem, OBJECT(vdev),
>                           &vfio_generic_window_address_quirk, window,
>                           "vfio-ati-bar4-window-address-quirk", 4);
>     memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
>                                         window->address_offset,
>                                         window->addr_mem, 1);
>    ...
> }
Yes that's it. In that case vfio_generic_window_address_quirk ops get
called when attempting to access this overlapping region.
> 
>  
> 
>      c) INTx emulation which disables mmaps MRs in order to detect device
>         access as a generic mechanism for inferring interrupt
>         acknowledgment.
> 
> 
> In the above two cases, in 'vfio_region_write/read' we always access the
> physical device's BAR.
> So as far as I can understand, the physical device(sometimes) will
> trigger interrupts. And the responsible of clear it 
> will be by the 'guest'. So I can't understand why there calls
> 'vbasedev->ops->vfio_eoi'. Could you please give me an
> example.
When a physical level sensitive IRQ hits it is signaled through an
eventfd. The eventfd can be handled by QEMU. In this case,
vfio_intx_interrupt gets called, for PCI. It turns the mmap off (slow
path on) and injects the corresponding virtual IRQ into the guest. The
reason why we turn the mmap off is we need to trap the guest end of
interrupt to eventually deactivate the IRQ at physical level and unmask
it (it was auto-masked by the vfio driver). The first access into the
region is assumed to correspond to the servicing of the pending
interrupt by the guest handler (pending status clear) and at this point
we deactivate the physical IRQ. So .vfio_eoi gets called on the first
read/write BAR access after an INTx gets pending. Note the mmapping
(fast path) is not immediatly turned on after deactivating the physical
INTx. See comment before vfio_intx_mmap_enable.

When an irqfd/resamplefd is used we do not need that trick as the guest
EOI is trapped at KVM level through the virtual interrupt controller.
when the guest EOI is trapped KVM deactivates the associated physical
IRQ and notifies the userspace through the resamplefd for this latter to
unmask the physical IRQ.

Hope this helps.

Thanks

Eric




> 
> 
> Thanks,
> Li Qiang
> 
>  
> 
> 
>     The latter being the reason we call vfio_eoi.  Thanks,
> 
>     Alex
> 

Re: Questions about the VFIO BAR region

[Li Qiang]

[-- Attachment #1: Type: text/plain, Size: 4574 bytes --]

Auger Eric <eric.auger@redhat.com> 于2019年11月5日周二 下午9:17写道：

> Hi Li,
>
> On 11/5/19 2:16 AM, Li Qiang wrote:
> >
> >
> > Alex Williamson <alex.williamson@redhat.com
> > <mailto:alex.williamson@redhat.com>> 于2019年11月5日周二 上午2:49写道：
> >
> >     On Tue, 5 Nov 2019 00:40:39 +0800
> >     Li Qiang <liq3ea@163.com <mailto:liq3ea@163.com>> wrote:
> >
> >     > Hello Alex, Auger and all,
> >     >
> >     > I have a question about the VFIO virtual device BAR.
> >     >
> >     > In vfio_region_setup, it initialize a ‘region->mem’ MR and set its
> >     ops to ‘vfio_regions_ops’.
> >     > In ‘vfio_region_mmap’, it maps the physical device’s MMIO to
> >     QEMU’s virtual address space
> >     > as a raw MR ‘region->mmaps[i].mem’.
> >     > And also it set the latter MR as a subregion of the first one.
> >     >
> >     > So when the guest accesses the BAR, it will direct go to the
> >     physical device’s BAR.
> >     > My question is here:
> >     > When the qemu will use the ‘vfio_regions_ops’ to read/write the
> BAR?
> >     > Also whey in the last of ‘vfio_region_write/read’ we need to call
> >     ‘vbasedev->ops->vfio_eoi(vbasedev);’?
> >
> >     We support:
> >
> >      a) sparse mmaps where the entire BAR is not covered by an mmap
> >
> >
> > Got.
> >
> >
> >
> >      b) quirks, which layer on top of the mmaps to provide virtualized
> >         access
> >
> >
> > Do you mean like in 'vfio_probe_ati_bar4_quirk', register a high
> > priority subregion of VFIORegion.mem.
> > So when the guest write the BAR, vfio_regions_ops will be used. Here
> > 'quirks' do you mean such things?
> >
> > static void vfio_probe_ati_bar4_quirk(VFIOPCIDevice *vdev, int nr)
> > {
> >     VFIOQuirk *quirk;
> >     VFIOConfigWindowQuirk *window;
> >
> >     ...
> >     memory_region_init_io(window->addr_mem, OBJECT(vdev),
> >                           &vfio_generic_window_address_quirk, window,
> >                           "vfio-ati-bar4-window-address-quirk", 4);
> >     memory_region_add_subregion_overlap(vdev->bars[nr].region.mem,
> >                                         window->address_offset,
> >                                         window->addr_mem, 1);
> >    ...
> > }
> Yes that's it. In that case vfio_generic_window_address_quirk ops get
> called when attempting to access this overlapping region.
> >
> >
> >
> >      c) INTx emulation which disables mmaps MRs in order to detect device
> >         access as a generic mechanism for inferring interrupt
> >         acknowledgment.
> >
> >
> > In the above two cases, in 'vfio_region_write/read' we always access the
> > physical device's BAR.
> > So as far as I can understand, the physical device(sometimes) will
> > trigger interrupts. And the responsible of clear it
> > will be by the 'guest'. So I can't understand why there calls
> > 'vbasedev->ops->vfio_eoi'. Could you please give me an
> > example.
> When a physical level sensitive IRQ hits it is signaled through an
> eventfd. The eventfd can be handled by QEMU. In this case,
> vfio_intx_interrupt gets called, for PCI. It turns the mmap off (slow
> path on) and injects the corresponding virtual IRQ into the guest. The
> reason why we turn the mmap off is we need to trap the guest end of
> interrupt to eventually deactivate the IRQ at physical level and unmask
> it (it was auto-masked by the vfio driver). The first access into the
> region is assumed to correspond to the servicing of the pending
> interrupt by the guest handler (pending status clear) and at this point
> we deactivate the physical IRQ. So .vfio_eoi gets called on the first
> read/write BAR access after an INTx gets pending. Note the mmapping
> (fast path) is not immediatly turned on after deactivating the physical
> INTx. See comment before vfio_intx_mmap_enable.
>
> When an irqfd/resamplefd is used we do not need that trick as the guest
> EOI is trapped at KVM level through the virtual interrupt controller.
> when the guest EOI is trapped KVM deactivates the associated physical
> IRQ and notifies the userspace through the resamplefd for this latter to
> unmask the physical IRQ.
>
> Hope this helps.
>


Thank you Auger, very detailed information.

Thanks,
Li Qiang



>
> Thanks
>
> Eric
>
>
>
>
> >
> >
> > Thanks,
> > Li Qiang
> >
> >
> >
> >
> >     The latter being the reason we call vfio_eoi.  Thanks,
> >
> >     Alex
> >
>
>

[-- Attachment #2: Type: text/html, Size: 6197 bytes --]