[Qemu-devel] Question about the vfio device interrupt

[Qemu-devel] Question about the vfio device interrupt

[Li Qiang]

Hello Paolo, Alex and all,

In vfio_intx_enable_kvm(qemu) I see we associate a resamplefd with the
umask function(vfio_pci_intx_unmask_handler in linux).

    irq_set = g_malloc0(argsz);
    irq_set->argsz = argsz;
    irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
    irq_set->index = VFIO_PCI_INTX_IRQ_INDEX;
    irq_set->start = 0;
    irq_set->count = 1;
    pfd = (int32_t *)&irq_set->data;

    *pfd = irqfd.resamplefd;

    ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);

I know this resamplefd is triggered when the interrupt controller receives
an EOI and
then we unmask and re-enabled the VFIO devices' interrupt.

So I think there  there must be a mask process(so we unmask it later), I
mean there must a
call of function vfio_pci_set_intx_mask(in linux).

What I can't understand is when this process(mask VFIO devices' interrupt)
occurs?
I only find a place(vfio_basic_config_write) calls vfio_pci_intx_mask.
I think when the guest process the interrupt it may mask this
interrupt(finally call vfio_pci_set_intx_mask), but I can't find the code
path related with vfio_basic_config_write.


Any hints?

Thanks,
Li Qiang

Re: [Qemu-devel] Question about the vfio device interrupt

[Auger Eric]

Hi,

On 5/15/19 12:01 PM, Li Qiang wrote:
> Hello Paolo, Alex and all,
> 
> In vfio_intx_enable_kvm(qemu) I see we associate a resamplefd with the
> umask function(vfio_pci_intx_unmask_handler in linux).
> 
>     irq_set = g_malloc0(argsz);
>     irq_set->argsz = argsz;
>     irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK;
>     irq_set->index = VFIO_PCI_INTX_IRQ_INDEX;
>     irq_set->start = 0;
>     irq_set->count = 1;
>     pfd = (int32_t *)&irq_set->data;
> 
>     *pfd = irqfd.resamplefd;
> 
>     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
> 
> I know this resamplefd is triggered when the interrupt controller receives
> an EOI and
> then we unmask and re-enabled the VFIO devices' interrupt.
> 
> So I think there  there must be a mask process(so we unmask it later), I
> mean there must a
> call of function vfio_pci_set_intx_mask(in linux).
> 
> What I can't understand is when this process(mask VFIO devices' interrupt)
> occurs?
It is done in the VFIO interrupt handler:
vfio_intx_handler/disable_irq_nosync for VFIO PCI (vfio_pci_intrs.c) or
vfio_automasked_irq_handler (vfio_platform_irq.c) for VFIO PLATFORM .

Thanks

Eric

> I only find a place(vfio_basic_config_write) calls vfio_pci_intx_mask> I think when the guest process the interrupt it may mask this
> interrupt(finally call vfio_pci_set_intx_mask), but I can't find the code
> path related with vfio_basic_config_write.
> 
> 
> Any hints?
> 
> Thanks,
> Li Qiang
> 

Re: [Qemu-devel] Question about the vfio device interrupt

[Li Qiang]

Auger Eric <eric.auger@redhat.com> 于2019年5月15日周三 下午6:16写道：

> Hi,
>
> On 5/15/19 12:01 PM, Li Qiang wrote:
> > Hello Paolo, Alex and all,
> >
> > In vfio_intx_enable_kvm(qemu) I see we associate a resamplefd with the
> > umask function(vfio_pci_intx_unmask_handler in linux).
> >
> >     irq_set = g_malloc0(argsz);
> >     irq_set->argsz = argsz;
> >     irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |
> VFIO_IRQ_SET_ACTION_UNMASK;
> >     irq_set->index = VFIO_PCI_INTX_IRQ_INDEX;
> >     irq_set->start = 0;
> >     irq_set->count = 1;
> >     pfd = (int32_t *)&irq_set->data;
> >
> >     *pfd = irqfd.resamplefd;
> >
> >     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
> >
> > I know this resamplefd is triggered when the interrupt controller
> receives
> > an EOI and
> > then we unmask and re-enabled the VFIO devices' interrupt.
> >
> > So I think there  there must be a mask process(so we unmask it later), I
> > mean there must a
> > call of function vfio_pci_set_intx_mask(in linux).
> >
> > What I can't understand is when this process(mask VFIO devices'
> interrupt)
> > occurs?
> It is done in the VFIO interrupt handler:
> vfio_intx_handler/disable_irq_nosync for VFIO PCI (vfio_pci_intrs.c) or
> vfio_automasked_irq_handler (vfio_platform_irq.c) for VFIO PLATFORM .
>
>
Thanks Eric,

Got it.
I just forget to look at the interrupt handler(vfio_intx_handler)!


Thanks,
Li Qiang



> Thanks
>
> Eric
>
> > I only find a place(vfio_basic_config_write) calls vfio_pci_intx_mask> I
> think when the guest process the interrupt it may mask this
> > interrupt(finally call vfio_pci_set_intx_mask), but I can't find the code
> > path related with vfio_basic_config_write.
> >
> >
> > Any hints?
> >
> > Thanks,
> > Li Qiang
> >
>

Re: [Qemu-devel] Question about the vfio device interrupt

[Alex Williamson]

On Wed, 15 May 2019 18:22:54 +0800
Li Qiang <liq3ea@gmail.com> wrote:

> Auger Eric <eric.auger@redhat.com> 于2019年5月15日周三 下午6:16写道：
> 
> > Hi,
> >
> > On 5/15/19 12:01 PM, Li Qiang wrote:  
> > > Hello Paolo, Alex and all,
> > >
> > > In vfio_intx_enable_kvm(qemu) I see we associate a resamplefd with the
> > > umask function(vfio_pci_intx_unmask_handler in linux).
> > >
> > >     irq_set = g_malloc0(argsz);
> > >     irq_set->argsz = argsz;
> > >     irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |  
> > VFIO_IRQ_SET_ACTION_UNMASK;  
> > >     irq_set->index = VFIO_PCI_INTX_IRQ_INDEX;
> > >     irq_set->start = 0;
> > >     irq_set->count = 1;
> > >     pfd = (int32_t *)&irq_set->data;
> > >
> > >     *pfd = irqfd.resamplefd;
> > >
> > >     ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set);
> > >
> > > I know this resamplefd is triggered when the interrupt controller  
> > receives  
> > > an EOI and
> > > then we unmask and re-enabled the VFIO devices' interrupt.
> > >
> > > So I think there  there must be a mask process(so we unmask it later), I
> > > mean there must a
> > > call of function vfio_pci_set_intx_mask(in linux).
> > >
> > > What I can't understand is when this process(mask VFIO devices'  
> > interrupt)  
> > > occurs?  
> > It is done in the VFIO interrupt handler:
> > vfio_intx_handler/disable_irq_nosync for VFIO PCI (vfio_pci_intrs.c) or
> > vfio_automasked_irq_handler (vfio_platform_irq.c) for VFIO PLATFORM .
> >
> >  
> Thanks Eric,
> 
> Got it.
> I just forget to look at the interrupt handler(vfio_intx_handler)!

Importantly, PCI INTx interrupts are exposed in
VFIO_DEVICE_GET_IRQ_INFO with the VFIO_IRQ_INFO_AUTOMASKED flag which
indicates that the interrupt is masked in the host kernel when it is
signaled via the eventfd and userspace is only required to perform an
unmask operation to receive the next interrupt.  Thanks,

Alex

Re: [Qemu-devel] Question about the vfio device interrupt

[Maxim Levitsky]

Hi!

Let me add my 0.2 cents to this discussion by sharing my summary of INTx emulation in VFIO:

COMMON:
* real interrupt is signaled in the kernel
* vfio-pci kernel interrupt hanlder (vfio_intx_handler), masks the INTx flag
* vfio-pci kernel interrupt handler triggers an eventfd, if the interrupt is not virtually intx masked
  (an eventfd which was registred for VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER)


KERNEL INTERRUPT CONTROLLER CASE:

* the vfio-pci eventfd is listened by the kvm (set by KVM_IRQFD ioctl), and delivered to guest ioapic
  (the setup is done by qemu in vfio_intx_enable_kvm, the same eventfd is given to vfio-pci and to kvm)
* when guest writes eoi, an 'resamplefd' is (which is registered by vfio_intx_enable_kvm too)
  is triggered.
  the vfio_intx_enable_kvm tells the vfio pci module (VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK)
  to listen to that 'resamplefd' and unmask the interrupt
  (here too, a same eventfd is given to kvm and to vfio-pci, but here the kvm signals it, and vfio-pci listens to it).


USERSPACE QEMU CASE:

  (only happens in short period while vfio_intx_enable is running and before it calles the vfio_intx_enable_kvm)
  (or when vfio_intx_enable_kvm fails)

* the vfio-pci eventfd is listened by the qemu (vfio_intx_interrupt)
* the vfio_intx_interrupt (in qemu)
	* masks the interrupt using kernel vfio interface
	* unmaps all the device bars and sets a timer to map them again
	* sends the interrupt down the qemu stack till it is reinjected to the guest 
* any bar access now goes to qemu via EPT fault (vfio_region_read/vfio_region_write)
* the bar access 
	* triggers (vfio_eoi->vfio_intx_eoi) which unmasks (using vfio-pci) the interrupt
  	  the idea is that first few bar accesses should ack the interrupt and thus avoid flood
  	  but some interrupts could still happen (one per bar access)

Best regards,
	Maxim Levitsky

Re: [Qemu-devel] Question about the vfio device interrupt

[Alex Williamson]

On Wed, 15 May 2019 17:27:52 +0300
Maxim Levitsky <mlevitsk@redhat.com> wrote:

> Hi!
> 
> Let me add my 0.2 cents to this discussion by sharing my summary of INTx emulation in VFIO:
> 
> COMMON:
> * real interrupt is signaled in the kernel
> * vfio-pci kernel interrupt hanlder (vfio_intx_handler), masks the INTx flag
> * vfio-pci kernel interrupt handler triggers an eventfd, if the interrupt is not virtually intx masked
>   (an eventfd which was registred for VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER)
> 
> 
> KERNEL INTERRUPT CONTROLLER CASE:
> 
> * the vfio-pci eventfd is listened by the kvm (set by KVM_IRQFD ioctl), and delivered to guest ioapic
>   (the setup is done by qemu in vfio_intx_enable_kvm, the same eventfd is given to vfio-pci and to kvm)
> * when guest writes eoi, an 'resamplefd' is (which is registered by vfio_intx_enable_kvm too)
>   is triggered.
>   the vfio_intx_enable_kvm tells the vfio pci module (VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK)
>   to listen to that 'resamplefd' and unmask the interrupt
>   (here too, a same eventfd is given to kvm and to vfio-pci, but here the kvm signals it, and vfio-pci listens to it).
> 
> 
> USERSPACE QEMU CASE:
> 
>   (only happens in short period while vfio_intx_enable is running and before it calles the vfio_intx_enable_kvm)
>   (or when vfio_intx_enable_kvm fails)
> 
> * the vfio-pci eventfd is listened by the qemu (vfio_intx_interrupt)
> * the vfio_intx_interrupt (in qemu)
> 	* masks the interrupt using kernel vfio interface
> 	* unmaps all the device bars and sets a timer to map them again
> 	* sends the interrupt down the qemu stack till it is reinjected to the guest 
> * any bar access now goes to qemu via EPT fault (vfio_region_read/vfio_region_write)
> * the bar access 
> 	* triggers (vfio_eoi->vfio_intx_eoi) which unmasks (using vfio-pci) the interrupt
>   	  the idea is that first few bar accesses should ack the interrupt and thus avoid flood
>   	  but some interrupts could still happen (one per bar access)

Full points :)  Thanks,

Alex

Re: [Qemu-devel] Question about the vfio device interrupt

[Li Qiang]

Alex Williamson <alex.williamson@redhat.com> 于2019年5月15日周三 下午10:54写道：

> On Wed, 15 May 2019 17:27:52 +0300
> Maxim Levitsky <mlevitsk@redhat.com> wrote:
>
> > Hi!
> >
> > Let me add my 0.2 cents to this discussion by sharing my summary of INTx
> emulation in VFIO:
> >
> > COMMON:
> > * real interrupt is signaled in the kernel
> > * vfio-pci kernel interrupt hanlder (vfio_intx_handler), masks the INTx
> flag
> > * vfio-pci kernel interrupt handler triggers an eventfd, if the
> interrupt is not virtually intx masked
> >   (an eventfd which was registred for VFIO_IRQ_SET_DATA_EVENTFD |
> VFIO_IRQ_SET_ACTION_TRIGGER)
> >
> >
> > KERNEL INTERRUPT CONTROLLER CASE:
> >
> > * the vfio-pci eventfd is listened by the kvm (set by KVM_IRQFD ioctl),
> and delivered to guest ioapic
> >   (the setup is done by qemu in vfio_intx_enable_kvm, the same eventfd
> is given to vfio-pci and to kvm)
> > * when guest writes eoi, an 'resamplefd' is (which is registered by
> vfio_intx_enable_kvm too)
> >   is triggered.
> >   the vfio_intx_enable_kvm tells the vfio pci module
> (VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_UNMASK)
> >   to listen to that 'resamplefd' and unmask the interrupt
> >   (here too, a same eventfd is given to kvm and to vfio-pci, but here
> the kvm signals it, and vfio-pci listens to it).
> >
> >
> > USERSPACE QEMU CASE:
> >
> >   (only happens in short period while vfio_intx_enable is running and
> before it calles the vfio_intx_enable_kvm)
> >   (or when vfio_intx_enable_kvm fails)
> >
> > * the vfio-pci eventfd is listened by the qemu (vfio_intx_interrupt)
> > * the vfio_intx_interrupt (in qemu)
> >       * masks the interrupt using kernel vfio interface
> >       * unmaps all the device bars and sets a timer to map them again
> >       * sends the interrupt down the qemu stack till it is reinjected to
> the guest
> > * any bar access now goes to qemu via EPT fault
> (vfio_region_read/vfio_region_write)
> > * the bar access
> >       * triggers (vfio_eoi->vfio_intx_eoi) which unmasks (using
> vfio-pci) the interrupt
> >         the idea is that first few bar accesses should ack the interrupt
> and thus avoid flood
> >         but some interrupts could still happen (one per bar access)
>
> Full points :)  Thanks,
>


Thanks very much Alex and Maxim!
I appreciate it.

Thanks,
Li Qiang




>
> Alex
>