Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device

Yunhong Jiang
2015-12-03T22:45:22Z

On Thu, Dec 03, 2015 at 11:55:53AM -0700, Alex Williamson wrote:
> On Thu, 2015-12-03 at 10:22 -0800, Yunhong Jiang wrote:
> > When assigning a VFIO device to a KVM guest with low latency requirement, it
> > is better to handle the interrupt in the hard interrupt context, to reduce
> > the context switch to/from the IRQ thread.
> >
> > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi
> > interrupt is changed to use request_threaded_irq(). The primary interrupt
> > handler tries to set the guest interrupt atomically. If it fails to achieve
> > it, a threaded interrupt handler will be invoked.
> >
> > The irq_bypass manager is extended for this purpose. The KVM eventfd will
> > provide a irqbypass consumer to handle the interrupt at hard interrupt
> > context. The producer will invoke the consumer's handler then.
>
> Do you have any performance data? Thanks,

Sorry, I should include the data on the commit messages.

The test:
Launch a VM with a FPGA device, which triggers an interrpt every 1ms. The VM
is launched on a isolated CPU with NONHZ_FULL enabled.
Two data are collected. On the guest, a special program will check the
latency from the time the interrupt been triggered to the time the interrupt
received by guest IRS. On the host, I use the perf to collect the perf data.

The performance Data with the patch:

Host perf data:
[root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
[root@otcnfv02 test-tools]# perf kvm stat report
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max Time
Avg time
EXTERNAL_INTERRUPT 9997 98.55% 99.31% 1.73us 17.07us
2.09us ( +- 0.21% )
PAUSE_INSTRUCTION 127 1.25% 0.51% 0.69us 1.20us
0.84us ( +- 1.34% )
MSR_WRITE 20 0.20% 0.18% 1.62us 3.21us
1.93us ( +- 3.95% )

Guest data:
[nfv@rt-test-1 ~]$ ./run-int-test.sh
Latency is Min: 3.74us Max: 20.08us Avg: 4.49us
No of interrupts = 74995

The performance data without the patch:
Host perf data:
[root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10
[root@otcnfv02 test-tools]# perf kvm stat report
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max Time
Avg time
PAUSE_INSTRUCTION 141136 87.74% 50.39% 0.69us 8.51us
0.77us ( +- 0.07% )
EXTERNAL_INTERRUPT 19701 12.25% 49.59% 2.31us 15.93us
5.46us ( +- 0.12% )
MSR_WRITE 24 0.01% 0.02% 1.51us 2.22us
1.91us ( +- 1.91% )

Notice:
The EXTERNAL_INTERRUPT VMExit is different w/ the patch (9997) and w/o the
patch (19701). It is because with threaded IRQ, the NOHZ_FULL it not working
because two threads on the pCPU, thus we have both the FPGA device interrupt
and the tick timer interrupt. After calculation, the average time for the
FPGA device interrupt is 4.72 us.

Guest data:
[nfv@rt-test-1 ~]$ ./run-int-test.sh
Latency is Min: 6.70us Max: 50.38us Avg: 7.44us
No of interrupts = 42639

Thanks
--jyh

>
> Alex
>