Re: PCI passthrough with VT-d - native performance

Ben-Ami Yassour
2008-07-16T15:19:27Z

On Wed, 2008-07-16 at 17:36 +0300, Avi Kivity wrote:
> Ben-Ami Yassour wrote:
> > In last few tests that we made with PCI-passthrough and VT-d using
> > iperf, we were able to get the same throughput as on native OS with a 1G
> > NIC
>
> Excellent!
>
> > (with higher CPU utilization).
> >
>
> How much higher?

Here are some numbers for running iperf -l 1M:

e1000 NIC (behind a PCI bridge)
Bandwidth (Mbit/sec) CPU utilization
Native OS 771 18%
Native OS with VT-d 760 18%
KVM VT-d 390 95%
﻿KVM VT-d with direct mmio 770 84%
KVM emulated 57 100%

Comment: its not clear to me why the native linux can not get closer to 1G for this NIC,
(I verified that its not external network issues). But clearly we shouldn't hope to
get more then the host does with a KVM guest (especially if the guest and host are the
same OS as in this case...).

e1000e NIC (onboard)
Bandwidth (Mbit/sec) CPU utilization
Native OS 915 18%
Native OS with VT-d 915 18%
﻿KVM VT-d with direct mmio 914 98%

Clearly we need to try and improve the CPU utilization, but I think that this is good enough
for the first phase.

>
> > The following patches are the PCI-passthrough patches that Amit sent
> > (re-based on the last kvm tree), followed by a few improvements and the
> > VT-d extension.
> > I am also sending the userspace patches: the patch that Amit sent for
> > PCI passthrough and the direct-mmio extension for userspace (note that
> > without the direct mmio extension we get less then half the throughput).
> >
>
> Is mmio passthrough the reason for the performance improvement? If not,
> what was the problem?
>
Direct mmio was definitely a major improvement, without it we got half the throughput,
as you can see above.
In addition patch 4/8 improves the interrupt handling and removes unnecessary locks,
and I assume that it also fixed performance issues (I did not investigate exactly in what way).

Regards,
Ben