Re: performance of virtual functions compared to virtio

[David Ahern <dsahern@gmail.com>]

On 04/25/11 12:13, Alex Williamson wrote:
>> So, basically, 192.168.102 is the network where the VMs have a VF, and
>> 192.168.103 is the network where the VMs use virtio for networking.
>>
>> The netperf commands are all run on either Host-A or VM-C:
>>
>>   netperf -H $ip -jcC -v 2 -t TCP_RR      -- -r 1024 -D L,R
>>   netperf -H $ip -jcC -v 2 -t TCP_STREAM  -- -m 1024 -D L,R
>>
>>
>>                    latency      throughput
>>                     (usec)         Mbps
>> cross-host:
>>   A-B, eth2          185            932
>>   A-B, eth3          185            935
> 
> This is actually PF-PF, right?  It would be interesting to load igbvf on
> the hosts and determine VF-VF latency as well.

yes, PF-PF. eth3 has the added bridge layer, but from what I can see the
overhead is noise. I added host-to-host to put the host-to-VM numbers in
perspective.

> 
>> same host, host-VM:
>>   A-C, using VF      488           1085 (seen as high as 1280's)
>>   A-C, virtio        150           4282
> 
> We know virtio has a shorter path for this test.

No complaints about the throughput numbers; the latency is the problem.

> 
>> cross-host, host-VM:
>>   A-D, VF            489            938
>>   A-D, virtio        288            889
>>
>> cross-host, VM-VM:
>>   C-D, VF            488            934
>>   C-D, virtio        490            933
>>
>>
>> While throughput for VFs is fine (near line-rate when crossing hosts),
> 
> FWIW, it's not too difficult to get line rate on a 1Gbps network, even
> some of the emulated NICs can do it.  There will be a difference in host
> CPU power to get it though, where it should theoretically be emulated >
> virtio > pci-assign.

10GB is the goal; 1GB offers a cheaper learning environment. ;-)

> 
>> the latency is horrible. Any options to improve that?
> 
> If you don't mind testing, I'd like to see VF-VF between the hosts (to
> do this, don't assign eth2 an IP, just make sure it's up, then load the
> igbvf driver on the host and assign an IP to one of the VFs associated
> with the eth2 PF), and cross host testing using the PF for the guest
> instead of the VF.  This should help narrow down how much of the latency
> is due to using the VF vs the PF, since all of the virtio tests are
> using the PF.  I've been suspicious that the VF adds some latency, but
> haven't had a good test setup (or time) to dig very deep into it.

It's a quad nic, so I left eth2 and eth3 alone and added the VF-VF test
using VFs on eth4.

Indeed latency is 488 usec and throughput is 925 Mbps. This is
host-to-host using VFs.

David

> Thanks,
> 
> Alex
>