Re: performance of virtual functions compared to virtio

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 2011-04-25 at 11:39 -0600, David Ahern wrote:
> On 04/20/11 20:35, Alex Williamson wrote:
> > Device assignment via a VF provides the lowest latency and most
> > bandwidth for *getting data off the host system*, though virtio/vhost is
> > getting better.  If all you care about is VM-VM on the same host or
> > VM-host, then virtio is only limited by memory bandwidth/latency and
> > host processor cycles.  Your processor has 25GB/s of memory bandwidth.
> > On the other hand, the VF has to send data all the way out to the wire
> > and all the way back up through the NIC to get to the other VM/host.
> > You're using a 1Gb/s NIC.  Your results actually seem to indicate you're
> > getting better than wire rate, so maybe you're only passing through an
> > internal switch on the NIC, in any case, VFs are not optimal for
> > communication within the same physical system.  They are optimal for off
> > host communication.  Thanks,
> 
> Hi Alex:
> 
> Host-host was the next focus for the tests. I have 2 of the
> aforementioned servers, each configured identically. As a reminder:
> 
> Host:
>   Dell R410
>   2 quad core E5620@2.40 GHz processors
>   16 GB RAM
>   Intel 82576 NIC (Gigabit ET Quad Port)
>   - devices eth2, eth3, eth4, eth5
>   Fedora 14
>   kernel: 2.6.35.12-88.fc14.x86_64
>   qemu-kvm.git, ffce28fe6 (18-April-11)
> 
> VMs:
>   Fedora 14
>   kernel 2.6.35.11-83.fc14.x86_64
>   2 vcpus
>   1GB RAM
>   2 NICs - 1 virtio, 1 VF
> 
> The virtio network arguments to qemu-kvm are:
>   -netdev type=tap,vhost=on,ifname=tap0,id=netdev1
>   -device virtio-net-pci,mac=${mac},netdev=netdev1
> 
> 
> For this round of tests I have the following setup:
> 
>   .======================================.
>   | Host - A                             |
>   |                                      |
>   |      .-------------------------.     |
>   |      |  Virtual Machine - C    |     |
>   |      |                         |     |
>   |      |  .------.     .------.  |     |
>   |      '--| eth1 |-----| eth0 |--'     |
>   |         '------'     '------'        |
>   |   192.168. |             | 192.168.103.71
>   |     102.71 |          .------.       |
>   |            |          | tap0 |       |
>   |            |          '------'       |
>   |            |             |           |
>   |            |          .------.       |
>   |            |          |  br  | 192.168.103.79
>   |            |          '------'       |
>   |           {VF}           |           |
>   |       .--------.      .------.       |
>   '=======|  eth2  |======| eth3 |======='
>           '--------'      '------'
> 192.168.102.79 |             |
>                | point-to-   |
>                |       point |
>                | connections |
> 192.168.102.80 |             |
>           .--------.      .------.
>   .=======|  eth2  |======| eth3 |=======.
>   |       '--------'      '------'       |
>   |           {VF}           |           |
>   |            |          .------.       |
>   |            |          |  br  | 192.168.103.80
>   |            |          '------'       |
>   |            |             |           |
>   |            |          .------.       |
>   |            |          | tap0 |       |
>   |   192.168. |          '------'       |
>   |     102.81 |             | 192.168.103.81
>   |         .------.     .------.        |
>   |      .--| eth1 |-----| eth0 |--.     |
>   |      |  '------'     '------'  |     |
>   |      |                         |     |
>   |      |  Virtual Machine - D    |     |
>   |      '-------------------------'     |
>   |                                      |
>   | Host - B                             |
>   '======================================'
> 
> 
> So, basically, 192.168.102 is the network where the VMs have a VF, and
> 192.168.103 is the network where the VMs use virtio for networking.
> 
> The netperf commands are all run on either Host-A or VM-C:
> 
>   netperf -H $ip -jcC -v 2 -t TCP_RR      -- -r 1024 -D L,R
>   netperf -H $ip -jcC -v 2 -t TCP_STREAM  -- -m 1024 -D L,R
> 
> 
>                    latency      throughput
>                     (usec)         Mbps
> cross-host:
>   A-B, eth2          185            932
>   A-B, eth3          185            935

This is actually PF-PF, right?  It would be interesting to load igbvf on
the hosts and determine VF-VF latency as well.

> same host, host-VM:
>   A-C, using VF      488           1085 (seen as high as 1280's)
>   A-C, virtio        150           4282

We know virtio has a shorter path for this test.

> cross-host, host-VM:
>   A-D, VF            489            938
>   A-D, virtio        288            889
> 
> cross-host, VM-VM:
>   C-D, VF            488            934
>   C-D, virtio        490            933
> 
> 
> While throughput for VFs is fine (near line-rate when crossing hosts),

FWIW, it's not too difficult to get line rate on a 1Gbps network, even
some of the emulated NICs can do it.  There will be a difference in host
CPU power to get it though, where it should theoretically be emulated >
virtio > pci-assign.

> the latency is horrible. Any options to improve that?

If you don't mind testing, I'd like to see VF-VF between the hosts (to
do this, don't assign eth2 an IP, just make sure it's up, then load the
igbvf driver on the host and assign an IP to one of the VFs associated
with the eth2 PF), and cross host testing using the PF for the guest
instead of the VF.  This should help narrow down how much of the latency
is due to using the VF vs the PF, since all of the virtio tests are
using the PF.  I've been suspicious that the VF adds some latency, but
haven't had a good test setup (or time) to dig very deep into it.
Thanks,

Alex