Re: performance of virtual functions compared to virtio

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 2011-04-20 at 19:57 -0600, David Ahern wrote:
> In general should virtual functions outperform virtio+vhost for
> networking performance - latency and throughput?
> 
> I have 2 VMs running on a host. Each VM has 2 nics -- one tied to a VF
> and the other going through virtio and a tap device like so:
> 
>    ------                  ----
>   |      |----------------| VF |---
>   |      |                 ----   |
>   | VM 1 |                        |
>   |      |    -----               |
>   |      |---| tap |---           |
>    ------     -----   |          ---
>                      ---        | e |
>                     | b |       | t |
>                     | r |       | h |
>                      ---        | 2 |
>    ------     -----   |          ---
>   |      |---| tap |---           |
>   |      |    -----               |
>   | VM 2 |                        |
>   |      |                 ----   |
>   |      |----------------| VF |---
>    ------                  ----
> 
> The network arguments to qemu-kvm are:
> -netdev type=tap,vhost=on,ifname=tap2,id=netdev1
> -device virtio-net-pci,mac=${mac},netdev=netdev1
> 
> where ${mac} is unique to each VM and for the VF:
> -device pci-assign,host=${pciid}
> 
> netserver is running within the VMs, and the netperf commands I am
> running are:
> 
>   netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_RR -- -r 1024
>   netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_STREAM
> 
> where <ip> changes depending on which interface I want to send the
> traffic through. To say the least results are a bit disappointing for
> the VF:
> 
>                   latency   throughput
>               (usec/Tran)   (MB/sec)
> Host-VM
>  over virtio      139.160    1199.40
>  over VF          488.124     209.22
> 
> VM-VM
>  over virtio      322.056     773.54
>  over VF          488.051     328.88
> 
> I am just getting started with VFs and could use some hints on how to
> improve the performance.

Device assignment via a VF provides the lowest latency and most
bandwidth for *getting data off the host system*, though virtio/vhost is
getting better.  If all you care about is VM-VM on the same host or
VM-host, then virtio is only limited by memory bandwidth/latency and
host processor cycles.  Your processor has 25GB/s of memory bandwidth.
On the other hand, the VF has to send data all the way out to the wire
and all the way back up through the NIC to get to the other VM/host.
You're using a 1Gb/s NIC.  Your results actually seem to indicate you're
getting better than wire rate, so maybe you're only passing through an
internal switch on the NIC, in any case, VFs are not optimal for
communication within the same physical system.  They are optimal for off
host communication.  Thanks,

Alex