performance of virtual functions compared to virtio

[David Ahern <dsahern@gmail.com>]

In general should virtual functions outperform virtio+vhost for
networking performance - latency and throughput?

I have 2 VMs running on a host. Each VM has 2 nics -- one tied to a VF
and the other going through virtio and a tap device like so:

   ------                  ----
  |      |----------------| VF |---
  |      |                 ----   |
  | VM 1 |                        |
  |      |    -----               |
  |      |---| tap |---           |
   ------     -----   |          ---
                     ---        | e |
                    | b |       | t |
                    | r |       | h |
                     ---        | 2 |
   ------     -----   |          ---
  |      |---| tap |---           |
  |      |    -----               |
  | VM 2 |                        |
  |      |                 ----   |
  |      |----------------| VF |---
   ------                  ----

The network arguments to qemu-kvm are:
-netdev type=tap,vhost=on,ifname=tap2,id=netdev1
-device virtio-net-pci,mac=${mac},netdev=netdev1

where ${mac} is unique to each VM and for the VF:
-device pci-assign,host=${pciid}

netserver is running within the VMs, and the netperf commands I am
running are:

  netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_RR -- -r 1024
  netperf -p 12346 -H <ip> -l 20 -jcC -fM -v 2 -t TCP_STREAM

where <ip> changes depending on which interface I want to send the
traffic through. To say the least results are a bit disappointing for
the VF:

                  latency   throughput
              (usec/Tran)   (MB/sec)
Host-VM
 over virtio      139.160    1199.40
 over VF          488.124     209.22

VM-VM
 over virtio      322.056     773.54
 over VF          488.051     328.88

I am just getting started with VFs and could use some hints on how to
improve the performance.

Host:
  Dell R410
  2 quad core E5620@2.40 GHz processors
  16 GB RAM
  Intel 82576 NIC (Gigabit ET Quad Port)
  Fedora 14
  kernel: 2.6.35.12-88.fc14.x86_64
  qemu-kvm-0.13.0-1.fc14.x86_64

VMs:
  Fedora 14
  kernel 2.6.35.11-83.fc14.x86_64
  2 vcpus
  1GB RAM

Thanks,

David