BCM5708 performance issues

BCM5708 performance issues

[Pete Ashdown]

I'm running two separate KVM hosts that are showing the same network
performance issue - KVM-guest is 35% slower than non-KVM (or ESXi for that
matter).  I'm wondering if the "Broadcom NetXtreme II BCM5708" interface is at
fault.  I've tried all sorts of tweaks and configurations and I've only
succeeded in making the network slower.

The first box is running ProxMox 1.6.  The second is Ubuntu 10.04.  The guests
on both boxes are 64-bit Ubuntu 10.04 server installs with virtio.

ProxMox guest:
/usr/bin/kvm -monitor unix:/var/run/qemu-server/104.mon,server,nowait -vnc unix:/var/run/qemu-server/104.vnc,password -pidfile /var/run/qemu-server/104.pid -daemonize -usbdevice tablet -name UbuntuServer -smp sockets=2,cores=2 -nodefaults -boot menu=on -vga cirrus -tdf -k en-us -drive file=/var/lib/vz/images/104/vm-104-disk-2.raw,if=ide,index=3 -drive file=/var/lib/vz/images/104/vm-104-disk-1.raw,if=virtio,index=0,boot=on -m 1024 -net tap,vlan=0,ifname=vmtab104i0,script=/var/lib/qemu-server/bridge-vlan -net nic,vlan=0,model=virtio,macaddr=76:3F:1A:03:6D:6F

Ubuntu guest:
/usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name ubutest -uuid c0537369-fffa-9680-2f29-2e0cc0406561 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/ubutest.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/dev/vg/ubutest,if=virtio,index=0,boot=on -net nic,macaddr=52:54:00:35:11:f1,vlan=0,model=virtio,name=virtio.0 -net tap,fd=51,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 0.0.0.0

Netperf looks like this on the Ubuntu host:
 Recv   Send    Send                          
 Socket Socket  Message  Elapsed              
 Size   Size    Size     Time     Throughput  
 bytes  bytes   bytes    secs.    10^6bits/sec  

 1048576  16384  16384    10.01     941.62   

This is what I get on the Ubuntu guest for either setup:
 Recv   Send    Send                          
 Socket Socket  Message  Elapsed              
 Size   Size    Size     Time     Throughput  
 bytes  bytes   bytes    secs.    10^6bits/sec  

 1048576  16384  16384    10.00     615.65

Thank you for any guidance you can lend.

Re: BCM5708 performance issues

[Chris Wright]

* Pete Ashdown (pashdown@xmission.com) wrote:
> ProxMox guest:
> /usr/bin/kvm -monitor unix:/var/run/qemu-server/104.mon,server,nowait -vnc unix:/var/run/qemu-server/104.vnc,password -pidfile /var/run/qemu-server/104.pid -daemonize -usbdevice tablet -name UbuntuServer -smp sockets=2,cores=2 -nodefaults -boot menu=on -vga cirrus -tdf -k en-us -drive file=/var/lib/vz/images/104/vm-104-disk-2.raw,if=ide,index=3 -drive file=/var/lib/vz/images/104/vm-104-disk-1.raw,if=virtio,index=0,boot=on -m 1024 -net tap,vlan=0,ifname=vmtab104i0,script=/var/lib/qemu-server/bridge-vlan -net nic,vlan=0,model=virtio,macaddr=76:3F:1A:03:6D:6F
> 
> Ubuntu guest:
> /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name ubutest -uuid c0537369-fffa-9680-2f29-2e0cc0406561 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/ubutest.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/dev/vg/ubutest,if=virtio,index=0,boot=on -net nic,macaddr=52:54:00:35:11:f1,vlan=0,model=virtio,name=virtio.0 -net tap,fd=51,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 0.0.0.0

Not sure what userspace you are using, but you are probably not getting
any of the useful offload features set.  Checking "ethtool -k $ETH"
in the guest will verify that.

Try changing this:

-net nic,macaddr=52:54:00:35:11:f1,vlan=0,model=virtio,name=virtio.0 \
-net tap,fd=51,vlan=0,name=tap.0

to use newer syntax:

-netdev type=tap,id=netdev0
-device virtio-net-pci,mac=52:54:00:35:11:f1,netdev=netdev0

With just a 1Gb link, you should see line rate from guest via virtio.

thanks,
-chris

virtio network performance [was: Re: BCM5708 performance issues]

[Chris Wright]

* Chris Wright (chrisw@sous-sol.org) wrote:
> * Pete Ashdown (pashdown@xmission.com) wrote:
> > ProxMox guest:
> > /usr/bin/kvm -monitor unix:/var/run/qemu-server/104.mon,server,nowait -vnc unix:/var/run/qemu-server/104.vnc,password -pidfile /var/run/qemu-server/104.pid -daemonize -usbdevice tablet -name UbuntuServer -smp sockets=2,cores=2 -nodefaults -boot menu=on -vga cirrus -tdf -k en-us -drive file=/var/lib/vz/images/104/vm-104-disk-2.raw,if=ide,index=3 -drive file=/var/lib/vz/images/104/vm-104-disk-1.raw,if=virtio,index=0,boot=on -m 1024 -net tap,vlan=0,ifname=vmtab104i0,script=/var/lib/qemu-server/bridge-vlan -net nic,vlan=0,model=virtio,macaddr=76:3F:1A:03:6D:6F
> > 
> > Ubuntu guest:
> > /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name ubutest -uuid c0537369-fffa-9680-2f29-2e0cc0406561 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/ubutest.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/dev/vg/ubutest,if=virtio,index=0,boot=on -net nic,macaddr=52:54:00:35:11:f1,vlan=0,model=virtio,name=virtio.0 -net tap,fd=51,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 0.0.0.0
> 
> Not sure what userspace you are using, but you are probably not getting
> any of the useful offload features set.  Checking "ethtool -k $ETH"
> in the guest will verify that.
> 
> Try changing this:
> 
> -net nic,macaddr=52:54:00:35:11:f1,vlan=0,model=virtio,name=virtio.0 \
> -net tap,fd=51,vlan=0,name=tap.0
> 
> to use newer syntax:
> 
> -netdev type=tap,id=netdev0
> -device virtio-net-pci,mac=52:54:00:35:11:f1,netdev=netdev0
> 
> With just a 1Gb link, you should see line rate from guest via virtio.

Just to follow-up for the archives.  Pete replied offlist that using
the above cmdline eliminates the performance issue.

thanks,
-chris