[network performance question] only ~2Gbps throughput between two linux guests which are running on the same host via netperf -t TCP_STREAM -m 1400, but xen can achieve ~7Gbps

Zhang Haoyu
2014-06-06T03:31:11Z

Hi, all

I ran two linux guest on the same kvm host, then start the netserver on one vm, start netperf on the other one, netperf command and test result shown as below,
netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET : nodelay
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 1400 60.01 2355.45

but I ran two linux guest on the same xen hypervisor, ~7Gbps throughput achived, netperf command and test result shown as below,
netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 1400 60.01 2349.82

many times test performed, the result is similar as above.

When I tcpdump backend tap netdevice, found that most packets' size is 1448bytes on kvm, and few packets are ~60000Bytes,
but I tcpdump backend vif netdevice, found that most packets' size is >60000bytes on xen.
Test result of netperf -t TCP_STREAM -m 64 is similar, more larger packets on xen than kvm.

And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm.

Any ideas?

Thanks,
Zhang Haoyu

<environment>
guest: linux-3.10, or centos-6.3
host: linux-3.10(kvm), xenserver-6.2

<test tool>
netperf-2.6

<test result details>
1. KVM
netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 65000 60.00 13743.44

netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 1400 60.01 2349.82

netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 64 -M 64
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 64 60.01 294.70

2. XEN
netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 65000 60.01 6327.46

netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 16384 1400 60.00 7250.63

netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 64 -M 64
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET