From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Ahern Subject: Re: performance of virtual functions compared to virtio Date: Mon, 25 Apr 2011 13:07:08 -0600 Message-ID: <4DB5C65C.20306@gmail.com> References: <4DAF8EF0.8010203@gmail.com> <1303353349.3110.181.camel@x201> <4DB5B1C4.4000602@gmail.com> <1303755193.3431.50.camel@x201> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: KVM mailing list To: Alex Williamson Return-path: Received: from mail-pv0-f174.google.com ([74.125.83.174]:35268 "EHLO mail-pv0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758203Ab1DYTHL (ORCPT ); Mon, 25 Apr 2011 15:07:11 -0400 Received: by pvg12 with SMTP id 12so1344781pvg.19 for ; Mon, 25 Apr 2011 12:07:11 -0700 (PDT) In-Reply-To: <1303755193.3431.50.camel@x201> Sender: kvm-owner@vger.kernel.org List-ID: On 04/25/11 12:13, Alex Williamson wrote: >> So, basically, 192.168.102 is the network where the VMs have a VF, and >> 192.168.103 is the network where the VMs use virtio for networking. >> >> The netperf commands are all run on either Host-A or VM-C: >> >> netperf -H $ip -jcC -v 2 -t TCP_RR -- -r 1024 -D L,R >> netperf -H $ip -jcC -v 2 -t TCP_STREAM -- -m 1024 -D L,R >> >> >> latency throughput >> (usec) Mbps >> cross-host: >> A-B, eth2 185 932 >> A-B, eth3 185 935 > > This is actually PF-PF, right? It would be interesting to load igbvf on > the hosts and determine VF-VF latency as well. yes, PF-PF. eth3 has the added bridge layer, but from what I can see the overhead is noise. I added host-to-host to put the host-to-VM numbers in perspective. > >> same host, host-VM: >> A-C, using VF 488 1085 (seen as high as 1280's) >> A-C, virtio 150 4282 > > We know virtio has a shorter path for this test. No complaints about the throughput numbers; the latency is the problem. > >> cross-host, host-VM: >> A-D, VF 489 938 >> A-D, virtio 288 889 >> >> cross-host, VM-VM: >> C-D, VF 488 934 >> C-D, virtio 490 933 >> >> >> While throughput for VFs is fine (near line-rate when crossing hosts), > > FWIW, it's not too difficult to get line rate on a 1Gbps network, even > some of the emulated NICs can do it. There will be a difference in host > CPU power to get it though, where it should theoretically be emulated > > virtio > pci-assign. 10GB is the goal; 1GB offers a cheaper learning environment. ;-) > >> the latency is horrible. Any options to improve that? > > If you don't mind testing, I'd like to see VF-VF between the hosts (to > do this, don't assign eth2 an IP, just make sure it's up, then load the > igbvf driver on the host and assign an IP to one of the VFs associated > with the eth2 PF), and cross host testing using the PF for the guest > instead of the VF. This should help narrow down how much of the latency > is due to using the VF vs the PF, since all of the virtio tests are > using the PF. I've been suspicious that the VF adds some latency, but > haven't had a good test setup (or time) to dig very deep into it. It's a quad nic, so I left eth2 and eth3 alone and added the VF-VF test using VFs on eth4. Indeed latency is 488 usec and throughput is 925 Mbps. This is host-to-host using VFs. David > Thanks, > > Alex >