From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: performance of virtual functions compared to virtio Date: Mon, 25 Apr 2011 15:18:47 -0600 Message-ID: <1303766327.3431.100.camel@x201> References: <4DAF8EF0.8010203@gmail.com> <1303353349.3110.181.camel@x201> <4DB5B1C4.4000602@gmail.com> <1303755193.3431.50.camel@x201> <4DB5C65C.20306@gmail.com> <1303759773.3431.64.camel@x201> <4DB5D053.1070401@gmail.com> <1303763254.3431.79.camel@x201> <4DB5DC41.7060308@gmail.com> <1303765341.3431.94.camel@x201> <4DB5E434.1040708@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: KVM mailing list To: David Ahern Return-path: Received: from mx1.redhat.com ([209.132.183.28]:18052 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751186Ab1DYVSu (ORCPT ); Mon, 25 Apr 2011 17:18:50 -0400 In-Reply-To: <4DB5E434.1040708@gmail.com> Sender: kvm-owner@vger.kernel.org List-ID: On Mon, 2011-04-25 at 15:14 -0600, David Ahern wrote: > > On 04/25/11 15:02, Alex Williamson wrote: > > On Mon, 2011-04-25 at 14:40 -0600, David Ahern wrote: > >> > >> On 04/25/11 14:27, Alex Williamson wrote: > >>> On Mon, 2011-04-25 at 13:49 -0600, David Ahern wrote: > >>>> > >>>> On 04/25/11 13:29, Alex Williamson wrote: > >>>>> So we're effectively getting host-host latency/throughput for the VF, > >>>>> it's just that in the 82576 implementation of SR-IOV, the VF takes a > >>>>> latency hit that puts it pretty close to virtio. Unfortunate. I think > >>>> > >>>> For host-to-VM using VFs is worse than virtio which is counterintuitive. > >>> > >>> On the same host, just think about the data path of one versus the > >>> other. On the guest side, there's virtio vs a physical NIC. virtio is > >>> designed to be virtualization friendly, so hopefully has less context > >>> switches in setting up and processing transactions. Once the packet > >>> leaves the assigned physical NIC, it has to come back up the entire host > >>> I/O stack, while the virtio device is connected to an internal bridge > >>> and bypasses all but the upper level network routing. > >> > >> I get the virtio path, but you lost me on the physical NIC. I thought > >> the point of VFs is to bypass the host from having to touch the packet, > >> so the processing path with a VM using a VF would be the same as a non-VM. > > > > In the VF case, the host is only involved in processing the packet on > > it's end of the connection, but the packet still has to go all the way > > out to the physical device and all the way back. Handled on one end by > > the VM and the other end by the host. > > > > An analogy might be sending a letter to an office coworker in a > > neighboring cube. You could just pass the letter over the wall (virtio) > > or you could go put it in the mailbox, signal the mail carrier, who > > comes and moves it to your neighbor's mailbox, who then gets signaled > > that they have a letter (device assignment). > > > > Since the networks stacks are completely separate from one another, > > there's very little difference in data path whether you're talking to > > the host, a remote system, or a remote VM, which is reflected in your > > performance data. Hope that helps, > > Got you. I was thinking host-VM as VM on separate host; I didn't make > that clear. Thanks for clarifying - I like the letter example. I should probably also note that being able to "pass a letter over the wall" is possible because of the bridge/tap setup used for that communication path, so it's available to emulated NICs as well. virtio is just a paravirtualization layer that makes it lower overhead than emulation. To get a letter "out of the office" (ie. off host), all paths still eventually need to put the letter in the mailbox. Thanks, Alex