Re: performance of virtual functions compared to virtio

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 2011-04-25 at 15:14 -0600, David Ahern wrote:
> 
> On 04/25/11 15:02, Alex Williamson wrote:
> > On Mon, 2011-04-25 at 14:40 -0600, David Ahern wrote:
> >>
> >> On 04/25/11 14:27, Alex Williamson wrote:
> >>> On Mon, 2011-04-25 at 13:49 -0600, David Ahern wrote:
> >>>>
> >>>> On 04/25/11 13:29, Alex Williamson wrote:
> >>>>> So we're effectively getting host-host latency/throughput for the VF,
> >>>>> it's just that in the 82576 implementation of SR-IOV, the VF takes a
> >>>>> latency hit that puts it pretty close to virtio.  Unfortunate.  I think
> >>>>
> >>>> For host-to-VM using VFs is worse than virtio which is counterintuitive.
> >>>
> >>> On the same host, just think about the data path of one versus the
> >>> other.  On the guest side, there's virtio vs a physical NIC.  virtio is
> >>> designed to be virtualization friendly, so hopefully has less context
> >>> switches in setting up and processing transactions.  Once the packet
> >>> leaves the assigned physical NIC, it has to come back up the entire host
> >>> I/O stack, while the virtio device is connected to an internal bridge
> >>> and bypasses all but the upper level network routing.
> >>
> >> I get the virtio path, but you lost me on the physical NIC. I thought
> >> the point of VFs is to bypass the host from having to touch the packet,
> >> so the processing path with a VM using a VF would be the same as a non-VM.
> > 
> > In the VF case, the host is only involved in processing the packet on
> > it's end of the connection, but the packet still has to go all the way
> > out to the physical device and all the way back.  Handled on one end by
> > the VM and the other end by the host.
> > 
> > An analogy might be sending a letter to an office coworker in a
> > neighboring cube.  You could just pass the letter over the wall (virtio)
> > or you could go put it in the mailbox, signal the mail carrier, who
> > comes and moves it to your neighbor's mailbox, who then gets signaled
> > that they have a letter (device assignment).
> > 
> > Since the networks stacks are completely separate from one another,
> > there's very little difference in data path whether you're talking to
> > the host, a remote system, or a remote VM, which is reflected in your
> > performance data.  Hope that helps,
> 
> Got you. I was thinking host-VM as VM on separate host; I didn't make
> that clear. Thanks for clarifying - I like the letter example.

I should probably also note that being able to "pass a letter over the
wall" is possible because of the bridge/tap setup used for that
communication path, so it's available to emulated NICs as well.  virtio
is just a paravirtualization layer that makes it lower overhead than
emulation.  To get a letter "out of the office" (ie. off host), all
paths still eventually need to put the letter in the mailbox.  Thanks,

Alex