splice() based interguest networking

splice() based interguest networking

[Anthony Liguori]

Here's a random thought I had after seeing the new Xen netchannel2 tree 
had fast-path support for guest<=>guest communication.

With virtio, we could do really fast interguest networking in 
userspace.  We have a few requirements though:

1) There should be a minimal number of copies, just one in almost all cases.
2) The copy should occur on the receiving end since the receiver is most 
likely going to be accessing the data in the future
3) The copy should be done in the kernel so that in the future it could 
be accelerated with a generic DMA engine.

So far, all the approaches required mmap()'ing the guest memory in both 
QEMU instances which makes it much less useful.  I think splice solves 
this problem though and gets us most of the above for free.

If we have two shared pipes() between the two QEMU processes, then:

1) On TX, we vmsplice() from the sg buffer to one pipe.  This will end 
up being vmsplice_to_pipe() in the kernel which is zero-copy.

2) The pipe becomes readable which will result in an RX notification in 
the other process, we see if we have any buffers available in the 
receive queue.  If so, we vmsplice() from the pipe to the sg buffer.  
This will result in a copy via vmsplice_to_user().  In the future, 
vmsplice_to_user() would be an obvious candidate for IO-AT acceleration.

Since the copy is happening in the kernel, assuming you're not in a 
highmem situation, no page table manipulation is required.

We still have to address feature negotation and such.

Regards,

Anthony Liguori

Re: splice() based interguest networking

[Avi Kivity]

Anthony Liguori wrote:
>
> 1) On TX, we vmsplice() from the sg buffer to one pipe.  This will end 
> up being vmsplice_to_pipe() in the kernel which is zero-copy.
>

That implies we do the MAC address switching in userspace (or that this 
is a point-to-point protocol, which severely limits its usefulness).

I think we can still have one-copy interguest networking if we have 
proper skb destructors; and since we need that for the more important 
guest->external copyless support, we basically get it for free (if delayed).

-- 
error compiling committee.c: too many arguments to function