splice() based interguest networking

Anthony Liguori
2008-12-01T19:33:15Z

Here's a random thought I had after seeing the new Xen netchannel2 tree
had fast-path support for guest<=>guest communication.

With virtio, we could do really fast interguest networking in
userspace. We have a few requirements though:

1) There should be a minimal number of copies, just one in almost all cases.
2) The copy should occur on the receiving end since the receiver is most
likely going to be accessing the data in the future
3) The copy should be done in the kernel so that in the future it could
be accelerated with a generic DMA engine.

So far, all the approaches required mmap()'ing the guest memory in both
QEMU instances which makes it much less useful. I think splice solves
this problem though and gets us most of the above for free.

If we have two shared pipes() between the two QEMU processes, then:

1) On TX, we vmsplice() from the sg buffer to one pipe. This will end
up being vmsplice_to_pipe() in the kernel which is zero-copy.

2) The pipe becomes readable which will result in an RX notification in
the other process, we see if we have any buffers available in the
receive queue. If so, we vmsplice() from the pipe to the sg buffer.
This will result in a copy via vmsplice_to_user(). In the future,
vmsplice_to_user() would be an obvious candidate for IO-AT acceleration.

Since the copy is happening in the kernel, assuming you're not in a
highmem situation, no page table manipulation is required.

We still have to address feature negotation and such.

Regards,

Anthony Liguori