Re: copyless virtio net thoughts?

[Rusty Russell <rusty@rustcorp.com.au>]

On Thursday 05 February 2009 12:37:32 Chris Wright wrote:
> There's been a number of different discussions re: getting copyless virtio
> net (esp. for KVM).  This is just a poke in that general direction to
> stir the discussion.  I'm interested to hear current thoughts?

This thread seems to have died out, time for me to weigh in!

There are four promising areas that I see when looking at virtio_net performance.  I list them all here because they may interact:

1) Async tap access.
2) Direct NIC attachment.
3) Direct interguest networking.
4) Multiqueue virtio_net.

1) Async tap access
Either via aio, or something like the prototype virtio_ring patches I produced last year.  This is potentially copyless networking for xmit (bar header), with one copy on recv.

2) Direct NIC attachment
This is particularly interesting with SR-IOV or other multiqueue nics, but for boutique cases or benchmarks, could be for normal NICs.  So far I have some very sketched-out patches: for the attached nic dev_alloc_skb() gets an skb from the guest (which supplies them via some kind of AIO interface), and a branch in netif_receive_skb() which returned it to the guest.  This bypasses all firewalling in the host though; we're basically having the guest process drive the NIC directly.

3) Direct interguest networking
Anthony has been thinking here: vmsplice has already been mentioned.  The idea of passing directly from one guest to another is an interesting one: using dma engines might be possible too.  Again, host can't firewall this traffic.  Simplest as a dedicated "internal lan" NIC, but we could theoretically do a fast-path for certain MAC addresses on a general guest NIC.

4) Multiple queues
This is Herbert's.  Should be fairly simple to add; it was in the back of my mind when we started.  Not sure whether the queues should be static or dynamic (imagine direct interguest networking, one queue pair for each other guest), and how xmit queues would be selected by the guest (anything anywhere, or dst mac?).

Anyone else want to make comments?

Thanks,
Rusty.