Re: async network I/O, event channels, etc

[David Miller <davem@davemloft.net>]

From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
Date: Thu, 27 Jul 2006 11:49:02 +0400

> I.e. map skb's data to userspace? Not a good idea especially with it's
> tricky lifetime and unability for userspace to inform kernel when it
> finished and skb can be freed (without additional syscall).

Hmmm...

If it is paged based, I do not see the problem.  Events and calls to
AIO I/O routines make transfer of buffer ownership.  The fact that
while kernel (and thus networking stack) "owns" the buffer for an AIO
call, the user can have a valid mapping to it is a unimportant detail.

If the user will scramble a piece of data that is in flight to or from
the network card, it is his problem.

If we are using a primitive network card that does not support
scatter-gather I/O and thus not page based SKBs, we will make
copies.  But this is transparent to the user.

The idea is that DMA mappings have page granularity.

At least on transmit it should work well.  Receive side is more
difficult and initial implementation will need to copy.

> I did it with af_tlb zero-copy sniffer (but I substitute mapped pages
> with physical skb->data pages), and it was not very good.

Trying to be too clever with skb->data has always been catastrophic. :)

> Well, I think preallocate some buffers and use that in AIO setup is a
> plus, since in that case user does not care about when it is possible to
> reuse the same buffer - when appropriate kevent is completed, that means
> that provided buffer is no longer in use by kernel, and user can reuse
> it.

We now enter the most interesting topic of AIO buffer pool management
and where it belongs. :-)  We are assuming up to this point that the
user manages this stuff with explicit DMA calls for allocation, then
passes the key based references to those buffers as arguments to the
AIO I/O calls.

But I want to suggest another possibility.  What if the kernel managed
the AIO buffer pool for a task?  It could grow this dynamically based
upon need.  The only implementation road block is how large to we
allow this to grow, but I think normal VM mechanisms can take care
of it.

On transmit this is not straightforward, but for receive it has really
nice possibilities. :)