Re: [PATCH] af_packet: Add Doorbell transmit mode to AF_PACKET sockets

[John Fastabend <john.r.fastabend@intel.com>]

On 10/09/2014 06:36 AM, Neil Horman wrote:
> This patch adds a variation to the AF_PACKET memory mapped socket transmit
> mechanism.  Nominally, when using a memory mapped AF_PACKET socket, frames are
> written into the memory mapped buffer, and then the application calls sendmsg
> with a NULL buffer which triggers then cleans the mapped space of all pending
> buffers.
> 
> While this provides clean, synchronous operation, improvements can be made.  To
> this end, I've introduced a doorbell mode of operation to memory mapped packet
> sockets. When a packet socket is placed into doorbell mode, it write protects
> the mappings of any process using the packet socket, so that on the first write
> to it, a kernel trap is generated, which returns the mapping to a read-write
> state, and forks a task to begin cleaning the buffers on the applications
> behalf.  This thread contains some hysterisis to continue running a short while
> after the last buffer has been cleaned, allowing subsquent wrtites to be sent
> without needing to fork another task.  This allows for additional parallelism in
> that an application on an smp system can run in parallel with a cleaning task,
> so that the socket buffer can be filled and emptied in parallel without having
> to incur multiple system call traps.
> 
> I've only done some very rough performance estimates, but early results are
> promising.  Using this code here:
> http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap
> 
> I made some modifications to support using doorbell mode and compared the time
> it took to send 1500 packets (each of size 1492 bytes), in basic mmap and
> doorbell mmaped mode, and used tcpdump to capture the output.  Results:
> 
> trace	packets	start time	end time	delta		p/s	size
> ndb	1500	2.755605	3.000886	0.245281	6115.43	1492b
> db	1500	4.716448	4.846382	0.129934	11544.32 1492b
> 
> Its very rough of course but it would seem I get a 40% increase in throughput
> when using this method.  I'm sure thats an overestimate, and so more testing is
> required, but initial results look good.
> 
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> ---


Thanks Neil, This looks helpful I'll see if I can merge something like this with
my previous patch. Not likely to have anything by next week though ;)

.John