Re: Efficient IPC mechanism on Linux

[Manfred Spraul <manfred@colorfullife.com>]

>
>
>Hi Luca,
>There was a zero-copy pipe implementation floating around a while ago
>I think. Did you have a look at that? IIRC it had advantages and
>disadvantages over regular pipes in performance.
>
It has doesn't have any performance disadvantages over regular pipes.
The problems is that it only helps for lmbench. Real world apps use 
buffered io, which uses PAGE_SIZEd buffers, which mandate certain 
atomicity requirements, which limits the gains that can be achieved.
If someone wants to try it again I can search for my patches.
Actually there are two independant improvements that are possible for 
the pipe code:
- zero-copy. The name is misleading. In reality this does a single copy: 
the sender creates a list of the pages with the data instead of copying 
the data to kernel buffers. The receiver copies from the pages directly 
to user space. The main advantage is that this implementation avoids one 
copy without requiring any tlb flushes. It works with arbitrary aligned 
user space buffers.
- use larger kernel buffers. Linux uses a 4 kB internal buffer. The 
result is a very simple implementation that causes an incredible amount 
of context switches. Larger buffers reduce that. The problem is 
accounting, and avoiding to use too much memory for the pipe buffers. A 
patch without accounting should be somewhere. The BSD unices have larger 
pipe buffers, and a few pipes with huge buffers [perfect for lmbench 
bragging].

Luca: An algorithm that uses page flipping, or requires tlb flushes, 
probably performs bad in real-world scenarios. First you have the cost 
of the tlb flush, then the inter-processor-interrupt to flush all cpus 
on SMP. And if the target app doesn't expect that you use page flipping, 
then you get a page fault [with another tlb flush], to break the COW 
share. OTHO if you redesign the app for page flipping, then it could 
also use sysv shm, and send a short message with a pointer to the shm 
segment.

--
    Manfred