Re: Zero copy transmit

[Andi Kleen <ak@suse.de>]

> Don't get me wrong, we would certainly drop any notions of this if we 
> found that it was slower and I will be glad to post any results. The 
> goal is to take advantage of the hardware to make things faster.

You have no hardware to make the remote TLB flushes fast ;)

I'm sure you can show it being an advantage with a single threaded process.
But when you run it on a multithreaded application just with two threads
it may look very different.

> Going back to your example above, don't solaris and hpux also do COW for 
> write and send? (I don't have their sources)  If so, why would they do 
> it if it's slower?

I don't know if they do. The only Unix I'm aware of that has zero copy
sendmsg() is NetBSD and their focus does not seem to be SMP scalability.

I observed the problem recently just with swapping a big (10GB) process
whose working set slightly exceeded the available memory.
kswapd was running on one CPU; the process on another. kswapd 
was aging the pages of the memory hog all the time, which requires an unmapping
and a remote TLB flush in the process' page tables. The result 
was that two CPUs were 100% tied up in the kernel, just spinning on the 
page_table_lock of the mm and processing TLB IPIs (spinlock was ~50%; IPI 
overhead 40% or so). I predict that your proposed TLB flushing write will
cause the same problem with lots of writes. It's more or less the same thing,
except that kswapd has a builtin rate limit and runs only on a single CPU 
and write() has not.

Also last time I checked most Linux ports still used an single global
spinlock for the TLB flush IPI. You would add a nice new hot lock
to the network path.

-Andi