Re: Efficient IPC mechanism on Linux

[Arjan van de Ven <arjanv@redhat.com>]

On Wed, Sep 10, 2003 at 11:40:33AM +0200, Luca Veraldi wrote:
> To set the accessed or dirty bit you use
> 
> 38         __asm__ __volatile__( LOCK_PREFIX
> 39                 "btsl %1,%0"
> 40                 :"=m" (ADDR)
> 41                 :"Ir" (nr));
> 
> which is a ***SINGLE CLOCK CYCLE*** of cpu.
> I don't think really that on any machine Firmware 
> a btsl will require 4000 cycles.
> Neither on Intel x86.

For fun do the measurement on a pIV cpu. You'll be surprised.
The microcode "mark dirty" (which is NOT a btsl, it gets done when you do a write
memory access to the page content) result will be in the 2000 to 4000 range I
predict. There are things like SMP synchronisation to do, but also
if the cpu marks a page dirty in the page table, that means the page table
changes which means the pagetable needs to be marked in the
PMD. Which means the PMD changes, which means the PGD needs the PMD marked
dirty. Etc Etc. It's microcode. It'll take several 1000 cycles.



> > Changing pagetable content is even more because all the
> > tlb's and internal cpu state will need to be flushed... which is also a
> > microcode operation for the cpu. 
> 
> Good. The same overhead you will find accessing a message 
> after a read form a pipe. There will occur many TLB faults.
> And the same apply copying the message to the pipe.
> Many many TLB faults.

A TLB fault in the normal case is about 7 cycles. But that's for a TLB not
being present. For TLB that IS present being written to means going to
microcode.

> 
> > And it's deadly in an SMP environment.
> 
> You say "tlb's and internal cpu state will need to be flushed".
> The other cpus in an SMP environment can continue to work, indipendently.
> TLBs and cpu state registers are ***PER-CPU*** resorces.

if you change a page table, you need to flush the TLB on all other cpus
that have that same page table mapped, like a thread app running
on all cpu's at once with the same pagetables.

> Probably, it is worse the case of copying a memory page,
> because you have to hold some global lock all the time.

why would you need a global lock for copying memory ?