UltraSPARC lock performance

UltraSPARC lock performance

[Mike Bennett]

Hi -

I've looked with great interest at the UltraLinux codebase; great
accomplishment!

I've recently been doing a deal of performance work on SPARC/Solaris, and
had a suggestion with regards to SMP lock performance.

Currently, I see (based on the 2.1.102 release snapshot shown by the
LXR folks) that the locking primitive used is the ldstub primitive.
I've been doing performance analysis of wait-free algorithms and have 
found a very striking performance difference between using ldstub 
and the 'cas' (compare-and-swap) uSPARC instruction to implement locking 
primitives.

On a single processor, cas outperforms ldstub by a good margin.  On an
SMP machine, the difference is extreme.  One of the properties of ldstub
on the 2+ cpu uSPARCS is the requirement that ldstub lock the bus; cas
does not have this characteristic.

If you take a simple lock call implemented with 'ldstub' and one with 'cas'
and time it on a 3-way or better system, a graph of execution time shows
a cas primitive having a fairly linear improvement in throughput; however,
ldstub takes a HUGE performance hit on the 3rd cpu and gets progressively
worse as you add cpus.  Moral, is, if you use ldstub don't run more than
2 cpus (2 is slightly better performing than one, but nowhere near the
speed of 'cas').

There are sample 'cas' lock implementations in the SPARC v9 architecture
manual (section J?).  For a kernel implementation, it might be worthwhile
to check it out for yourself.

Great work; hope this helps -
  -mike bennett  (mbennett@netcom.com)

Re: UltraSPARC lock performance

[David S. Miller]

We use CAS where applicable in the Ultra SMP support code.

Also your claims about ldstub are incorrect, they do not lock the bus,
just like CAS, if the L2 cache line being accessed is valid and owned
exclusively, as per the UPA interface documentation in the UltraSparc
programmers manuals.

What you are seeing is the difference between wait free algorithms and
spinlock based resource protection in general, which you failed to
mention at all.

Later,
David S. Miller
davem@dm.cobaltmicro.com