Re: [PATCH] WorkStruct: Implement generic UP cmpxchg() where an arch doesn't support it

[Matthew Wilcox <matthew@wil.cx>]

On Wed, Dec 06, 2006 at 11:34:52AM -0800, Linus Torvalds wrote:
> On Wed, 6 Dec 2006, Matthew Wilcox wrote:
> > Given parisc's paucity of atomic operations (load-and-zero-32bit and
> > load-and-zero-64bit), cmpxchg() is impossible to implement safely.
> > There has to be something we can hook to exclude another processor
> > modifying the variable.  I'm OK with using atomic_cmpxchg(); we have
> > atomic_set locked against it.
> 
> How do you to the atomic bitops?

The same way we do atomic_t.

What I hadn't realised (because I hadn't read dhowell's implementation
... because it hasn't shown up on git2.kernel.org yet) is that he
doesn't actually *use* this unlocked-assignment that would cause the
problem.  He uses bitops which use the same locks.

> Also, I don't see quite why you think "cmpxchg()" and "atomic_cmpxchg()" 
> would be different. ANY cmpxchg() needs to be atomic - if it's not, 
> there's no point to the operation at all, since you'd just write it as
> 
> 	if (*p == x)
> 		*p = y;
> 
> instead, and not use cmpxchg(). 

The difference is that we can (and do) acquire a lock for atomic_set.
We can't acquire one for X = 6.

>  - on SMP, use a spinlock (with interrupts disabled), and share that 
>    spinlock with all the other atomic ops (bitops at a minimum - the 
>    "atomic_t" operations have traditionally been in another "locking 
>    space" because of sparc32 historic braindamage, but I'd suggest 
>    sharing the same spinlock between them all).

Yep, we agree.

> And yeah, it sucks. You _can_ (if you really want to) make the spinlock be 
> hashed based on the address of the atomic data structure. That at least 
> allows you to do _multiple_ spinlocks, but let's face it, your real 
> problem is _likely_ going to be cacheline bouncing, not contention, and 
> then using a hashed lock won't be likely to buy you all that much.

We do hash based on the address -- and we try to arrange things such
that different spinlocks are acquired for different cachelines.  I don't
know if anyone's benchmarked it recently to see how well it performs.
It's a bit of a waltzing bear [1] at times ;-)

[1] The wonder is not how well it waltzes, but that it waltzes at all