Re: idle issues running sembench on 128 cpus

[Andi Kleen <andi@firstfloor.org>]

On Thu, May 05, 2011 at 01:29:49AM +0200, Thomas Gleixner wrote:
> > > hierarchical about that. It all boils down on _ONE_ single functional
> > 
> > The mask can be used to see if another thread on this core is still
> > running. If yes you don't need that. Right now Linux doesn't 
> > know that, but it could be taught. The only problem is that once
> > the other guy goes idle too their timeouts have to be merged.
> > 
> > This would cut contention in half.
> 
> That makes sense, but merging the timeouts race free will be a real
> PITA.

For this case one could actually use a spinlock between the siblings.
That shouldn't be a problem as long as it's not a global spinlock.

> 
> > Also if it's HPET you could actually use multiple independent HPET channels.
> > I remember us discussing this a long time ago... Not sure if it's worth
> > it, but it may be a small relief.
> 
> Multiple broadcast devices. That sounds still horrible :)


It would cut contention in half or more at least. Not great,
but sometimes you take everything you can get.

> Might be worth a try. Need some sleep to remember why I discarded that
> idea long ago.

Ok.

Here's a new patch without the raw. Boots on my Westmere.

---

From: Andi Kleen <ak@linux.intel.com>
Subject: [PATCH] Move C3 stop test outside lock

Avoid taking locks in the idle path for systems where the timer
doesn't stop in C3.

Signed-off-by: Andi Kleen <ak@linux.intel.com>

diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index da800ff..9cf0415 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -456,23 +456,22 @@ void tick_broadcast_oneshot_control(unsigned long reason)
 	unsigned long flags;
 	int cpu;
 
-	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
-
 	/*
 	 * Periodic mode does not care about the enter/exit of power
 	 * states
 	 */
 	if (tick_broadcast_device.mode == TICKDEV_MODE_PERIODIC)
-		goto out;
+		return;
 
+	cpu = raw_smp_processor_id();
 	bc = tick_broadcast_device.evtdev;
-	cpu = smp_processor_id();
 	td = &per_cpu(tick_cpu_device, cpu);
 	dev = td->evtdev;
 
 	if (!(dev->features & CLOCK_EVT_FEAT_C3STOP))
-		goto out;
+		return;
 
+	raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
 	if (reason == CLOCK_EVT_NOTIFY_BROADCAST_ENTER) {
 		if (!cpumask_test_cpu(cpu, tick_get_broadcast_oneshot_mask())) {
 			cpumask_set_cpu(cpu, tick_get_broadcast_oneshot_mask());