[parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Ryan Bradetich]

Hello parisc-linux hackers,


I have heard of SMP hangs on the A500 from the ESIEE and Grant's system,
so I installed the 2.4.18-pa35 on a dual processor J200 and tried to
duplicate the hangs and hopefully provide a different perspective on the
hang.

The good and bad news is that I can duplicate the process hangs on the 
J200 by simply running two instances of the setiathome program.

Digging into the system a bit, here is what I found:

	* The setiathome process that hung (PID 326) will hang any other
	  process that tries to access /proc/326/*.  (This is why top,
	  ps, etc all hang after the process gets stuck).

	* None of the other processes appear to be stuck. (ie. I can 
	  access the /proc/PID/* information and the command will 	  return).

	* The processes that hang while trying to read from the stuck 
	  process go into a disk sleep and never return.

		This is the a "hung" process that access the stuck 		process.

		# cat status 
		Name:	ps
		State:	D (disk sleep)
		Tgid:	1362
		Pid:	1362
		PPid:	1359
		TracerPid:	0
		Uid:	0	0	0	0
		Gid:	0	0	0	0
		FDSize:	256
		Groups:	0 
		VmSize:	    3032 kB
		VmLck:	       0 kB
		VmRSS:	     880 kB
		VmData:	    1136 kB
		VmStk:	       0 kB
		VmExe:	      68 kB
		VmLib:	    1420 kB
		SigPnd:	0000000000000000
		SigBlk:	0000000000000000
		SigIgn:	8000000000000000
		SigCgt:	000000007f2ffef9
		CapInh:	0000000000000000
		CapPrm:	00000000fffffeff
		CapEff:	00000000fffffeff


It appears that the stuck process also affected my serial console login,
so I am not able to gather more information using the magic-sysrq
commands.  I will try to reboot the system, and see if I can keep a
console up and use the magic-sysrq commands next time the process gets
stuck.


Is this the same behavior people are seeing on the SMP A500's?  Any
ideas on where to continue debugging this (possible deadlock problem?
Can I see what locks are being held by the stuck process? etc..)  I will
continue to poke around and see what I can find also.

Thanks,

- Ryan

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Randolph Chung]

> Is this the same behavior people are seeing on the SMP A500's?  

yes, this is what i have seen too.

> Can I see what locks are being held by the stuck process? etc..)  I will
> continue to poke around and see what I can find also.

Paul Bame has done some work on showing wait channels that might be
useful.

randolph
-- 
   @..@                                         http://www.TauSq.org/
  (----)
 ( >__< )
 ^^ ~~ ^^

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Thibaut VARENE]

Le mardi 11 juin 2002, =E0 07:36 , Ryan Bradetich a =E9crit :

>
> Digging into the system a bit, here is what I found:
>
> 	* The setiathome process that hung (PID 326) will hang any other
> 	  process that tries to access /proc/326/*.  (This is why top,
> 	  ps, etc all hang after the process gets stuck).
Here we run a setiathome client, a distributed.net client, and from=20
times to times we build isos.

This is consistent with what we experienced, but the first time we=20
remarked a hung (after 44 days of uptime), all proccesses that try to=20
get info about running processes, such as 'top', 'ps' or even a 'ls=20
/proc/PID/' would hang. 'w' would also hang, but not 'uptime'. It was=20
with -pa16.

The second time we got a hang, with -pa33, 'ps -ef' hung just before=20
displaying info about the setiathome client.

Another interesting thing is that after issuing any of these commands,=20=

the loadavg increased by 1, ie if we got a loadavg of 2.00 before trying=20=

a ps, a w and a ls /proc/PID, we would get a loadavg of 5.00 after these=20=

3 commands.
>
> 	* None of the other processes appear to be stuck. (ie. I can
> 	  access the /proc/PID/* information and the command will 	 =
 return).
We cannot access such info, alas.
>
>
Anyway, as Randolph Chung said, Paul Bame's work might be interesting...



Thibaut VARENE
PA/Linux ESIEE Team
http://pateam.esiee.fr/

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Michael S.Zick]

On Tuesday 11 June 2002 03:08 am, Thibaut VARENE wrote:
> Le mardi 11 juin 2002, à 07:36 , Ryan Bradetich a écrit :
> > Digging into the system a bit, here is what I found:
> >
> > 	* The setiathome process that hung (PID 326) will hang any other
> > 	  process that tries to access /proc/326/*.  (This is why top,
> > 	  ps, etc all hang after the process gets stuck).
Sirs...

I SEEM to experience a similar situation on 2.4.18-linux-x86-SMP using the 
P111-S processors (with 512K L2 cache).
Quite rare; evidently some very small window for failure.
I don't have any solid leads yet (although I have theories).

Just a note that it might not be HPPA specific.

Mike

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Randolph Chung]

> I SEEM to experience a similar situation on 2.4.18-linux-x86-SMP using the 
> P111-S processors (with 512K L2 cache).
> Quite rare; evidently some very small window for failure.
> I don't have any solid leads yet (although I have theories).

On hppa this is quite easy to reproduce. When Dave is compiling gcc we
can crash the box every hour or so.... :-)

It's interesting to note that even with a UP kernel on the rp2470, the
kernel doesn't seem to be very stable (though better than SMP).... I
think we are looking at multiple bugs here.....

randolph
-- 
   @..@                                         http://www.TauSq.org/
  (----)
 ( >__< )
 ^^ ~~ ^^

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[John David Anglin]

> It's interesting to note that even with a UP kernel on the rp2470, the
> kernel doesn't seem to be very stable (though better than SMP).... I
> think we are looking at multiple bugs here.....

Quite likely.  The A500 seemed more stable.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6605)

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Michael S.Zick]

On Tuesday 11 June 2002 09:58 am, Randolph Chung wrote:
>
> think we are looking at multiple bugs here.....
>
Very likely...
You might check the HPPA version of:

arch/i386/kernel/irq.c::__global_save_flags.

using objdump -d on the kernel.o produced by my compiler
(a different version of GCC than used for pa-risc) I find that
for the "c" code sequence:
....
int cpu = smp_processor_id() ;

__save_flags(flags) ;
....
generates assembly code which modifies the flags during the
initialization of "cpu" before the flags are actually saved.
....
in the pa-risc branch, try the equivalent of:
....
int cpu ;
....
__save_flags(flags) ;
....
cpu = smp_processor_id() ;
....

Mike

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Michael S.Zick]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

On Tuesday 11 June 2002 01:39 pm, Michael S. Zick wrote:
> On Tuesday 11 June 2002 09:58 am, Randolph Chung wrote:
> > think we are looking at multiple bugs here.....
>
> Very likely...
Sirs...
I now have a definate lead for the x86-SMP case, which might 
aid the troubleshooters in the pa-risc branch (diff's attached).

Evidently, some code path is causing these portions of IRQ 
service to be entered with the IRQ state not as designed.

These patches preserve callers interrupt state, eliminating
the erratic behavior, which is only an artifact of the true cause.

True cause still unknown.

Additional note: On the HP models with the worst of the problems;
is the physical_cpu_id and the logical_cpu_id a 1:1 mapping as
it is on x86 systems?

Mike

[-- Attachment #2: x86 branch, don't presume IRQ state --]
[-- Type: text/x-c, Size: 3677 bytes --]

--- linux-2.4.18/arch/i386/kernel/irq.c.org	Thu Oct 25 15:53:46 2001
+++ linux-2.4.18/arch/i386/kernel/irq.c	Tue Jun 11 19:09:14 2002
@@ -387,7 +387,7 @@
 	int retval;
 	int local_enabled;
 	unsigned long flags;
-	int cpu = smp_processor_id();
+	int cpu ;  /* don't initialize cpu here, it changes the flags - MSZ */
 
 	__save_flags(flags);
 	local_enabled = (flags >> EFLAGS_IF_SHIFT) & 1;
@@ -395,6 +395,7 @@
 	retval = 2 + local_enabled;
 
 	/* check for global flags if we're not in an interrupt */
+	cpu = smp_processor_id();
 	if (!local_irq_count(cpu)) {
 		if (local_enabled)
 			retval = 1;
@@ -507,9 +508,12 @@
  
 void disable_irq(unsigned int irq)
 {
+	int cpu_id ;
+
 	disable_irq_nosync(irq);
+	cpu_id = smp_processor_id() ;
 
-	if (!local_irq_count(smp_processor_id())) {
+	if (!local_irq_count(cpu_id)) {
 		do {
 			barrier();
 			cpu_relax();
@@ -806,6 +810,7 @@
 	irq_desc_t *desc;
 	unsigned long val;
 	unsigned long delay;
+	unsigned long flags;  /* Don't presume interrupt state - MSZ */
 
 	down(&probe_sem);
 	/* 
@@ -815,10 +820,12 @@
 	for (i = NR_IRQS-1; i > 0; i--)  {
 		desc = irq_desc + i;
 
-		spin_lock_irq(&desc->lock);
+		spin_lock_irqsave(&desc->lock,flags);
+/* MSZ		spin_lock_irq(&desc->lock); */
 		if (!irq_desc[i].action) 
 			irq_desc[i].handler->startup(i);
-		spin_unlock_irq(&desc->lock);
+		spin_unlock_irqrestore(&desc->lock,flags);
+/* MSZ		spin_unlock_irq(&desc->lock); */
 	}
 
 	/* Wait for longstanding interrupts to trigger. */
@@ -833,13 +840,15 @@
 	for (i = NR_IRQS-1; i > 0; i--) {
 		desc = irq_desc + i;
 
-		spin_lock_irq(&desc->lock);
+		spin_lock_irqsave(&desc->lock,flags);
+/* MSZ		spin_lock_irq(&desc->lock); */
 		if (!desc->action) {
 			desc->status |= IRQ_AUTODETECT | IRQ_WAITING;
 			if (desc->handler->startup(i))
 				desc->status |= IRQ_PENDING;
 		}
-		spin_unlock_irq(&desc->lock);
+		spin_unlock_irqrestore(&desc->lock,flags);
+/* MSZ		spin_unlock_irq(&desc->lock); */
 	}
 
 	/*
@@ -856,7 +865,8 @@
 		irq_desc_t *desc = irq_desc + i;
 		unsigned int status;
 
-		spin_lock_irq(&desc->lock);
+		spin_lock_irqsave(&desc->lock,flags);
+/* MSZ		spin_lock_irq(&desc->lock); */
 		status = desc->status;
 
 		if (status & IRQ_AUTODETECT) {
@@ -868,7 +878,8 @@
 				if (i < 32)
 					val |= 1 << i;
 		}
-		spin_unlock_irq(&desc->lock);
+		spin_unlock_irqrestore(&desc->lock,flags);
+/* MSZ		spin_unlock_irq(&desc->lock); */
 	}
 
 	return val;
@@ -895,13 +906,15 @@
 {
 	int i;
 	unsigned int mask;
+	unsigned int flags;  /* don't presume interrupt state - MSZ */
 
 	mask = 0;
 	for (i = 0; i < NR_IRQS; i++) {
 		irq_desc_t *desc = irq_desc + i;
 		unsigned int status;
 
-		spin_lock_irq(&desc->lock);
+		spin_lock_irqsave(&desc->lock,flags);
+/* MSZ		spin_lock_irq(&desc->lock); */
 		status = desc->status;
 
 		if (status & IRQ_AUTODETECT) {
@@ -911,7 +924,8 @@
 			desc->status = status & ~IRQ_AUTODETECT;
 			desc->handler->shutdown(i);
 		}
-		spin_unlock_irq(&desc->lock);
+		spin_unlock_irqrestore(&desc->lock,flags);
+/* MSZ		spin_unlock_irq(&desc->lock); */
 	}
 	up(&probe_sem);
 
@@ -950,8 +964,10 @@
 	for (i = 0; i < NR_IRQS; i++) {
 		irq_desc_t *desc = irq_desc + i;
 		unsigned int status;
+		unsigned long flags;  /* Don't presume interrupt state -  MSZ */
 
-		spin_lock_irq(&desc->lock);
+		spin_lock_irqsave(&desc->lock,flags);
+/* MSZ		spin_lock_irq(&desc->lock); */
 		status = desc->status;
 
 		if (status & IRQ_AUTODETECT) {
@@ -963,7 +979,8 @@
 			desc->status = status & ~IRQ_AUTODETECT;
 			desc->handler->shutdown(i);
 		}
-		spin_unlock_irq(&desc->lock);
+		spin_unlock_irqrestore(&desc->lock,flags);
+/* MSZ		spin_unlock_irq(&desc->lock); */
 	}
 	up(&probe_sem);
 


[-- Attachment #3: generic, don't presume IRQ state --]
[-- Type: text/x-c, Size: 1756 bytes --]

--- linux-2.4.18/kernel/softirq.c.rml	Tue Jun 11 20:24:24 2002
+++ linux-2.4.18/kernel/softirq.c	Tue Jun 11 20:50:38 2002
@@ -177,11 +177,14 @@
 {
 	int cpu = smp_processor_id();
 	struct tasklet_struct *list;
+	unsigned long flags;  /* Don't presume interrupt state */
 
-	local_irq_disable();
+	local_irq_save(flags);  /* MSZ */
+/* MSZ	local_irq_disable(); */
 	list = tasklet_vec[cpu].list;
 	tasklet_vec[cpu].list = NULL;
-	local_irq_enable();
+	local_irq_restore(flags); /* MSZ */
+/* MSZ	local_irq_enable();  */
 
 	while (list) {
 		struct tasklet_struct *t = list;
@@ -199,11 +202,13 @@
 			tasklet_unlock(t);
 		}
 
-		local_irq_disable();
+		local_irq_save(flags);  /* MSZ */
+/* MSZ		local_irq_disable(); */
 		t->next = tasklet_vec[cpu].list;
 		tasklet_vec[cpu].list = t;
 		__cpu_raise_softirq(cpu, TASKLET_SOFTIRQ);
-		local_irq_enable();
+		local_irq_restore(flags); /* MSZ */
+/* MSZ		local_irq_enable();  */
 	}
 }
 
@@ -211,11 +216,14 @@
 {
 	int cpu = smp_processor_id();
 	struct tasklet_struct *list;
+	unsigned long flags;  /* Don't presume interrupt state */
 
-	local_irq_disable();
+	local_irq_save(flags);  /* MSZ */
+/* MSZ	local_irq_disable(); */
 	list = tasklet_hi_vec[cpu].list;
 	tasklet_hi_vec[cpu].list = NULL;
-	local_irq_enable();
+	local_irq_restore(flags); /* MSZ */
+/* MSZ	local_irq_enable(); */
 
 	while (list) {
 		struct tasklet_struct *t = list;
@@ -233,11 +241,13 @@
 			tasklet_unlock(t);
 		}
 
-		local_irq_disable();
+		local_irq_save(flags);  /* MSZ */
+/* MSZ		local_irq_disable(); */
 		t->next = tasklet_hi_vec[cpu].list;
 		tasklet_hi_vec[cpu].list = t;
 		__cpu_raise_softirq(cpu, HI_SOFTIRQ);
-		local_irq_enable();
+		local_irq_restore(flags); /* MSZ */
+/* MSZ		local_irq_enable(); */
 	}
 }
 

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Grant Grundler]

Michael S. Zick wrote:
> Sirs...
> I now have a definate lead for the x86-SMP case, which might 
> aid the troubleshooters in the pa-risc branch (diff's attached).

ok - thanks. I'll take a look.

But I can tell you now that very little of the code between
x86 and parisc is shared in this area. I doubt it's as helpful
as you might think.

> Additional note: On the HP models with the worst of the problems;
> is the physical_cpu_id and the logical_cpu_id a 1:1 mapping as
> it is on x86 systems?

Yes. parisc didn't have a concept of physical CPU ID until
PAT PDC (eg A500, N-class, etc). When we want to support
CPU addition and removal, we can add physical/logical
mappings and it will no longer be 1:1.

grant

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Matthew Wilcox]

On Thu, Jun 13, 2002 at 03:52:46PM -0600, Grant Grundler wrote:
> > Additional note: On the HP models with the worst of the problems;
> > is the physical_cpu_id and the logical_cpu_id a 1:1 mapping as
> > it is on x86 systems?
> 
> Yes. parisc didn't have a concept of physical CPU ID until
> PAT PDC (eg A500, N-class, etc). When we want to support
> CPU addition and removal, we can add physical/logical
> mappings and it will no longer be 1:1.

Actually this came up in an unrelated question today.  I found that
where other architectures use a physical ID, we use an address.  So we
really have no use for logical/physical mappings.  When the CPU hotplug
code is introduced, it will actually remove the logical numbering and
loops over all cpus will be done with loops over all physical cpus and
a check to see whether that cpu is online.

-- 
Revolutions do not require corporate support.

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Matthew Wilcox]

On Tue, Jun 11, 2002 at 01:39:38PM -0500, Michael S. Zick wrote:
> using objdump -d on the kernel.o produced by my compiler
> (a different version of GCC than used for pa-risc) I find that
> for the "c" code sequence:
> ....
> int cpu = smp_processor_id() ;
> 
> __save_flags(flags) ;
> ....
> generates assembly code which modifies the flags during the
> initialization of "cpu" before the flags are actually saved.

are you sure?  every architecture (apart from sparc32) uses:

#define smp_processor_id() (current->processor)

admittedly on i386 (which is where you seem to be investigating), this
expands to:

get_current()->processor

which expands to

andl %%esp,%0

where 8191UL has previously been loaded into register %0.  but i don't
see how this can affect the interrupt flags.  admittedly, i know very
little about x86, but there are many other instances of code like this
in the kernel.

-- 
Revolutions do not require corporate support.

Re: [parisc-linux] 2.4.18-pa35 SMP process hangs on a J200

[Thibaut VARENE]

Hi all,

We hung again our A500  :/

Still the same symptoms.

It seems that it is the seti process that hung it:
We can access info in /proc/1/, but nothing about the seti process:
see the following:
[varenet@mkhppa3 ~]$ cat seti1/pid.sah
5972
[varenet@mkhppa3 ~]$ ls /proc/5972/

And the command would not return.

We issued a 'w' that got hung too, but we can still access its proc info:
[varenet@mkhppa3 /proc]$ cat 6576/cmdline
w
[varenet@mkhppa3 /proc]$ cat 6576/mem
cat: 6576/mem: No such process
[varenet@mkhppa3 /proc]$ cat 6576/status
Name:   w
State:  D (disk sleep)
Tgid:   6576
Pid:    6576
PPid:   1
TracerPid:      0
Uid:    1001    1001    1001    1001
Gid:    100     100     100     100
FDSize: 256
Groups: 100
VmSize:     1912 kB
VmLck:         0 kB
VmRSS:       812 kB
VmData:       76 kB
VmStk:         0 kB
VmExe:        12 kB
VmLib:      1420 kB
SigPnd: 0000000000000001
SigBlk: 0000000000000000
SigIgn: 8000000000000000
SigCgt: 0000000000000000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Of course, a 'kill -9 6576' would do nothing.

We can also access the info about the distributed.net client, which 
seems to be hang too:
[varenet@mkhppa3 /proc]$ cat 224/status
Name:   dnetc
State:  D (disk sleep)
Tgid:   224
Pid:    224
PPid:   1
TracerPid:      0
Uid:    1001    1001    1001    1001
Gid:    100     100     100     100
FDSize: 64
Groups: 100
VmSize:     2300 kB
VmLck:         0 kB
VmRSS:      1136 kB
VmData:       84 kB
VmStk:         0 kB
VmExe:       348 kB
VmLib:      1368 kB
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 8000000000001000
SigCgt: 0000000003004027
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Now we are about to test with only 2 dnetc, no seti, to see whether it 
crashes or not.

We'll keep the m-l informed.

Greetings,


Thibaut VARENE
PA/Linux ESIEE Team
http://pateam.esiee.fr/