2.2.19 locks up on SMP

2.2.19 locks up on SMP

[Ion Badulescu]

Hi Alan,

Over the last week I've tried to upgrade a 4-CPU Xeon box to 2.2.19, but 
the it keeps locking up whenever the disks are stresses a bit, e.g. when 
updatedb is running. I get the following messages on the console:

wait_on_bh, CPU 1:
irq:  1 [1 0]
bh:   1 [1 0]
<[8010af71]>

over and over again, until somebody pushes the reset button.  8010af71 is 
somewhere in the middle of synchronize_bh().

The hardware configuration is: 4 Xeon/500MHz, 1GB RAM, 3 SCSI disks
attached to a symbios controller, 2 eepro100 interfaces. The kernel is
compiled with support for SMP and 2GB of RAM (hence the kernel address
starting with 8 instead of c).  It was compiled from a pristine source
tree, no patches were applied.

I had more problems with 2.2.19 and another SMP box, which was also 
locking up under stress. I'm not sure if it had the same messages on the 
console, since it's headless, but it was running the same 2.2.19 kernel as 
the previous one and was locking up in a very similar fashion. The 
hardware in that box is 2 P-III/750MHz, 512MB RAM, 1 IDE disk on a PIIX 
controller, and an unused aic7xxx SCSI controller with no SCSI devices 
attached to it.

Both boxes are rock-solid when running 2.2.18-SMP.

Any ideas? Has anybody else reported this with 2.2.19?

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[idalton]

On Sun, Apr 29, 2001 at 01:16:04AM +0200, bert hubert wrote:
> On Sat, Apr 28, 2001 at 02:21:29PM -0700, Ion Badulescu wrote:
> > Hi Alan,
> > 
> > Over the last week I've tried to upgrade a 4-CPU Xeon box to 2.2.19, but 
> > the it keeps locking up whenever the disks are stresses a bit, e.g. when 
> > updatedb is running. I get the following messages on the console:
> > 
> > wait_on_bh, CPU 1:
> > irq:  1 [1 0]
> > bh:   1 [1 0]
> > <[8010af71]>
> 
> Obvious question is, which compiler.

I hadn't seen any locks, but (on a dual Pmmx 200) it started crawling
right after the NIC module (tulip) was loaded. System load decided to
skyrocket.

Yadda... 2.2.19 with devfs patch.
bicycle:~# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.3/specs
gcc version 2.95.3 20010315 (Debian release)

Might be the same problem.

-- Ferret

Re: 2.2.19 locks up on SMP

[bert hubert]

On Sat, Apr 28, 2001 at 02:21:29PM -0700, Ion Badulescu wrote:
> Hi Alan,
> 
> Over the last week I've tried to upgrade a 4-CPU Xeon box to 2.2.19, but 
> the it keeps locking up whenever the disks are stresses a bit, e.g. when 
> updatedb is running. I get the following messages on the console:
> 
> wait_on_bh, CPU 1:
> irq:  1 [1 0]
> bh:   1 [1 0]
> <[8010af71]>

Obvious question is, which compiler.

-- 
http://www.PowerDNS.com      Versatile DNS Services  
Trilab                       The Technology People   
'SYN! .. SYN|ACK! .. ACK!' - the mating call of the internet

Re: 2.2.19 locks up on SMP

[Tim Moore]

> > Obvious question is, which compiler.
> 
> I hadn't seen any locks, but (on a dual Pmmx 200) it started crawling
> right after the NIC module (tulip) was loaded. System load decided to
> skyrocket.
> 
> Yadda... 2.2.19 with devfs patch.
> bicycle:~# gcc -v
> Reading specs from /usr/lib/gcc-lib/i386-linux/2.95.3/specs
> gcc version 2.95.3 20010315 (Debian release)
> 
> Might be the same problem.

Twin Abit BP6's, 2.2.19 + 9-Apr ide patch, no problems.

egcs-2.91.66

tulip.c:v0.91g-ppc 7/16/99 becker@cesdis.gsfc.nasa.gov
eth0: Lite-On 82c168 PNIC rev 32 at 0xc800, 00:A0:CC:57:89:93, IRQ 16.

-- 
  |  650.390.9613  |  6502247437@messaging.cellone-sf.com

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Sun, 29 Apr 2001 01:16:04 +0200, bert hubert <ahu@ds9a.nl> wrote:
> On Sat, Apr 28, 2001 at 02:21:29PM -0700, Ion Badulescu wrote:
>> Hi Alan,
>> 
>> Over the last week I've tried to upgrade a 4-CPU Xeon box to 2.2.19, but 
>> the it keeps locking up whenever the disks are stresses a bit, e.g. when 
>> updatedb is running. I get the following messages on the console:
>> 
>> wait_on_bh, CPU 1:
>> irq:  1 [1 0]
>> bh:   1 [1 0]
>> <[8010af71]>
> 
> Obvious question is, which compiler.

These are rh62 systems, the compiler is egcs-1.1.2. So that's not it.

I'd be willing to do the binary search through the 2.2.19pre series,
but I'd rather avoid it if it's a known bug. It's pretty painful, both
for myself and for the real users of this box, to go through the pains
of 10-20 cycles of reboot-crash-fsck_3_large_disks...

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Alan Cox]

> I had more problems with 2.2.19 and another SMP box, which was also 
> locking up under stress. I'm not sure if it had the same messages on the 
> console, since it's headless, but it was running the same 2.2.19 kernel as 
> the previous one and was locking up in a very similar fashion. The 
> hardware in that box is 2 P-III/750MHz, 512MB RAM, 1 IDE disk on a PIIX 
> controller, and an unused aic7xxx SCSI controller with no SCSI devices 
> attached to it.
> 
> Both boxes are rock-solid when running 2.2.18-SMP.
> 
> Any ideas? Has anybody else reported this with 2.2.19?

A couple. It looks lik the VM changes may have upset something (based on
reports saying it began at that point). Can you see if 2.2.19pre stuff is
stable ?

Re: 2.2.19 locks up on SMP

[Andrea Arcangeli]

On Mon, Apr 30, 2001 at 06:55:54PM +0100, Alan Cox wrote:
> A couple. It looks lik the VM changes may have upset something (based on
> reports saying it began at that point). Can you see if 2.2.19pre stuff is
> stable ?

I also have reports but related to the network driver updates. So I
suggest to try again with 2.2.19 but with the drivers/net/* of 2.2.18.

Andrea

Re: 2.2.19 locks up on SMP

[Andrea Arcangeli]

On Mon, Apr 30, 2001 at 08:15:47PM +0200, Andrea Arcangeli wrote:
> suggest to try again with 2.2.19 but with the drivers/net/* of 2.2.18.

even better try vanilla 2.2.19aa2 and if it crashes too, try 2.2.19aa2
plus the drivers/net/* of 2.2.18.

Andrea

Re: 2.2.19 locks up on SMP

[Alan Cox]

> On Mon, Apr 30, 2001 at 06:55:54PM +0100, Alan Cox wrote:
> > A couple. It looks lik the VM changes may have upset something (based on
> > reports saying it began at that point). Can you see if 2.2.19pre stuff is
> > stable ?
> 
> I also have reports but related to the network driver updates. So I
> suggest to try again with 2.2.19 but with the drivers/net/* of 2.2.18.

Thats probably a better starting point. Its easier to back out than the VM
changes and it would also explain the reports I saw.

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Mon, 30 Apr 2001, Alan Cox wrote:

> > I also have reports but related to the network driver updates. So I
> > suggest to try again with 2.2.19 but with the drivers/net/* of 2.2.18.
> 
> Thats probably a better starting point. Its easier to back out than the VM
> changes and it would also explain the reports I saw.

Except that the only driver I'm using is eepro100, and the only change to 
that driver was the patch I submitted myself and which is also in 2.4.

Also, another data point: those two SMP boxes have been running 2.2.18 + 
Andrea's VM-global patch since January, without a hitch.

Ok, so onto the binary search through the 2.2.19pre series...

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Mohammad A. Haque]

On Mon, 30 Apr 2001, Ion Badulescu wrote:

> Except that the only driver I'm using is eepro100, and the only change to
> that driver was the patch I submitted myself and which is also in 2.4.
>
> Also, another data point: those two SMP boxes have been running 2.2.18 +
> Andrea's VM-global patch since January, without a hitch.
>
> Ok, so onto the binary search through the 2.2.19pre series...

Just to give another data point...

2.2.19 + LVM patches - dual P3 550
1 GB RAM
eepro100
ncr53c8xx scsi
mylex accelRAID 1100 RAID controller

We've transferred around 1 GB of stuff over the network and about 200 GB
between two raids w/o problems in a little under 3 days.

We've only scratched into swap. Free show 128K being used.

-- 

=====================================================================
Mohammad A. Haque                              http://www.haque.net/
                                               mhaque@haque.net

  "Alcohol and calculus don't mix.             Project Lead
   Don't drink and derive." --Unknown          http://wm.themes.org/
                                               batmanppc@themes.org
=====================================================================

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Mon, 30 Apr 2001, Mohammad A. Haque wrote:

> Just to give another data point...
> 
> 2.2.19 + LVM patches - dual P3 550
> 1 GB RAM
> eepro100
> ncr53c8xx scsi
> mylex accelRAID 1100 RAID controller
> 
> We've transferred around 1 GB of stuff over the network and about 200 GB
> between two raids w/o problems in a little under 3 days.
> 
> We've only scratched into swap. Free show 128K being used.

Ok. Have you tried running a large bonnie (1GB) while at the same time 
pummeling the network? That's how I trigger it, quite reliably.

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Mon, 30 Apr 2001, Ion Badulescu wrote:

> Ok, so onto the binary search through the 2.2.19pre series...

I think it started in 2.2.19pre10. I can reproduce the hang on pre10, 
quite easily, but I couldn't reproduce it on pre5, pre7 and pre9. I'll try 
a few other pre versions, just to make sure.

One of the things that are different between pre10 and the others is NFS:
the client is broken in all versions except pre10. I'm not sure how much 
it matters, since I wasn't pounding on NFS. Anyway, that's why I want to 
try a few other kernels -- maybe it does matter after all.

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Alan Cox]

> I think it started in 2.2.19pre10. I can reproduce the hang on pre10, 
> quite easily, but I couldn't reproduce it on pre5, pre7 and pre9. I'll try 
> a few other pre versions, just to make sure.

Ok the main candidates there would be:

	The sunrpc/nfs changes
	EEpro100/starfire
	aic7xxx

which of those drivers are you using ?

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Alan Cox wrote:

> Ok the main candidates there would be:
> 
> 	The sunrpc/nfs changes

I'm currently testing this one -- just preparing to reboot pre9 + these 
changes.

> 	EEpro100/starfire

eepro100 is in use. But that patch is harmless.

> 	aic7xxx

Loaded but not used, no devices attached to it.

Right now I'm pretty sure it's the NFS/SunRPC changes, but I'll know for 
sure in about 30 minutes.

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Ion Badulescu wrote:

> > 	aic7xxx
> 
> Loaded but not used, no devices attached to it.

Scratch that, I was confusing it with another box. There is no trace of 
aic7xxx on this system.

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Ion Badulescu wrote:

> Right now I'm pretty sure it's the NFS/SunRPC changes, but I'll know for 
> sure in about 30 minutes.

As I suspected, 2.2.19pre9 + the NFS/SunRPC changes locked up under load 
with the now familiar:

wait_on_bh, CPU 2:
irq:  1 [0 0]
bh:   1 [0 1]
<[8010af71]> 

This time it happened precisely when I ran a ls -lR on a large tree over 
NFS, so I'm pretty sure this is it.

I'll do another test, 2.2.18 + the NFS/SunRPC changes, and see how it 
goes. Hopefully they'll apply easily...

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Ion Badulescu wrote:

> I'll do another test, 2.2.18 + the NFS/SunRPC changes, and see how it 
> goes. Hopefully they'll apply easily...

As I suspected, 2.2.18 + all the NFS/NFSd/SunRPC changes present in 
2.2.19pre10 locks up with wait_on_bh as soon as I run ls -lR on a large 
NFS directory tree, while at the same time pummeling the network and the 
local disks.

NFS is not enough to trigger the bug, the extra disk/network stress *is*
necessary. The network stress actually seems to be enough, I just 
triggered the bug again...

2.2.18 vanilla is fine.

So I guess the next round is in Trond's court. :-)

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Trond Myklebust]

>>>>> " " == Ion Badulescu <ionut@cs.columbia.edu> writes:

     > On Tue, 1 May 2001, Ion Badulescu wrote:
    >> I'll do another test, 2.2.18 + the NFS/SunRPC changes, and see
    >> how it goes. Hopefully they'll apply easily...

     > As I suspected, 2.2.18 + all the NFS/NFSd/SunRPC changes
     > present in 2.2.19pre10 locks up with wait_on_bh as soon as I
     > run ls -lR on a large NFS directory tree, while at the same
     > time pummeling the network and the local disks.

     > NFS is not enough to trigger the bug, the extra disk/network
     > stress *is* necessary. The network stress actually seems to be
     > enough, I just triggered the bug again...

     > 2.2.18 vanilla is fine.

     > So I guess the next round is in Trond's court. :-)

Did you apply the following patch which I put out on the lists a
couple of weeks ago?

Cheers,
  Trond

--- net/sunrpc/xprt.c.orig	Sun Mar 25 18:37:42 2001
+++ net/sunrpc/xprt.c	Wed Apr 18 11:10:21 2001
@@ -1145,9 +1145,11 @@
 		unsigned long	oldflags;
 		spin_lock_irqsave(&xprt_sock_lock, oldflags);
 		xprt->snd_task = NULL;
-		if (!rpc_wake_up_next(&xprt->sending) && xprt->stream)
+		if (!rpc_wake_up_next(&xprt->sending) && xprt->stream) {
+			spin_unlock_irqrestore(&xprt_sock_lock, oldflags);
 			xprt_add_tcp_timer(xprt, RPCXPRT_TIMEOUT);
-		spin_unlock_irqrestore(&xprt_sock_lock, oldflags);
+		} else
+			spin_unlock_irqrestore(&xprt_sock_lock, oldflags);
 	}
 }
 

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Trond Myklebust wrote:

> Did you apply the following patch which I put out on the lists a
> couple of weeks ago?

No, I was testing with 2.2.19 and then I started going back into the 
2.2.19pre series until I found the culprit.

I'll give your patch a spin tomorrow, after I catch some zzz's. :-)

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Trond Myklebust]

>>>>> " " == Ion Badulescu <ionut@cs.columbia.edu> writes:

     > On Tue, 1 May 2001, Trond Myklebust wrote:
    >> Did you apply the following patch which I put out on the lists
    >> a couple of weeks ago?

     > No, I was testing with 2.2.19 and then I started going back
     > into the 2.2.19pre series until I found the culprit.

     > I'll give your patch a spin tomorrow, after I catch some
     > zzz's. :-)

Right you are.

FYI I've now put up those patches of which I am aware against 2.2.19
on

  http://www.fys.uio.no/~trondmy/src/2.2.19

I'll try to keep that area updated with a brief explanation for each
patch...

Cheers,
  Trond

Re: 2.2.19 locks up on SMP

[Ion Badulescu]

On Tue, 1 May 2001, Trond Myklebust wrote:

>      > I'll give your patch a spin tomorrow, after I catch some
>      > zzz's. :-)
> 
> Right you are.

And indeed, the tcp-hang patch fixed the problem! Thanks a lot!

> FYI I've now put up those patches of which I am aware against 2.2.19
> on
> 
>   http://www.fys.uio.no/~trondmy/src/2.2.19
> 
> I'll try to keep that area updated with a brief explanation for each
> patch...

That's where I tried looking first, two days ago, but couldn't find 
anything, and I must have overlooked the patch you sent to the list.

Thanks for crediting me, btw. :-) Just one little nit: the readdir() 
problem appears only when using glibc-2.0, glibc-2.1 seems to be fine.

Thanks again to everybody,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: 2.2.19 locks up on SMP

[Scott Nursten]

Hi there, 

Was there ever any resolution to this thread? I'm running a bunch of Compaq DL-360's which seem to work fine on the 2.2.19pre series. As soon as I go to 2.2.19, networking doesn't work. Machines are spec'd as follows:

2 x P3-933
1.4GB RAM
Compaq RLO card
Compaq Smart2 Array Controller
2 x EtherExpress Pro onboard
2 x EtherExpress Pro PCI (the dual port server adapter from Intel)

Caveat: whenever I run `ifconfig device down` the machine locks up completely. 

Willing to give any information necessary in exchange for working kernel :) Any takers? Tell me what you guys need. 

Rgds, 

-- 
Scott Nursten - Systems Administrator
Streets Online Ltd.

Direct:		+44 (0) 1293 744 122
Business:       +44 (0) 1293 402 040
Fax:            +44 (0) 1293 402 050
Email:          scottn@streetsonline.co.uk

      -----------------------------------------------------------------------
	"Unix is user friendly. It's just selective when choosing friends."
      -----------------------------------------------------------------------

Re: 2.2.19 locks up on SMP

[Scott Nursten]

Hey guys,

Just to confirm - I've compiled 2.2.19 w/out SMP and it works sweet. 

Rgds,

Scott

Scott Nursten wrote:
> 
> Hi there,
> 
> Was there ever any resolution to this thread? I'm running a bunch of Compaq DL-360's which seem to work fine on the 2.2.19pre series. As soon as I go to 2.2.19, networking doesn't work. Machines are spec'd as follows:
> 
> 2 x P3-933
> 1.4GB RAM
> Compaq RLO card
> Compaq Smart2 Array Controller
> 2 x EtherExpress Pro onboard
> 2 x EtherExpress Pro PCI (the dual port server adapter from Intel)
> 
> Caveat: whenever I run `ifconfig device down` the machine locks up completely.
> 
> Willing to give any information necessary in exchange for working kernel :) Any takers? Tell me what you guys need.
> 
> Rgds,
> 
> --
> Scott Nursten - Systems Administrator
> Streets Online Ltd.
> 
> Direct:         +44 (0) 1293 744 122
> Business:       +44 (0) 1293 402 040
> Fax:            +44 (0) 1293 402 050
> Email:          scottn@streetsonline.co.uk
> 
>       -----------------------------------------------------------------------
>         "Unix is user friendly. It's just selective when choosing friends."
>       -----------------------------------------------------------------------
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Scott Nursten - Systems Administrator
Streets Online Ltd.

Direct:		+44 (0) 1293 744 122
Business:       +44 (0) 1293 402 040
Fax:            +44 (0) 1293 402 050
Email:          scottn@streetsonline.co.uk

      -----------------------------------------------------------------------
	"Unix is user friendly. It's just selective when choosing friends."
      -----------------------------------------------------------------------