2.6.20: reproducible hard lockup with RAID-5 resync

2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

I think I have found an easily-reproducible bug in Linux 2.6.20. I have
already applied the "Fix various bugs with aligned reads in RAID5"
patch, and that had no effect. It appears to be related to the resync
process, and makes the system lock up, hard.

The steps to reproduce are:
1. Be running Linux 2.6.20 and do whatever is necessary to prepare for a
crash (close open files, sync, unmount filesystems, or whatever).
Alternatively, just boot with 'init=/bin/bash'.
2. Run 'mdadm -S /dev/md2', where /dev/md2 is a RAID-5.
3. Run 'mdadm -A /dev/md2 -U resync'.
4. Wait about 1 second. The system will lock up.

During the lock up, nothing is printed to the console, and the magic
SysRQ key has no effect; I have to poke the reset button. Normally, I
wouldn't rule out a hardware problem, but I have reasonable faith in my
computer. Neither memtest86+ nor cpuburn nor normal operation have
flushed out any instability.

Upon reboot, 2.6.20 will lock up almost immediately when it tries to
resync the array. This appears to occur regardless of whether the resync
is just starting; if I run 2.6.19 for a while until the resync is, say,
50% done and then reboot to 2.6.20, the lockup still happens.

I have provided what I hope is enough information below.

--------------------------------------------------------------------
System information:
Athlon64 3400+
64-bit Linux 2.6.20 compiled with GCC 4.1.2
64-bit Debian Sid
RAID-5 of 5 devices:
   /dev/hda   (IDE hard drive)
   /dev/sda6  (partition on SATA hard drive)
   /dev/sdb   (SATA hard drive)
   /dev/sdc6  (partition on SATA hard drive)
   /dev/sdd   (SATA hard drive)

--------------------------------------------------------------------
bugfood:~# mdadm -D /dev/md2
/dev/md2:
        Version : 00.90.03
  Creation Time : Mon May 29 22:13:47 2006
     Raid Level : raid5
     Array Size : 781433344 (745.23 GiB 800.19 GB)
    Device Size : 195358336 (186.31 GiB 200.05 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Thu Feb 15 22:07:26 2007
          State : active, resyncing
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 26% complete

           UUID : d016a205:bd3106ef:b19cb15b:b6d70494
         Events : 0.3971003

    Number   Major   Minor   RaidDevice State
       0       8        6        0      active sync   /dev/sda6
       1       8       38        1      active sync   /dev/sdc6
       2       3        0        2      active sync   /dev/hda
       3       8       16        3      active sync   /dev/sdb
       4       8       48        4      active sync   /dev/sdd

--------------------------------------------------------------------

Thank you,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Neil Brown]

On Thursday February 15, bugfood-ml@fatooh.org wrote:
> I think I have found an easily-reproducible bug in Linux 2.6.20. I have
> already applied the "Fix various bugs with aligned reads in RAID5"
> patch, and that had no effect. It appears to be related to the resync
> process, and makes the system lock up, hard.

I'm guessing that the problem is at a lower level than raid.
What IDE/SATA controllers do you have?  Google to see if anyone else
has had problems with them in 2.6.20.
> 
> During the lock up, nothing is printed to the console, and the magic
> SysRQ key has no effect; I have to poke the reset button.

Sound's like interrupts are disabled, but x86_64 always enables the
NMI watchdog which should trigger if interrupts are off for too long. 

Do you have CONFIG_DETECT_SOFTLOCKUP=y in your .config (it is in the
kernel debugging options menu I think).  If not, setting that would be
worth a try.

A raid5 resync across 5 sata drives on a couple of different
silicon-image controllers doesn't lock up for me.

NeilBrown

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Neil Brown wrote:
> On Thursday February 15, bugfood-ml@fatooh.org wrote:
>> I think I have found an easily-reproducible bug in Linux 2.6.20. I have
>> already applied the "Fix various bugs with aligned reads in RAID5"
>> patch, and that had no effect. It appears to be related to the resync
>> process, and makes the system lock up, hard.
> 
> I'm guessing that the problem is at a lower level than raid.
> What IDE/SATA controllers do you have?  Google to see if anyone else
> has had problems with them in 2.6.20.
>> During the lock up, nothing is printed to the console, and the magic
>> SysRQ key has no effect; I have to poke the reset button.
> 
> Sound's like interrupts are disabled, but x86_64 always enables the
> NMI watchdog which should trigger if interrupts are off for too long. 
> 
> Do you have CONFIG_DETECT_SOFTLOCKUP=y in your .config (it is in the
> kernel debugging options menu I think).  If not, setting that would be
> worth a try.
> 
> A raid5 resync across 5 sata drives on a couple of different
> silicon-image controllers doesn't lock up for me.

Wow, thanks for the quick response. I have to go to bed now, but I'll
try to get you that information tomorrow.

Thanks,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Neil Brown wrote:
> On Thursday February 15, bugfood-ml@fatooh.org wrote:
>> I think I have found an easily-reproducible bug in Linux 2.6.20. I have
>> already applied the "Fix various bugs with aligned reads in RAID5"
>> patch, and that had no effect. It appears to be related to the resync
>> process, and makes the system lock up, hard.
> 
> I'm guessing that the problem is at a lower level than raid.
> What IDE/SATA controllers do you have?  Google to see if anyone else
> has had problems with them in 2.6.20.

I have an nForce3 motherboard. lspci calls my IDE:
nVidia Corporation CK8S Parallel ATA Controller (v2.5) (rev a2)
...and my SATA:
nVidia Corporation CK8S Serial ATA Controller (v2.5) (rev a2)

I'm using libata for my SATA drives and the old IDE driver for my IDE 
drive. For reference, I have uploaded my kernel configuration and the 
output of lspci:
http://fatooh.org/files/tmp/config-2.6.20
http://fatooh.org/files/tmp/lspci-v

Anyway, I googled a bit, and I also looked through the recent threads in 
the linux-kernel archives, but I haven't found anything. I don't follow 
kernel development closely, though, so it's quite possible I missed 
something.

When I get home (late) tonight I'll try running dd and badblocks on the 
corresponding drives and partitions.

>> During the lock up, nothing is printed to the console, and the magic
>> SysRQ key has no effect; I have to poke the reset button.
> 
> Sound's like interrupts are disabled, but x86_64 always enables the
> NMI watchdog which should trigger if interrupts are off for too long. 

How long is "too long"? I waited a few minutes, at least, on the first 
few tries.

> Do you have CONFIG_DETECT_SOFTLOCKUP=y in your .config (it is in the
> kernel debugging options menu I think).  If not, setting that would be
> worth a try.

I do indeed have CONFIG_DETECT_SOFTLOCKUP enabled. The Kconfig 
description says it should detect lockups > 10 seconds, I've waited 
longer than that many times.

> A raid5 resync across 5 sata drives on a couple of different
> silicon-image controllers doesn't lock up for me.

Heck. ;)  Would it by any chance make a difference that I'm running 
RAID-5 across a mixture of drives and partitions?

Thanks again,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Corey Hickey wrote:
> When I get home (late) tonight I'll try running dd and badblocks on the 
> corresponding drives and partitions.

Well, I haven't been able to reproduce the problem that way. I tried the
following:

$ dd id=/dev/hda of=/dev/null
$ badblocks /dev/hda
$ badblocks -n /dev/hda

...and the same for sda6, sdb, sdc6, sdd, and md2. In each case I killed
the test after several seconds, on the assumption that if the problem
was reproducible within less than a second by triggering a resync, it
wouldn't take long any other way.

If anyone has any suggestions for further tests I can do, I'll be happy
to try them out.

Thanks,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Justin Piszcz]

On Sat, 17 Feb 2007, Corey Hickey wrote:

> Corey Hickey wrote:
>> When I get home (late) tonight I'll try running dd and badblocks on the
>> corresponding drives and partitions.
>
> Well, I haven't been able to reproduce the problem that way. I tried the
> following:
>
> $ dd id=/dev/hda of=/dev/null
> $ badblocks /dev/hda
> $ badblocks -n /dev/hda
>
> ...and the same for sda6, sdb, sdc6, sdd, and md2. In each case I killed
> the test after several seconds, on the assumption that if the problem
> was reproducible within less than a second by triggering a resync, it
> wouldn't take long any other way.
>
> If anyone has any suggestions for further tests I can do, I'll be happy
> to try them out.
>
> Thanks,
> Corey
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I noticed you are running a disk on /dev/hda, if this is an older mobo and 
you use multiple PCI IDE controllers, if they are in certain slots, or you 
put too many in, they can cause these problems, I have had a few mobos 
that do it if the PCI cards are in a 'certain combination'-- I have had 
this happen with an MSI and ABIT board.

Justin.

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Justin Piszcz wrote:
>> If anyone has any suggestions for further tests I can do, I'll be happy
>> to try them out.
>
> I noticed you are running a disk on /dev/hda, if this is an older mobo and 
> you use multiple PCI IDE controllers, if they are in certain slots, or you 
> put too many in, they can cause these problems, I have had a few mobos 
> that do it if the PCI cards are in a 'certain combination'-- I have had 
> this happen with an MSI and ABIT board.

I have a single Promise PCI IDE controller, but I don't have any hard
drives hooked up to it. /dev/hda and all the SATA drives are hooked up
to the onboard nVidia controllers.

My motherboard is a "DFI LanParty UT nf3 250Gb" (despite the unwieldy
name, it's a rather nice board) and I think it dates back to August
2004. I built my machine in November 2004, and I've never had an IDE or
SATA problem with it.

I started testing (and then using) my RAID-5 in May 2006 with Linux
2.6.16. I built the array degraded, so I know that resync worked for me
then. So did restriping, which I did twice. Since then, I've run 2.6.17
and 2.6.19, without any problems.

Thanks,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Corey Hickey wrote:
> I think I have found an easily-reproducible bug in Linux 2.6.20. I have
> already applied the "Fix various bugs with aligned reads in RAID5"
> patch, and that had no effect. It appears to be related to the resync
> process, and makes the system lock up, hard.

I now have a different build of 2.6.20 running resync without the
reported problem. I'll try to isolate the specific difference between
the two builds some time tomorrow.

-Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Corey Hickey wrote:
> Corey Hickey wrote:
>> I think I have found an easily-reproducible bug in Linux 2.6.20. I have
>> already applied the "Fix various bugs with aligned reads in RAID5"
>> patch, and that had no effect. It appears to be related to the resync
>> process, and makes the system lock up, hard.
> 
> I now have a different build of 2.6.20 running resync without the
> reported problem. I'll try to isolate the specific difference between
> the two builds some time tomorrow.

Ok, so the difference is CONFIG_SYSFS_DEPRECATED. If that is not
defined, the kernel locks up. There's not a lot of code under
#ifdef/#ifndef CONFIG_SYSFS_DEPRECATED, but since I'm not familiar with
any of it I don't expect trying to locate the bug on my own would be
very productive.

Neil, do you have CONFIG_SYSFS_DEPRECATED enabled? If so, does disabling
it reproduce my problem? If you can't reproduce it, should it take the
problem over to linux-kernel?

Thanks,
Corey

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Neil Brown]

On Sunday February 18, bugfood-ml@fatooh.org wrote:
> 
> Ok, so the difference is CONFIG_SYSFS_DEPRECATED. If that is not
> defined, the kernel locks up. There's not a lot of code under
> #ifdef/#ifndef CONFIG_SYSFS_DEPRECATED, but since I'm not familiar with
> any of it I don't expect trying to locate the bug on my own would be
> very productive.
> 
> Neil, do you have CONFIG_SYSFS_DEPRECATED enabled? If so, does disabling
> it reproduce my problem? If you can't reproduce it, should it take the
> problem over to linux-kernel?

# CONFIG_SYSFS_DEPRECATED is not set

No, it is not set, and yet it all still works for me.
It is very hard to see how this CONFIG option can make a difference.
Have you double checked that setting it removed the problem and
clearing it causes the problem?
If so, then maybe an email to linux-kernel would be appropriate.
Include the full config..

NeilBrown

Re: 2.6.20: reproducible hard lockup with RAID-5 resync

[Corey Hickey]

Neil Brown wrote:
>> Ok, so the difference is CONFIG_SYSFS_DEPRECATED. If that is not
>> defined, the kernel locks up. There's not a lot of code under
>> #ifdef/#ifndef CONFIG_SYSFS_DEPRECATED, but since I'm not familiar with
>> any of it I don't expect trying to locate the bug on my own would be
>> very productive.
>>
>> Neil, do you have CONFIG_SYSFS_DEPRECATED enabled? If so, does disabling
>> it reproduce my problem? If you can't reproduce it, should it take the
>> problem over to linux-kernel?
> 
> # CONFIG_SYSFS_DEPRECATED is not set
> 
> No, it is not set, and yet it all still works for me.

Dang, again. :)

> It is very hard to see how this CONFIG option can make a difference.
> Have you double checked that setting it removed the problem and
> clearing it causes the problem?

Yes, it seems odd to me too, but I have double-checked. If I build a
kernel with CONFIG_SYSFS_DEPRECATED enabled, it works; if I disable that
option and rebuild the kernel, it locks up.

I just tried running 'make defconfig' and then enabling only RAID,
RAID-0, RAID-1, and RAID-4/5/6. If I then disable
CONFIG_SYSFS_DEPRECATED, there aren't any problems. ...so, I'll try to
isolate the problem some more later.

Thanks,
Corey