Re: PROBLEM: kernel crashes on RAID1 drive error

[Jens Axboe <axboe@suse.de>]

On Thu, Oct 21 2004, Paul Clements wrote:
> Jens Axboe wrote:
> >On Thu, Oct 21 2004, Paul Clements wrote:
> >
> >>Jens Axboe wrote:
> >>
> >>>On Wed, Oct 20 2004, Mark Rustad wrote:
> >>>
> >>>
> >>>>Folks,
> >>>>
> >>>>I have been having trouble with kernel crashes resulting from RAID1 
> >>>>component device failures. I have been testing the robustness of an 
> >>>>embedded system and have been using a drive that is known to fail after 
> >>>>a time under load. When this device returns a media error, I always 
> >>>>wind up with either a kernel hang or reboot. In this environment, each 
> >>>>drive has four partitions, each of which is part of a RAID1 with its 
> >>>>partner on the other device. Swap is on md2 so even it should be 
> >>>>robust.
> >>>>
> >>>>I have gotten this result with the SuSE standard i386 smp kernels 
> >>>>2.6.5-7.97 and 2.6.5-7.108. I also get these failures with the 
> >>>>kernel.org kernels 2.6.8.1, 2.6.9-rc4 and 2.6.9.
> >>>>
> >>>>The hardware setup is a two cpu Nacona with an Adaptec 7902 SCSI 
> >>>>controller with two Seagate drives on a SAF-TE bus. I run three or four 
> >>>>dd commands copying /dev/md0 to /dev/null to provide the activity that 
> >>>>stimulates the failure.
> >>>>
> >>>>I suspect that something is going wrong in the retry of the failed I/O 
> >>>>operations, but I'm really not familiar with any of this area of the 
> >>>>kernel at all.
> >>>>
> >>>>In one failure, I get the following messages from kernel 2.6.9:
> >>>>
> >>>>raid1: Disk failure on sdb1, disabling device.
> >>>>raid1: sdb1: rescheduling sector 176
> >>>>raid1: sda1: redirecting sector 176 to another mirror
> >>>>raid1: sdb1: rescheduling sector 184
> >>>>raid1: sda1: redirecting sector 184 to another mirror
> >>>>Incorrect number of segments after building list
> >>>>counted 2, received 1
> >>>>req nr_sec 0, cur_nr_sec 7
> >>>
> >>>
> >>>This should be fixed by this patch, can you test it?
> >>
> >>There may well be two problems here, but the original problem you're 
> >>seeing (infinite read retries, and failures) is due to a bug in raid1. 
> >>Basically the bio handling on read error retry was not quite right. Neil 
> >>Brown just posted the patch to correct this a couple of days ago:
> >>
> >>http://marc.theaimsgroup.com/?l=linux-raid&m=109824318202358&w=2
> >>
> >>Please try that. (If you need a patch that applies to SUSE 2.6.5, I also 
> >>have a version of the patch which should apply to that).
> >
> >
> >Is 2.6.9 not uptodate wrt those raid1 patches?!
> 
> Unfortunately, no. This latest problem (the one he's reporting) is not 
> fixed in mainline. I discovered the problem a month or so ago while 
> testing with SLES 9. I posted a patch and Neil expanded on it (to 
> include raid10, which is now in mainline, and also suffers from the same 
> problem). Neil just posted the patch two days ago to linux-raid, so I 
> expect it's in -mm now.

Irk, that's too bad. So we are now looking at probably a month before
mainline has a stable release with that fixed too :/

> >>Please be aware that there are several other bugs in the SUSE 2.6.5-7.97 
> >>kernel in md and raid1 (basically it's a matter of that kernel being 
> >>somewhat behind mainline, where most of these bugs are now fixed). I've 
> >>sent several patches to SUSE to fix these issues, that hopefully will 
> >>get into their SP1 release that should be forthcoming soon...
> >
> >
> >-97 is the release kernel, -111 is the current update kernel. And it has
> >those raid1 issues fixed already, at least the ones that are known. The
> >scsi segment issue is not, however.
> 
> Thanks. Good to know that. -111 is currently available to customers? We 
> may recommend that our customers use that, rather than patching -97 
> ourselves.

Yes it is, it's generally available through the online updates.

-- 
Jens Axboe