Re: File corruption on LVM2 on top of software RAID1

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

"Simon Matter" <simon.matter@invoca.ch> wrote:
>
> Hi,
> 
> Please CC me as I'm not subscribed to the kernel list.
> 
> I had a hard time identifying a serious problem in the 2.6 linux kernel.

Yes, it is a serious problem.

> It all started while evaluating RHEL4 for new servers. My data integrity
> tests gave me bad results - which I couldn't believe - and my first idea
> was - of course - bad hardware. I ordered new SCSI disks instead of the
> IDE disks, took another server, spent some money again, tried again and
> again. That's all long ago now...
> 
> In my tests I get corrupt files on LVM2 which is on top of software raid1.
> (This is a common setup even mentioned in the software RAID HOWTO and has
> worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
> favourite configuration). Now, I tested two different distributions, three
> kernels, three different filesystems and three different hardware. I can
> always reproduce it with the following easy scripts:
> 
> LOGF=/root/diff.log
> while true; do
>   rm -rf /home/XXX2
>   rsync -a /home/XXX/ /home/XXX2
>   date >> $LOGF
>   diff -r /home/XXX /home/XXX2 >> $LOGF
> done
> 
> the files in /home/XXX are ~15G of ISO images and rpms.
> 
> diff.log looks like this:
> Wed Aug  3 13:45:57 CEST 2005
> Binary files /home/XXX/ES3-U3/rhel-3-U3-i386-es-disc3.iso and
> /home/XXX2/ES3-U3/rhel-3-U3-i386-es-disc3.iso differ
> Wed Aug  3 14:09:14 CEST 2005
> Binary files /home/XXX/8.0/psyche-i386-disc1.iso and
> /home/XXX2/8.0/psyche-i386-disc1.iso differ
> Wed Aug  3 14:44:17 CEST 2005
> Binary files /home/XXX/7.3/valhalla-i386-disc3.iso and
> /home/XXX2/7.3/valhalla-i386-disc3.iso differ
> Wed Aug  3 15:15:05 CEST 2005
> Wed Aug  3 15:45:40 CEST 2005
> 
> 
> Tested software:
> 1) RedHat EL4
> kernel-2.6.9-11.EL
> vanilla 2.6.12.3 kernel
> filesystems: EXT2, EXT3, XFS
> 
> 2) NOVELL/SUSE 9.3
> kernel-default-2.6.11.4-21.7
> filesystem: EXT3
> 
> Tested Hardware:
> 1)
> - ASUS P2B-S board
> - CPU PIII 450MHz
> - Intel 440BX/ZX/DX Chipset
> - 4x128M memory (ECC enabled)
> - 2x IDE disks Seagate Barracuda 400G, connected to onboard "Intel PIIX4
> Ultra 33"
> - Promise Ultra100TX2 adapter for additional tests
> 
> 2)
> - DELL PowerEdge 1400
> - CPU PIII 800MHz
> - ServerWorks OSB4 Chipset
> - 4x256M memory (ECC enabled)
> - 2x U320 SCSI disks Maxtor Atlas 10K 146G
> - onboard Adaptec aic7899 Ultra160 SCSI adapter
> 
> 3)
> - DELL PowerEdge 1850
> - CPU P4 XEON 2.8GHz
> - 2G memory
> - 2x U320 SCSI disks Maxtor Atlas 10K 73G SCA
> - onboard LSI53C1030 SCSI adapter
> 
> I've put some files toghether from the last test on the PE1850 server:
> http://www.invoca.ch/bugs/linux-2.6-corruption-on-lvm2-on-raid1/
> 
> I've also filed a bug with RedHat:
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=164696
> 
> I have spent a lot of time on this bug because I consider it very serious.
> I'm not a kernel hacker but if there is anything I can do to fix this, let
> me know.
> 

Thanks for doing that.

There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have the
bug which this fix addresses.  So unless 2.6.9 has a different problem,
this won't help.

But still, you should include this in testing please.



diff -puN fs/bio.c~bio_clone-fix fs/bio.c
--- devel/fs/bio.c~bio_clone-fix	2005-07-28 00:39:40.000000000 -0700
+++ devel-akpm/fs/bio.c	2005-07-28 01:02:34.000000000 -0700
@@ -261,6 +261,7 @@ inline void __bio_clone(struct bio *bio,
 	 */
 	bio->bi_vcnt = bio_src->bi_vcnt;
 	bio->bi_size = bio_src->bi_size;
+	bio->bi_idx = bio_src->bi_idx;
 	bio_phys_segments(q, bio);
 	bio_hw_segments(q, bio);
 }
_

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

Andrew Morton <akpm@osdl.org> wrote:
>
> There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have the
>  bug which this fix addresses.

aargh, I see that it did fix it.

Don't blame me.  Blame people who screw up list threading by reading a
mail->news gateway and hitting "reply".

Re: File corruption on LVM2 on top of software RAID1

[Simon Matter]

> "Simon Matter" <simon.matter@invoca.ch> wrote:
>>
>> Hi,
>>
>> Please CC me as I'm not subscribed to the kernel list.
>>
>> I had a hard time identifying a serious problem in the 2.6 linux kernel.
>
> Yes, it is a serious problem.
>
>> It all started while evaluating RHEL4 for new servers. My data integrity
>> tests gave me bad results - which I couldn't believe - and my first idea
>> was - of course - bad hardware. I ordered new SCSI disks instead of the
>> IDE disks, took another server, spent some money again, tried again and
>> again. That's all long ago now...
>>
>> In my tests I get corrupt files on LVM2 which is on top of software
>> raid1.
>> (This is a common setup even mentioned in the software RAID HOWTO and
>> has
>> worked for me on RedHat 9 / kernel 2.4 for a long time now and it's my
>> favourite configuration). Now, I tested two different distributions,
>> three
>> kernels, three different filesystems and three different hardware. I can
>> always reproduce it with the following easy scripts:
>>
>
> Thanks for doing that.
>
> There's one fix against 2.6.12.3 which is needed, but 2.6.9 didn't have
> the
> bug which this fix addresses.  So unless 2.6.9 has a different problem,
> this won't help.

Sorry for the confusion, the 2.6.9 in question is a "RedHat Enterprise
Kernel" which includes lots of backported patches and features. So it is
no plain 2.6.9.

The real problem is that most current distributions include this bug in
one or the other way. They either have one of the affected versions in the
kernel package or have an older kernel with the bug backported. What I
know for sure is that the following distributions are affected:
RedHat Enterprise 4
Fedora Core 4
SuSE 9.3
Debian with the latest "unstable" 2.6 kernel
I couldn't verify Mandrake, Gentoo and others, but I expect them to be at
risk too.

While looking at some data corruption vulnerability reports on
Securityfocus I wonder why this issue does not get any attention from
distributors. I have an open bugzilla report with RedHat, have an open
customer service request with RedHat, have mailed peoples directly. No
real feedback.
I'm now in the process of restoring intergrity of my data with the help of
backups and mirrored data. Maybe I just care too much about other peoples
data, but I know that this bug will corrupt files on hundreds or thousands
of servers today and most people simply don't know it. Did I miss
something?

Simon

>
> But still, you should include this in testing please.
>
>
> diff -puN fs/bio.c~bio_clone-fix fs/bio.c
> --- devel/fs/bio.c~bio_clone-fix	2005-07-28 00:39:40.000000000 -0700
> +++ devel-akpm/fs/bio.c	2005-07-28 01:02:34.000000000 -0700
> @@ -261,6 +261,7 @@ inline void __bio_clone(struct bio *bio,
>  	 */
>  	bio->bi_vcnt = bio_src->bi_vcnt;
>  	bio->bi_size = bio_src->bi_size;
> +	bio->bi_idx = bio_src->bi_idx;
>  	bio_phys_segments(q, bio);
>  	bio_hw_segments(q, bio);
>  }
> _
>
>

Re: File corruption on LVM2 on top of software RAID1

[Andrew Morton]

"Simon Matter" <simon.matter@invoca.ch> wrote:
>
> While looking at some data corruption vulnerability reports on
>  Securityfocus I wonder why this issue does not get any attention from
>  distributors. I have an open bugzilla report with RedHat, have an open
>  customer service request with RedHat, have mailed peoples directly. No
>  real feedback.
>  I'm now in the process of restoring intergrity of my data with the help of
>  backups and mirrored data. Maybe I just care too much about other peoples
>  data, but I know that this bug will corrupt files on hundreds or thousands
>  of servers today and most people simply don't know it. Did I miss
>  something?

I guess the bug hit really rarely and maybe the reports were lost in the
permanent background noise of dodgy hardware.

We only found and fixed it last week, due to much sleuthing by Matthew
Stapleton.  I assume vendor updates are in the pipeline.

How to force a spare drive to take over in a RAID5?

[Mark Cuss]

Hi!

I have a 4 drive SW RAID5 running on my machine.  One of the drives is upset 
for some reason - I'm not sure if the drive itself is bad, but that's not 
too important right now.  The important thing is to get the RAID5 to stop 
using this drive and start using a spare drive that I just added.

The current raid status is:
[root@hal cdl]# lsraid -a /dev/md4
lsraid: Unable to read raid superblock from device "/dev/sdr1": Input/output 
error
lsraid: Device "/dev/sdr1" does not have a valid raid superblock
lsraid: Unable to read raid superblock from device "/dev/sdr1": Input/output 
error
lsraid: Device "/dev/sdr1" does not have a valid raid superblock
lsraid: Unable to read raid superblock from device "/dev/sdr1": Input/output 
error
lsraid: Device "/dev/sdr1" does not have a valid raid superblock
lsraid: Unable to read raid superblock from device "/dev/sdr1": Input/output 
error
lsraid: Device "/dev/sdr1" does not have a valid raid superblock
lsraid: Unable to read raid superblock from device "/dev/sdr1": Input/output 
error
lsraid: Device "/dev/sdr1" does not have a valid raid superblock
[dev   9,   4] /dev/md4         24989439.D4F0D908.67494E4A.97B3BA91 online
[dev   ?,   ?] (unknown)        00000000.00000000.00000000.00000000 missing
[dev   8,  161] /dev/sdk1        24989439.D4F0D908.67494E4A.97B3BA91 good
[dev   8,  177] /dev/sdl1        24989439.D4F0D908.67494E4A.97B3BA91 good
[dev   8,  193] /dev/sdm1        24989439.D4F0D908.67494E4A.97B3BA91 good
[dev  65,  33] /dev/sds1        24989439.D4F0D908.67494E4A.97B3BA91 spare

[root@hal cdl]#

Clearly, sdr is unhappy.  However, /proc/mdstat says that all the drives are 
online:
md4 : active raid5 sds1[4] sdr1[0] sdm1[3] sdl1[2] sdk1[1]
      430115904 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

I did a raidhotadd to add in a new drive, sds.  Now, I would like the array 
to stop using sdr and reconstruct all of the parity tables on sds so I can 
pull sdr and get it replaced or whatever...

Any ideas?

Thanks
Mark

Re: How to force a spare drive to take over in a RAID5?

[David Greaves]

Mark Cuss wrote:

> Hi!
>
> I have a 4 drive SW RAID5 running on my machine.  One of the drives is
> upset for some reason - I'm not sure if the drive itself is bad, but
> that's not too important right now.  The important thing is to get the
> RAID5 to stop using this drive and start using a spare drive that I
> just added.


> I did a raidhotadd to add in a new drive, sds.  Now, I would like the
> array to stop using sdr and reconstruct all of the parity tables on
> sds so I can pull sdr and get it replaced or whatever...
>
> Any ideas?

Install and read the manpage for mdadm

What kernel version?

David

Re: How to force a spare drive to take over in a RAID5?

[Mark Cuss]

Kernel 2.4.27

I've been using the old raidtools stuff and am new to mdadm - sorry if this 
is a obvisouly simply question...

After perusing the man page, is looks to me that I should use mdadm to mark 
the drive I want to remove as failed to force a rebuild on the spare drive. 
I want to double check that this is correct first though, as this md device 
contains 300 gig of production data.

Thanks for the help!

Mark

----- Original Message ----- 
From: "David Greaves" <david@dgreaves.com>
To: "Mark Cuss" <mcuss@cdlsystems.com>
Cc: <linux-raid@vger.kernel.org>
Sent: Sunday, August 07, 2005 5:29 AM
Subject: Re: How to force a spare drive to take over in a RAID5?


> Mark Cuss wrote:
>
>> Hi!
>>
>> I have a 4 drive SW RAID5 running on my machine.  One of the drives is
>> upset for some reason - I'm not sure if the drive itself is bad, but
>> that's not too important right now.  The important thing is to get the
>> RAID5 to stop using this drive and start using a spare drive that I
>> just added.
>
>
>> I did a raidhotadd to add in a new drive, sds.  Now, I would like the
>> array to stop using sdr and reconstruct all of the parity tables on
>> sds so I can pull sdr and get it replaced or whatever...
>>
>> Any ideas?
>
> Install and read the manpage for mdadm
>
> What kernel version?
>
> David
> 

Re: How to force a spare drive to take over in a RAID5?

[Mike Tran]

You're on the right track :) mdadm --fail /dev/sdr should do.

--
Mike T.

On Mon, 2005-08-08 at 16:43, Mark Cuss wrote:
> Kernel 2.4.27
> 
> I've been using the old raidtools stuff and am new to mdadm - sorry if this 
> is a obvisouly simply question...
> 
> After perusing the man page, is looks to me that I should use mdadm to mark 
> the drive I want to remove as failed to force a rebuild on the spare drive. 
> I want to double check that this is correct first though, as this md device 
> contains 300 gig of production data.
> 
> Thanks for the help!
> 
> Mark
> 
> ----- Original Message ----- 
> From: "David Greaves" <david@dgreaves.com>
> To: "Mark Cuss" <mcuss@cdlsystems.com>
> Cc: <linux-raid@vger.kernel.org>
> Sent: Sunday, August 07, 2005 5:29 AM
> Subject: Re: How to force a spare drive to take over in a RAID5?
> 
> 
> > Mark Cuss wrote:
> >
> >> Hi!
> >>
> >> I have a 4 drive SW RAID5 running on my machine.  One of the drives is
> >> upset for some reason - I'm not sure if the drive itself is bad, but
> >> that's not too important right now.  The important thing is to get the
> >> RAID5 to stop using this drive and start using a spare drive that I
> >> just added.
> >
> >
> >> I did a raidhotadd to add in a new drive, sds.  Now, I would like the
> >> array to stop using sdr and reconstruct all of the parity tables on
> >> sds so I can pull sdr and get it replaced or whatever...
> >>
> >> Any ideas?
> >
> > Install and read the manpage for mdadm
> >
> > What kernel version?
> >
> > David
> > 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How to force a spare drive to take over in a RAID5?

[Mark Cuss]

Hmm - Okay.  If I do that, I get "/dev/sdr1 does not appear to be an md 
device".  If I add --force with this it doesn't seem to matter.

I think the problem is that the superblock is hooched up on this drive and 
therefore mdadm doesn't think it's a part of the array, even though 
madam --detail /dev/md4 lists it:
[root@hal root]# mdadm --detail /dev/md4
/dev/md4:
        Version : 00.90.00
  Creation Time : Sat Sep 11 14:07:20 2004
     Raid Level : raid5
     Array Size : 430115904 (410.19 GiB 440.48 GB)
    Device Size : 143371968 (136.73 GiB 146.86 GB)
   Raid Devices : 4
  Total Devices : 5
Preferred Minor : 4
    Persistence : Superblock is persistent

    Update Time : Mon Jul 18 10:43:13 2005
          State : dirty, no-errors
 Active Devices : 4
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
       0      65       17        0      active sync   /dev/sdr1
       1       8      161        1      active sync   /dev/sdk1
       2       8      177        2      active sync   /dev/sdl1
       3       8      193        3      active sync   /dev/sdm1
       4      65       33        4        /dev/sds1
           UUID : 24989439:d4f0d908:67494e4a:97b3ba91
[root@hal root]#

I'd really like to get this md device to rebuilt onto sds and get rid of 
sdr, but I'm not sure what to do next.  I guess I could do something drastic 
like remove the device from the SCSI chain by echoing the 
"remove-single-device" command to /proc/scsi - I imagine that would trigger 
a rebuild onto the spare drive sds, but I'm nervous to try that on a 
production device until I ask the experts here :)

Thanks
Mark

----- Original Message ----- 
From: "Mike Tran" <mhtran@us.ibm.com>
To: <linux-raid@vger.kernel.org>
Sent: Monday, August 08, 2005 4:29 PM
Subject: Re: How to force a spare drive to take over in a RAID5?


> You're on the right track :) mdadm --fail /dev/sdr should do.
>
> --
> Mike T.
>
> On Mon, 2005-08-08 at 16:43, Mark Cuss wrote:
>> Kernel 2.4.27
>>
>> I've been using the old raidtools stuff and am new to mdadm - sorry if 
>> this
>> is a obvisouly simply question...
>>
>> After perusing the man page, is looks to me that I should use mdadm to 
>> mark
>> the drive I want to remove as failed to force a rebuild on the spare 
>> drive.
>> I want to double check that this is correct first though, as this md 
>> device
>> contains 300 gig of production data.
>>
>> Thanks for the help!
>>
>> Mark
>>
>> ----- Original Message ----- 
>> From: "David Greaves" <david@dgreaves.com>
>> To: "Mark Cuss" <mcuss@cdlsystems.com>
>> Cc: <linux-raid@vger.kernel.org>
>> Sent: Sunday, August 07, 2005 5:29 AM
>> Subject: Re: How to force a spare drive to take over in a RAID5?
>>
>>
>> > Mark Cuss wrote:
>> >
>> >> Hi!
>> >>
>> >> I have a 4 drive SW RAID5 running on my machine.  One of the drives is
>> >> upset for some reason - I'm not sure if the drive itself is bad, but
>> >> that's not too important right now.  The important thing is to get the
>> >> RAID5 to stop using this drive and start using a spare drive that I
>> >> just added.
>> >
>> >
>> >> I did a raidhotadd to add in a new drive, sds.  Now, I would like the
>> >> array to stop using sdr and reconstruct all of the parity tables on
>> >> sds so I can pull sdr and get it replaced or whatever...
>> >>
>> >> Any ideas?
>> >
>> > Install and read the manpage for mdadm
>> >
>> > What kernel version?
>> >
>> > David
>> >
>>
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: How to force a spare drive to take over in a RAID5?

[Neil Brown]

On Tuesday August 9, mcuss@cdlsystems.com wrote:
> Hmm - Okay.  If I do that, I get "/dev/sdr1 does not appear to be an md 
> device".  If I add --force with this it doesn't seem to matter.
> 
...
> From: "Mike Tran" <mhtran@us.ibm.com>
> To: <linux-raid@vger.kernel.org>
> Sent: Monday, August 08, 2005 4:29 PM
> Subject: Re: How to force a spare drive to take over in a RAID5?
> 
> 
> > You're on the right track :) mdadm --fail /dev/sdr should do.

Ofcourse, What Mike meant was 
  mdadm /dev/md4 --fail /dev/sdr

NeilBrown

Re: How to force a spare drive to take over in a RAID5?

[Mark Cuss]

Oh jeez how obvious - I guess I should've RTFM'd with a little more 
attention to detail - sorry 'bout the extremely obvious question!

Mark

----- Original Message ----- 
From: "Neil Brown" <neilb@cse.unsw.edu.au>
To: "Mark Cuss" <mcuss@cdlsystems.com>
Cc: <linux-raid@vger.kernel.org>
Sent: Thursday, August 11, 2005 4:11 AM
Subject: Re: How to force a spare drive to take over in a RAID5?


> On Tuesday August 9, mcuss@cdlsystems.com wrote:
>> Hmm - Okay.  If I do that, I get "/dev/sdr1 does not appear to be an md
>> device".  If I add --force with this it doesn't seem to matter.
>>
> ...
>> From: "Mike Tran" <mhtran@us.ibm.com>
>> To: <linux-raid@vger.kernel.org>
>> Sent: Monday, August 08, 2005 4:29 PM
>> Subject: Re: How to force a spare drive to take over in a RAID5?
>>
>>
>> > You're on the right track :) mdadm --fail /dev/sdr should do.
>
> Ofcourse, What Mike meant was
>  mdadm /dev/md4 --fail /dev/sdr
>
> NeilBrown
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>