data corruption after rebuild

data corruption after rebuild

[Pavel Herrmann]

Hi,

I have a big problem with mdadm, I removed a drive from my raid6, after 
replacing it the raid started an online resync. I accidentally pushed the 
computer and it shut down (power cord moved), and after booting it again the 
online resync continued.

the problem is that the rebuilt array is corrupted. most of the data is fine, 
but every several MB there is an error (which doesn't look like being caused 
by a crash), effectively invalidating all data on the drive (about 7TB, mainly 
HD video)

I do monthly scans, so the redundancy syndromes should have been up-to-date, 
the array is made of 8 disks, the setup is ext4 on lvm on mdraid

is there anything to solve this? or at least ideas what happened?

Re: data corruption after rebuild

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

Hello,

On Tue, 19 Jul 2011 15:55:35 +0200
Pavel Herrmann <morpheus.ibis@gmail.com> wrote:

> the problem is that the rebuilt array is corrupted. most of the data is fine, 
> but every several MB there is an error (which doesn't look like being caused 
> by a crash), effectively invalidating all data on the drive (about 7TB, mainly 
> HD video)

Which model of SATA controller/HBA do you use?

Kernel version, mdadm version?

Anything unusual in SMART reports of any of the drives (e.g. a nonzero UDMA CRC Error count)?

> 
> I do monthly scans, so the redundancy syndromes should have been up-to-date, 
> the array is made of 8 disks, the setup is ext4 on lvm on mdraid

Did you notice any nonzero mismatch_cnt during those scans?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: data corruption after rebuild

[Pavel Herrmann]

On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
> Hello,
> 
> On Tue, 19 Jul 2011 15:55:35 +0200
> 
> Pavel Herrmann <morpheus.ibis@gmail.com> wrote:
> > the problem is that the rebuilt array is corrupted. most of the data is
> > fine, but every several MB there is an error (which doesn't look like
> > being caused by a crash), effectively invalidating all data on the
> > drive (about 7TB, mainly HD video)
> 
> Which model of SATA controller/HBA do you use?

4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)

> Kernel version, mdadm version?

2.6.33-gentoo-r1, mdadm - v3.1.5 - 23rd March 2011

> Anything unusual in SMART reports of any of the drives (e.g. a nonzero UDMA
> CRC Error count)?

one current-pending-sector on the drive that was removed, and one on one more
> 
> > I do monthly scans, so the redundancy syndromes should have been
> > up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on
> > mdraid
> 
> Did you notice any nonzero mismatch_cnt during those scans?

where would I find this?

syslog for last scan is just:

Jul  2 08:40:01 Bloomfield kernel: [83795.157876] md: data-check of RAID array md0
Jul  2 08:40:01 Bloomfield mdadm[2613]: RebuildStarted event detected on md device /dev/md0
Jul  2 09:46:41 Bloomfield mdadm[2613]: Rebuild21 event detected on md device /dev/md0
Jul  2 10:53:21 Bloomfield mdadm[2613]: Rebuild42 event detected on md device /dev/md0
Jul  2 12:00:02 Bloomfield mdadm[2613]: Rebuild61 event detected on md device /dev/md0
Jul  2 13:40:02 Bloomfield mdadm[2613]: Rebuild82 event detected on md device /dev/md0
Jul  2 16:02:46 Bloomfield kernel: [110348.161984] md: md0: data-check done.
Jul  2 16:02:47 Bloomfield mdadm[2613]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 72

I presume the nonzero "mismatches found" is a bad thing?

just to mention, all files were fine two dayss ago (I do keep md5sums of all files to check for bit rot)

Re: data corruption after rebuild

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Tue, 19 Jul 2011 18:18:56 +0200
Pavel Herrmann <morpheus.ibis@gmail.com> wrote:

> On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
> > Hello,
> > 
> > On Tue, 19 Jul 2011 15:55:35 +0200
> > 
> > Pavel Herrmann <morpheus.ibis@gmail.com> wrote:
> > > the problem is that the rebuilt array is corrupted. most of the data is
> > > fine, but every several MB there is an error (which doesn't look like
> > > being caused by a crash), effectively invalidating all data on the
> > > drive (about 7TB, mainly HD video)
> > 
> > Which model of SATA controller/HBA do you use?
> 
> 4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)

OK, one more guess - any Samsung drives, F4EG? http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: data corruption after rebuild

[Pavel Herrmann]

On Tuesday 19 of July 2011 23:38:56 Roman Mamedov wrote:
> On Tue, 19 Jul 2011 18:18:56 +0200
> 
> Pavel Herrmann <morpheus.ibis@gmail.com> wrote:
> > On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
> > > Which model of SATA controller/HBA do you use?
> > 
> > 4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)
> 
> OK, one more guess - any Samsung drives, F4EG?
> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks

nope, all disks are WDC_WD20EARS-00MVWB0,
4 were bought about 10 months ago, other 4 about 6 months ago

Re: data corruption after rebuild

[Pavel Herrmann]

On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
> > I do monthly scans, so the redundancy syndromes should have been
> > up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on
> > mdraid
> 
> Did you notice any nonzero mismatch_cnt during those scans?

oh crap, the rebuild finished with

Jul 19 09:41:24 Bloomfield mdadm[3996]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 3184

this is really bad i presume? 

Re: data corruption after rebuild

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Tue, 19 Jul 2011 18:35:26 +0200
Pavel Herrmann <morpheus.ibis@gmail.com> wrote:

> On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
> > > I do monthly scans, so the redundancy syndromes should have been
> > > up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on
> > > mdraid
> > 
> > Did you notice any nonzero mismatch_cnt during those scans?
> 
> oh crap, the rebuild finished with
> 
> Jul 19 09:41:24 Bloomfield mdadm[3996]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 3184
> 
> this is really bad i presume? 

Well, this basically tells you what you already know - a part of the data you have was corrupted. In this case I think it's 3184 512-byte sectors, which is about 1.6MB total.
How it got there and how to prevent that from happening in the future - that's a whole different question.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: data corruption after rebuild

[Pavel Herrmann]

On Tuesday 19 of July 2011 22:48:19 Roman Mamedov wrote:
> On Tue, 19 Jul 2011 18:35:26 +0200
> Well, this basically tells you what you already know - a part of the data
> you have was corrupted. In this case I think it's 3184 512-byte sectors,
> which is about 1.6MB total.

the number seems too low to me, I have about 2000 video files larger than 1GB 
on that array, and every one i tried has been corrupted enough to create 
almost unstoppable visual artifacts in hi-def video.


> How it got there and how to prevent that from
> happening in the future - that's a whole different question.

would ZFS in raidz2 mode be much better than raid6+ext4? I understand its not 
the topic of this list, but file-level checksummed rebuild looks like a nice 
feature

Re: data corruption after rebuild

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

On Tue, 19 Jul 2011 19:05:39 +0200
Pavel Herrmann <morpheus.ibis@gmail.com> wrote:

> > How it got there and how to prevent that from
> > happening in the future - that's a whole different question.
> 
> would ZFS in raidz2 mode be much better than raid6+ext4? I understand its not 
> the topic of this list, but file-level checksummed rebuild looks like a nice 
> feature

Personally I prefer to not bother with ZFS, it brings way too many complications into software choice, and I just want to use my favorite GNU/Linux distro and not Solaris, and also not trusting 12 TB of data to a third-party kernel module or a FUSE driver which are barely tested and have uncertain future. I'd put more hope in BTRFS RAID5, but that one is a long way ahead from becoming a viable option too.

Regarding mdadm+raid6, AFAIK it currently does not try to heal itself from silent corruption inside a single chunk, even though that should be possible with RAID6. On a repair, if the data chunks are readable with no I/O error, they are considered to be the golden standard and all parity chunks are simply recalculated from data and overwritten (also incrementing mismatch_cnt, if they changed). So maybe implementing a more advanced repair feature could give protection against silent corruption not much weaker than what is offered by per-file checksumming RAID implementations.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: data corruption after rebuild

[NeilBrown]

On Tue, 19 Jul 2011 15:55:35 +0200 Pavel Herrmann <morpheus.ibis@gmail.com>
wrote:

> Hi,
> 
> I have a big problem with mdadm, I removed a drive from my raid6, after 
> replacing it the raid started an online resync. I accidentally pushed the 
> computer and it shut down (power cord moved), and after booting it again the 
> online resync continued.
> 
> the problem is that the rebuilt array is corrupted. most of the data is fine, 
> but every several MB there is an error (which doesn't look like being caused 
> by a crash), effectively invalidating all data on the drive (about 7TB, mainly 
> HD video)
> 
> I do monthly scans, so the redundancy syndromes should have been up-to-date, 
> the array is made of 8 disks, the setup is ext4 on lvm on mdraid
> 
> is there anything to solve this? or at least ideas what happened?

My suggestion would be to remove the drive you recently added and then see if
the data is still corrupted.  It may not help but is probably worth a try.

There was a bug prior to 2.6.32 where RAID6 could sometimes write the wrong
data when recovering to a spare.  It would only happen if you were accessing
that data at the same time as it was recovery it, and if you were unlucky.

However you are running a newer kernel so that shouldn't affect you, but you
never know.

BTW the monthly scans that you do are primarily for finding sleeping bad
blocks - blocks that you cannot read.  They do check for inconsistencies in
the parity but only report them, it doesn't correct them.  This is because
automatically correcting can cause more problems than it solves.

When the monthly check reported inconsistencies you "should" have confirmed
that all the drives seem to be functioning correctly and then run a 'repair'
pass to fix the parity blocks up.

As you didn't that bad parity would have created bad data when you recovered.


NeilBrown

Re: data corruption after rebuild

[Pavel Herrmann]

On Wednesday 20 of July 2011 16:24:31 you wrote:
> My suggestion would be to remove the drive you recently added and then see
> if the data is still corrupted.  It may not help but is probably worth a
> try.

tried that, did not help, probably due to the finished rebuild that "repaired" 
all the parity data

> There was a bug prior to 2.6.32 where RAID6 could sometimes write the wrong
> data when recovering to a spare.  It would only happen if you were accessing
> that data at the same time as it was recovery it, and if you were unlucky.

I was accessing some data (read-mostly), but now the largest undamaged file on 
the filesystem has just under 3MB - that looks a bit suspicious, as the stripe-
width is 6x 512K = 3M

> BTW the monthly scans that you do are primarily for finding sleeping bad
> blocks - blocks that you cannot read.  They do check for inconsistencies in
> the parity but only report them, it doesn't correct them.  This is because
> automatically correcting can cause more problems than it solves.
> 
> When the monthly check reported inconsistencies you "should" have confirmed
> that all the drives seem to be functioning correctly and then run a 'repair'
> pass to fix the parity blocks up.
> 
> As you didn't that bad parity would have created bad data when you
> recovered.

see the last part, at this point I would be perfectly OK with 72 damaged 
blocks, as per the last scan (or even a few hundred, for that matter)


PS: forgot to include maillist