Re: Corruption during RAID5->RAID6 migration

[Roman Mamedov <rm@romanrm.ru>]

[-- Attachment #1: Type: text/plain, Size: 3186 bytes --]

On Tue, 14 May 2013 14:56:22 -0500
James Doebbler <jamesdoebbler@gmail.com> wrote:

> Hello,
> 
> I have encountered a scary situation with corruption on my RAID array
> and would like any help/advice/pointers that might help me
> save/recover any data I can.  I'll try to describe the situation as
> best I can, so forgive the length of this email.
> 
> I have a personal file and media server running Ubuntu Linux Server
> 12.04.2, kernel version 3.2.0-41-generic.  I have a mdadm RAID5 array
> of 2TB disks that I've been adding disks to and growing as needed off
> the past couple of years and everything has been great other than a
> non-zero mismatch_cnt.  The array was currently at 10TB/6 device and I
> decided it was time to move to a RAID6 array since the number of
> devices was getting large.  I wanted to minimize the chance of a total
> failure during a rebuild as well as hopefully be able to resolve any
> future mismatch_cnts correctly with the extra parity information.
> 
> I had read on Neil Brown's blog that the migration would be much
> faster if I was also adding capacity, so I installed two new 2TB
> drives, added them to the array (as spares) and started the
> reshape/grow. I've appended the commands used and mdadm output to the
> end of this email.
> 
> The reshape seemed to be going along as expected except I was only
> getting ~5MB/s instead of the ~40MB/s I usually see.  Several hours
> later I noticed that some of my recent downloads were corrupt when
> extracting from archives.  I created some files from data in
> /dev/urandom and calculated the md5sum.  A minute or so later I
> recalculated the sum, and it was different.  Similarly, copying the
> file resulted in another md5sum that was not the same as the previous
> two.
> 
> At that point I am not sure where the problem is, but I know my RAID
> array is no longer correctly returning the data I store to them.  I do
> not have verification data for most of the data already on the drive,
> so I do not know if there is a problem reading any data, or a problem
> writing new data (in which case my pre-existing data might be okay).
> 
> Running iostat, I noticed that one drive was the bottleneck
> (/dev/sdh).  It was one of the new drives and even though I had tested
> them thoroughly, I worried that it was this drive that was returning
> bad data or something.  I failed the drive in question and the RAID
> reshape sped up considerably (to ~35MB/s).  However, doing the same
> md5sum of new random data files with the drive non-active in the array
> still failed in the same way.
> 
> I then became worried about a hardware problem with my RAM or SATA
> card

Which SATA card (vendor/model)?

Also describe what exactly you have connected and to where, e.g. do you also
use the onboard controller of your motherboard (vendor/model?) for how many
drives, and to which port and on which of the controllers you have added any
new drives recently.

Mentioning HDD models also won't hurt. Really, it's almost like you wrote a
long winded post but gave barely any significant details whatsoever.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]