Corruption during RAID5->RAID6 migration

Corruption during RAID5->RAID6 migration

[James Doebbler]

Hello,

I have encountered a scary situation with corruption on my RAID array
and would like any help/advice/pointers that might help me
save/recover any data I can.  I'll try to describe the situation as
best I can, so forgive the length of this email.

I have a personal file and media server running Ubuntu Linux Server
12.04.2, kernel version 3.2.0-41-generic.  I have a mdadm RAID5 array
of 2TB disks that I've been adding disks to and growing as needed off
the past couple of years and everything has been great other than a
non-zero mismatch_cnt.  The array was currently at 10TB/6 device and I
decided it was time to move to a RAID6 array since the number of
devices was getting large.  I wanted to minimize the chance of a total
failure during a rebuild as well as hopefully be able to resolve any
future mismatch_cnts correctly with the extra parity information.

I had read on Neil Brown's blog that the migration would be much
faster if I was also adding capacity, so I installed two new 2TB
drives, added them to the array (as spares) and started the
reshape/grow. I've appended the commands used and mdadm output to the
end of this email.

The reshape seemed to be going along as expected except I was only
getting ~5MB/s instead of the ~40MB/s I usually see.  Several hours
later I noticed that some of my recent downloads were corrupt when
extracting from archives.  I created some files from data in
/dev/urandom and calculated the md5sum.  A minute or so later I
recalculated the sum, and it was different.  Similarly, copying the
file resulted in another md5sum that was not the same as the previous
two.

At that point I am not sure where the problem is, but I know my RAID
array is no longer correctly returning the data I store to them.  I do
not have verification data for most of the data already on the drive,
so I do not know if there is a problem reading any data, or a problem
writing new data (in which case my pre-existing data might be okay).

Running iostat, I noticed that one drive was the bottleneck
(/dev/sdh).  It was one of the new drives and even though I had tested
them thoroughly, I worried that it was this drive that was returning
bad data or something.  I failed the drive in question and the RAID
reshape sped up considerably (to ~35MB/s).  However, doing the same
md5sum of new random data files with the drive non-active in the array
still failed in the same way.

I then became worried about a hardware problem with my RAM or SATA
card, although I hadn't had previous problems and found no errors in
dmesg/syslog and no UDMA CRC errors in any drive SMART data.  Since
the reshape operation reads and writes all data, I knew that the
longer I went, the more likely I was to corrupt data.  So I shut down
the server with around 45% of the reshape operation complete.  I hope
this doesn't cause future complications, but I didn't want to risk any
more data loss.

I ran Memtest86+ over the weekend for 60 passes (~65 hrs) straight
with no errors detected.

I have shut down the server and am trying to figure out what to do.
If there's not a miracle software solution, my leading idea is to boot
up, fail the other added disk, and use the original 6 disks in a
degraded array to try to get off any data I can.  This is under the
assumption that the data that hasn't been moved during the reshape
would still be good as these are the same drives connected to the same
SATA ports with the same cables that gave me no problems before.

I'm curious if anyone has ever seen this kind of behavior before and
has any recommendations on what to do next.  I believe I have backups
of 80%+ of the non-replaceable data on the array, but I'm not
completely current and I'd like to save as much data as possible.

Thanks,
James


Commands and output from the reshape:

$ sudo mdadm --add /dev/md2 /dev/sdi
mdadm: added /dev/sdi
$ sudo mdadm --add /dev/md2 /dev/sdh
mdadm: added /dev/sdh
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [linear] [multipath]
[raid0] [raid10]
md1 : active raid1 sdl2[1] sdk2[0]
      239256440 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdk1[0] sdl1[1]
      250868 blocks super 1.2 [2/2] [UU]

md2 : active raid5 sdh[9](S) sdi[8](S) sda[7] sdf[5] sdb[2] sde[6] sdc[0] sdd[3]
      9767564800 blocks super 1.2 level 5, 512k chunk, algorithm 2
[6/6] [UUUUUU]

unused devices: <none>
$ sudo mdadm --grow /dev/md2 --raid-devices=8 --level=6
--backup-file=/root/grow_md2_to_raid6.bak
mdadm: level of /dev/md2 changed to raid6
mdadm: Need to backup 15360K of critical section..
jamesd@oracle:~$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [linear] [multipath]
[raid0] [raid10]
md1 : active raid1 sdl2[1] sdk2[0]
      239256440 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdk1[0] sdl1[1]
      250868 blocks super 1.2 [2/2] [UU]

md2 : active raid6 sdh[9] sdi[8] sda[7] sdf[5] sdb[2] sde[6] sdc[0] sdd[3]
      9767564800 blocks super 1.2 level 6, 512k chunk, algorithm 18
[8/7] [UUUUUU_U]
      [>....................]  reshape =  0.0% (18432/1953512960)
finish=4234.9min speed=7680K/sec

unused devices: <none>

(later)
$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Mon Sep 12 22:07:25 2011
     Raid Level : raid6
     Array Size : 9767564800 (9315.08 GiB 10001.99 GB)
  Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

    Update Time : Thu May  9 02:11:21 2013
          State : active, degraded, reshaping
 Active Devices : 7
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric-6
     Chunk Size : 512K

 Reshape Status : 2% complete
  Delta Devices : 1, (7->8)
     New Layout : left-symmetric

           Name : oracle:2  (local to host oracle)
           UUID : ed86ce45:ba8fd59c:5c217ab5:e99eddfe
         Events : 115349

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       2       8       16        1      active sync   /dev/sdb
       3       8       48        2      active sync   /dev/sdd
       6       8       64        3      active sync   /dev/sde
       5       8       80        4      active sync   /dev/sdf
       7       8        0        5      active sync   /dev/sda
       9       8      112        6      spare rebuilding   /dev/sdh
       8       8      128        7      active sync   /dev/sdi

Re: Corruption during RAID5->RAID6 migration

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 3186 bytes --]

On Tue, 14 May 2013 14:56:22 -0500
James Doebbler <jamesdoebbler@gmail.com> wrote:

> Hello,
> 
> I have encountered a scary situation with corruption on my RAID array
> and would like any help/advice/pointers that might help me
> save/recover any data I can.  I'll try to describe the situation as
> best I can, so forgive the length of this email.
> 
> I have a personal file and media server running Ubuntu Linux Server
> 12.04.2, kernel version 3.2.0-41-generic.  I have a mdadm RAID5 array
> of 2TB disks that I've been adding disks to and growing as needed off
> the past couple of years and everything has been great other than a
> non-zero mismatch_cnt.  The array was currently at 10TB/6 device and I
> decided it was time to move to a RAID6 array since the number of
> devices was getting large.  I wanted to minimize the chance of a total
> failure during a rebuild as well as hopefully be able to resolve any
> future mismatch_cnts correctly with the extra parity information.
> 
> I had read on Neil Brown's blog that the migration would be much
> faster if I was also adding capacity, so I installed two new 2TB
> drives, added them to the array (as spares) and started the
> reshape/grow. I've appended the commands used and mdadm output to the
> end of this email.
> 
> The reshape seemed to be going along as expected except I was only
> getting ~5MB/s instead of the ~40MB/s I usually see.  Several hours
> later I noticed that some of my recent downloads were corrupt when
> extracting from archives.  I created some files from data in
> /dev/urandom and calculated the md5sum.  A minute or so later I
> recalculated the sum, and it was different.  Similarly, copying the
> file resulted in another md5sum that was not the same as the previous
> two.
> 
> At that point I am not sure where the problem is, but I know my RAID
> array is no longer correctly returning the data I store to them.  I do
> not have verification data for most of the data already on the drive,
> so I do not know if there is a problem reading any data, or a problem
> writing new data (in which case my pre-existing data might be okay).
> 
> Running iostat, I noticed that one drive was the bottleneck
> (/dev/sdh).  It was one of the new drives and even though I had tested
> them thoroughly, I worried that it was this drive that was returning
> bad data or something.  I failed the drive in question and the RAID
> reshape sped up considerably (to ~35MB/s).  However, doing the same
> md5sum of new random data files with the drive non-active in the array
> still failed in the same way.
> 
> I then became worried about a hardware problem with my RAM or SATA
> card

Which SATA card (vendor/model)?

Also describe what exactly you have connected and to where, e.g. do you also
use the onboard controller of your motherboard (vendor/model?) for how many
drives, and to which port and on which of the controllers you have added any
new drives recently.

Mentioning HDD models also won't hurt. Really, it's almost like you wrote a
long winded post but gave barely any significant details whatsoever.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Corruption during RAID5->RAID6 migration

[James Doebbler]

Roman,

I suppose that means you think it's hardware-related then.  Do you
have any suggestions on what/how I might be able to recover?

Hardware details:

The motherboard is ECS A785GM-M with 6x internal SATAII and 2x eSATAII
I have installed 2 x SYBA SD-SA2PEX-2IR (SiI3132 Chipset), each with
2x onboard SATAII

One of the addon cards I have been using for over 6 months, but not as
part of the RAID5.  The other one (identical model) was installed
several weeks ago.

I am only 70% sure about which drives are on which of the two add-on
cards, so I will need to confirm that tonight before responding.

Onboard IDE
  2 x WD2500LB (separate RAID1 array)
Onboard SATA:
  1 x HD203WI (/dev/sdd)
  5 x HD204UI (/dev/sd[abcef])
Onboard eSATA:
  1 x ST3000DM001 (/dev/sdg) - backup drive, not part of RAID array
  1 x (unused)

Addon-cards
  1 x WD20EFRX (/dev/sdh)
  1 x HD204UI (/dev/sdi)
  1 x ST2000DL003 (/dev/sdj) - not currently used, potential hot spare
  1 x DVD-RW


Nothing that looks like obvious hardware errors in dmesg.  The
following are the only unexpected entries:

[126480.924143] INFO: task jbd2/dm-5-8:22963 blocked for more than 120 seconds.
[126480.924147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[126480.924150] jbd2/dm-5-8     D ffffffff81806240     0 22963      2 0x00000000
[126480.924154]  ffff8801127fbce0 0000000000000046 ffff8801127fbc90
0000000300000001
[126480.924158]  ffff8801127fbfd8 ffff8801127fbfd8 ffff8801127fbfd8
00000000000137c0
[126480.924161]  ffff880118d31700 ffff880114be5c00 ffff8801127fbcf0
ffff8801127fbdf8
[126480.924163] Call Trace:
[126480.924172]  [<ffffffff8165cbbf>] schedule+0x3f/0x60
[126480.924177]  [<ffffffff81262d0a>]
jbd2_journal_commit_transaction+0x18a/0x1240
[126480.924181]  [<ffffffff8165ed5e>] ? _raw_spin_lock_irqsave+0x2e/0x40
[126480.924187]  [<ffffffff81078098>] ? lock_timer_base.isra.29+0x38/0x70
[126480.924190]  [<ffffffff8108bec0>] ? add_wait_queue+0x60/0x60
[126480.924193]  [<ffffffff81267a8b>] kjournald2+0xbb/0x220
[126480.924195]  [<ffffffff8108bec0>] ? add_wait_queue+0x60/0x60
[126480.924197]  [<ffffffff812679d0>] ? commit_timeout+0x10/0x10
[126480.924200]  [<ffffffff8108b41c>] kthread+0x8c/0xa0
[126480.924203]  [<ffffffff81669234>] kernel_thread_helper+0x4/0x10
[126480.924205]  [<ffffffff8108b390>] ? flush_kthread_worker+0xa0/0xa0
[126480.924207]  [<ffffffff81669230>] ? gs_change+0x13/0x13
[126480.924303] INFO: task smbd:23413 blocked for more than 120 seconds.
[126480.924304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[126480.924305] smbd            D ffffffff81806240     0 23413  23026 0x00000000
[126480.924308]  ffff88010b84dc98 0000000000000082 0000000000000060
000000000000004b
[126480.924311]  ffff88010b84dfd8 ffff88010b84dfd8 ffff88010b84dfd8
00000000000137c0
[126480.924313]  ffff880118d31700 ffff880117464500 ffff88010b84dca8
ffff8801001c4000
[126480.924316] Call Trace:
[126480.924318]  [<ffffffff8165cbbf>] schedule+0x3f/0x60
[126480.924320]  [<ffffffff81260442>] start_this_handle.isra.9+0x2b2/0x3f0
[126480.924322]  [<ffffffff8108bec0>] ? add_wait_queue+0x60/0x60
[126480.924325]  [<ffffffff8126064a>] jbd2__journal_start+0xca/0x110
[126480.924327]  [<ffffffff812606a3>] jbd2_journal_start+0x13/0x20
[126480.924329]  [<ffffffff81237c4f>] ext4_journal_start_sb+0x7f/0x1d0
[126480.924331]  [<ffffffff8121d7f6>] ? ext4_dirty_inode+0x26/0x60
[126480.924335]  [<ffffffff8118c1b0>] ? fillonedir+0xd0/0xd0
[126480.924337]  [<ffffffff812122d0>] ? ext4_dx_readdir+0x140/0x240
[126480.924339]  [<ffffffff8121d7f6>] ext4_dirty_inode+0x26/0x60
[126480.924342]  [<ffffffff811a2570>] __mark_inode_dirty+0x40/0x2a0
[126480.924344]  [<ffffffff811933a1>] touch_atime+0x121/0x160
[126480.924347]  [<ffffffff8118c1b0>] ? fillonedir+0xd0/0xd0
[126480.924349]  [<ffffffff8118c0c6>] vfs_readdir+0xc6/0xe0
[126480.924351]  [<ffffffff8118c389>] sys_getdents+0x89/0x100
[126480.924353]  [<ffffffff816670c2>] system_call_fastpath+0x16/0x1b

Thanks,
James

On Tue, May 14, 2013 at 3:08 PM, Roman Mamedov <rm@romanrm.ru> wrote:
> On Tue, 14 May 2013 14:56:22 -0500
> James Doebbler <jamesdoebbler@gmail.com> wrote:
>
>> Hello,
>>
>> I have encountered a scary situation with corruption on my RAID array
>> and would like any help/advice/pointers that might help me
>> save/recover any data I can.  I'll try to describe the situation as
>> best I can, so forgive the length of this email.
>>
>> I have a personal file and media server running Ubuntu Linux Server
>> 12.04.2, kernel version 3.2.0-41-generic.  I have a mdadm RAID5 array
>> of 2TB disks that I've been adding disks to and growing as needed off
>> the past couple of years and everything has been great other than a
>> non-zero mismatch_cnt.  The array was currently at 10TB/6 device and I
>> decided it was time to move to a RAID6 array since the number of
>> devices was getting large.  I wanted to minimize the chance of a total
>> failure during a rebuild as well as hopefully be able to resolve any
>> future mismatch_cnts correctly with the extra parity information.
>>
>> I had read on Neil Brown's blog that the migration would be much
>> faster if I was also adding capacity, so I installed two new 2TB
>> drives, added them to the array (as spares) and started the
>> reshape/grow. I've appended the commands used and mdadm output to the
>> end of this email.
>>
>> The reshape seemed to be going along as expected except I was only
>> getting ~5MB/s instead of the ~40MB/s I usually see.  Several hours
>> later I noticed that some of my recent downloads were corrupt when
>> extracting from archives.  I created some files from data in
>> /dev/urandom and calculated the md5sum.  A minute or so later I
>> recalculated the sum, and it was different.  Similarly, copying the
>> file resulted in another md5sum that was not the same as the previous
>> two.
>>
>> At that point I am not sure where the problem is, but I know my RAID
>> array is no longer correctly returning the data I store to them.  I do
>> not have verification data for most of the data already on the drive,
>> so I do not know if there is a problem reading any data, or a problem
>> writing new data (in which case my pre-existing data might be okay).
>>
>> Running iostat, I noticed that one drive was the bottleneck
>> (/dev/sdh).  It was one of the new drives and even though I had tested
>> them thoroughly, I worried that it was this drive that was returning
>> bad data or something.  I failed the drive in question and the RAID
>> reshape sped up considerably (to ~35MB/s).  However, doing the same
>> md5sum of new random data files with the drive non-active in the array
>> still failed in the same way.
>>
>> I then became worried about a hardware problem with my RAM or SATA
>> card
>
> Which SATA card (vendor/model)?
>
> Also describe what exactly you have connected and to where, e.g. do you also
> use the onboard controller of your motherboard (vendor/model?) for how many
> drives, and to which port and on which of the controllers you have added any
> new drives recently.
>
> Mentioning HDD models also won't hurt. Really, it's almost like you wrote a
> long winded post but gave barely any significant details whatsoever.
>
> --
> With respect,
> Roman

Re: Corruption during RAID5->RAID6 migration

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

On Tue, 14 May 2013 15:56:57 -0500
James Doebbler <jamesdoebbler@gmail.com> wrote:

> Roman,
> 
> I suppose that means you think it's hardware-related then.  Do you
> have any suggestions on what/how I might be able to recover?
> 
> Hardware details:
> 
> The motherboard is ECS A785GM-M with 6x internal SATAII and 2x eSATAII
> I have installed 2 x SYBA SD-SA2PEX-2IR (SiI3132 Chipset), each with
> 2x onboard SATAII

Hello,

This card and in fact the 3132 chipset is known to corrupt data when both
ports are accessed simultaneously at high speed.

> http://lime-technology.com/wiki/index.php/Hardware_Compatibility
> SYBA (PCI-E X1) 2 port SATA II (SiI3132)
> Beware!!! There are numerous providers of SiL3132-based addon cards, and a few of them (unknown how many) are known to be faulty, causing unseen (SILENT) data corruption - see
> http://lime-technology.com/forum/index.php?topic=18428.msg167122#msg167122
> http://lime-technology.com/forum/index.php?topic=21052.msg187437#msg187437

Earlier discussions on linux-raid:
http://comments.gmane.org/gmane.linux.raid/30629
http://permalink.gmane.org/gmane.linux.raid/33369
http://comments.gmane.org/gmane.linux.raid/33346

Some more reports:
http://s.lowendshare.com/4/1368565867.495.2013-05-14T211035Z-syba3132.png
http://forum.ixbt.com/topic.cgi?id=11:35147-39#1199 (Russian)

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Fwd: Corruption during RAID5->RAID6 migration

[James Doebbler]

Roman,

Thank you very much, I hadn't seen those reports.  I had searched
through the Newegg reviews for linux/ubuntu and saw good
support/reviews.  Now searching more generally I see several
corruption reports in the feedback.  Wow, that's really bad.  I'll
definitely be leaving some reviews there of my own to warn others.

I have on hand a HighPoint Rocket 640L 4-port SATAIII card (Marvell
88SE9235 chipset) which I can use.  I will do some research, but do
you know of any issues with this card/chipset?  If so, can you
recommend a reliable, inexpensive card or cards?  I have one each
PCI-Express 1x and 16x slots available and would like a total of 6 or
more SATAII or SATAIII ports.


It is interesting that I was still seeing problems after only having
one drive active across the two cards.  I'll be avoiding using even
one port at a time on these cards.

Based on my high-level understanding, the reshape shouldn't read any
data from the two new drives during the reshape, just write to them.
All read data should obviously have come from the original 6 RAID5
drives.  Hopefully this means all my original data is still valid
across the 6 original drives.

I'm hopeful that I can fail and remove both new drives, leaving a
degraded, partially-reshaped RAID6.  Then, re-add the new drives on a
reliable controller (after wiping, clearing superblocks?) and
rebuild/reshape from there.  I'd expect only the data written to the
drive after the migration started would potentially be corrupt, which
I can accept greatfully.

Any advice anyone can give on the proper steps to recover from this
situation after installing a working controller card would be
appreciated.

Thanks,
James


On Tue, May 14, 2013 at 4:13 PM, Roman Mamedov <rm@romanrm.ru> wrote:
>
> On Tue, 14 May 2013 15:56:57 -0500
> James Doebbler <jamesdoebbler@gmail.com> wrote:
>
> > Roman,
> >
> > I suppose that means you think it's hardware-related then.  Do you
> > have any suggestions on what/how I might be able to recover?
> >
> > Hardware details:
> >
> > The motherboard is ECS A785GM-M with 6x internal SATAII and 2x eSATAII
> > I have installed 2 x SYBA SD-SA2PEX-2IR (SiI3132 Chipset), each with
> > 2x onboard SATAII
>
> Hello,
>
> This card and in fact the 3132 chipset is known to corrupt data when both
> ports are accessed simultaneously at high speed.
>
> > http://lime-technology.com/wiki/index.php/Hardware_Compatibility
> > SYBA (PCI-E X1) 2 port SATA II (SiI3132)
> > Beware!!! There are numerous providers of SiL3132-based addon cards, and a few of them (unknown how many) are known to be faulty, causing unseen (SILENT) data corruption - see
> > http://lime-technology.com/forum/index.php?topic=18428.msg167122#msg167122
> > http://lime-technology.com/forum/index.php?topic=21052.msg187437#msg187437
>
> Earlier discussions on linux-raid:
> http://comments.gmane.org/gmane.linux.raid/30629
> http://permalink.gmane.org/gmane.linux.raid/33369
> http://comments.gmane.org/gmane.linux.raid/33346
>
> Some more reports:
> http://s.lowendshare.com/4/1368565867.495.2013-05-14T211035Z-syba3132.png
> http://forum.ixbt.com/topic.cgi?id=11:35147-39#1199 (Russian)
>
> --
> With respect,
> Roman