From: "Jakub Husák" <jakub@gooseman.cz>
To: linux-raid@vger.kernel.org
Subject: bug: 4-disk md raid10 far2 can be assembled clean with only two disks, causing "silent" data corruption
Date: Mon, 24 Sep 2012 10:46:38 +0200 [thread overview]
Message-ID: <50601DEE.3030200@gooseman.cz> (raw)
In-Reply-To: <50601CED.1050607@gooseman.cz>
Hi, I have found a serious bug, that affects at least 4-disk md raid10
with far2 layout. The kernel allows it to run with two failed drives
silently without failing the whole array, despite it's not possible for
it to work correctly because of chunks distribution with far2 layout.
The worst thing about it is that the write IO errors are invisible for
the file system and running processes, the written data are just lost,
only with IO errors reported in dmesg. Even force-reassembling ends up
with clean,degraded array, with TWO disks, ignoring that I've tried to
assemble it with all four devices. Recreating the array with
--assume-clean is the only way to put it together.
System:
Ubuntu 12.04
Linux version 3.2.0-30-generic (buildd@batsu) (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5) ) #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC 2012
mdadm - v3.2.5 - 18th May 2012
and
Debian 6.0
Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-35) (dannf@debian.org)
(gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Tue Jun 14 12:46:30 UTC 2011
mdadm - v3.1.4 - 31st August 2010
and
Centos 6.3
How to repeat:
dd if=/dev/zero of=d0 bs=1M count=100
dd if=/dev/zero of=d1 bs=1M count=100
dd if=/dev/zero of=d2 bs=1M count=100
dd if=/dev/zero of=d3 bs=1M count=100
losetup -f d0
losetup -f d1
losetup -f d2
losetup -f d3
mdadm -C /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/loop[0-3]
dd if=/dev/zero of=/dev/md0 bs=512K count=10
10+0 records in
10+0 records out
5242880 bytes (5,2 MB) copied, 0,0409824 s, 128 MB/s
OK
mdadm /dev/md0 --fail /dev/loop0
mdadm /dev/md0 --fail /dev/loop3
mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon Sep 24 08:47:10 2012
Raid Level : raid10
Array Size : 202752 (198.03 MiB 207.62 MB)
Used Dev Size : 101376 (99.02 MiB 103.81 MB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Mon Sep 24 08:48:55 2012
State : clean, degraded <<< !!!!!!!!
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Layout : far=2
Chunk Size : 512K
Name : koubas-desktop:0 (local to host koubas-desktop)
UUID : 3ea4ded7:c10b1778:dc9f92aa:6e7cb196
Events : 21
Number Major Minor RaidDevice State
0 0 0 0 removed <<< !!!!!!!!
1 7 1 1 active sync /dev/loop1
2 7 2 2 active sync /dev/loop2
3 0 0 3 removed <<< !!!!!!!!
0 7 0 - faulty spare /dev/loop0
3 7 3 - faulty spare /dev/loop3
dd if=/dev/zero of=/dev/md0 bs=512K count=10
10+0 records in
10+0 records out
5242880 bytes (5,2 MB) copied, 0,0245752 s, 213 MB/s
echo $?
0 <<< !!!!!!!
dmesg:
[883011.442366] md/raid10:md0: Disk failure on loop0, disabling device.
[883011.442367] md/raid10:md0: Operation continuing on 3 devices.
[883011.473292] RAID10 conf printout:
[883011.473296] --- wd:3 rd:4
[883011.473299] disk 0, wo:1, o:0, dev:loop0
[883011.473301] disk 1, wo:0, o:1, dev:loop1
[883011.473302] disk 2, wo:0, o:1, dev:loop2
[883011.473304] disk 3, wo:0, o:1, dev:loop3
[883011.492046] RAID10 conf printout:
[883011.492051] --- wd:3 rd:4
[883011.492054] disk 1, wo:0, o:1, dev:loop1
[883011.492056] disk 2, wo:0, o:1, dev:loop2
[883011.492058] disk 3, wo:0, o:1, dev:loop3
[883015.875089] md/raid10:md0: Disk failure on loop3, disabling device.
[883015.875090] md/raid10:md0: Operation continuing on 2 devices. <<< !!!!!
[883015.886686] RAID10 conf printout:
[883015.886692] --- wd:2 rd:4
[883015.886695] disk 1, wo:0, o:1, dev:loop1
[883015.886697] disk 2, wo:0, o:1, dev:loop2
[883015.886699] disk 3, wo:1, o:0, dev:loop3
[883015.900018] RAID10 conf printout:
[883015.900023] --- wd:2 rd:4
[883015.900025] disk 1, wo:0, o:1, dev:loop1
[883015.900027] disk 2, wo:0, o:1, dev:loop2
************* "successful" dd follows: *******************
[883015.903622] quiet_error: 6 callbacks suppressed
[883015.903624] Buffer I/O error on device md0, logical block 50672
[883015.903628] Buffer I/O error on device md0, logical block 50672
[883015.903635] Buffer I/O error on device md0, logical block 50686
[883015.903638] Buffer I/O error on device md0, logical block 50686
[883015.903669] Buffer I/O error on device md0, logical block 50687
[883015.903672] Buffer I/O error on device md0, logical block 50687
[883015.903706] Buffer I/O error on device md0, logical block 50687
[883015.903710] Buffer I/O error on device md0, logical block 50687
[883015.903714] Buffer I/O error on device md0, logical block 50687
[883015.903717] Buffer I/O error on device md0, logical block 50687
[883052.136435] quiet_error: 6 callbacks suppressed
[883052.136439] Buffer I/O error on device md0, logical block 384
[883052.136442] lost page write due to I/O error on md0
[883052.136448] Buffer I/O error on device md0, logical block 385
[883052.136450] lost page write due to I/O error on md0
[883052.136454] Buffer I/O error on device md0, logical block 386
[883052.136456] lost page write due to I/O error on md0
[883052.136460] Buffer I/O error on device md0, logical block 387
[883052.136462] lost page write due to I/O error on md0
[883052.136466] Buffer I/O error on device md0, logical block 388
[883052.136468] lost page write due to I/O error on md0
[883052.136472] Buffer I/O error on device md0, logical block 389
[883052.136474] lost page write due to I/O error on md0
[883052.136478] Buffer I/O error on device md0, logical block 390
[883052.136480] lost page write due to I/O error on md0
[883052.136484] Buffer I/O error on device md0, logical block 391
[883052.136486] lost page write due to I/O error on md0
[883052.136492] Buffer I/O error on device md0, logical block 392
[883052.136494] lost page write due to I/O error on md0
[883052.136498] Buffer I/O error on device md0, logical block 393
[883052.136500] lost page write due to I/O error on md0
next parent reply other threads:[~2012-09-24 8:46 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <50601CED.1050607@gooseman.cz>
2012-09-24 8:46 ` Jakub Husák [this message]
2012-09-24 13:37 bug: 4-disk md raid10 far2 can be assembled clean with only two disks, causing silent data corruption Jakub Husák
2012-09-25 4:19 ` NeilBrown
2012-09-25 5:00 ` Mikael Abrahamsson
2012-09-25 9:48 ` jakub
2012-09-25 11:14 ` keld
2012-09-25 11:47 ` John Robinson
2012-09-25 12:32 ` NeilBrown
[not found] ` <50628B39.90205@gooseman.cz>
2012-09-26 5:41 ` NeilBrown
2012-09-26 8:28 ` keld
2012-09-26 8:59 ` John Robinson
2012-09-26 9:08 ` keld
2012-09-26 9:23 ` keld
[not found] ` <5067F014.5020600@gooseman.cz>
2012-09-30 10:24 ` keld
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50601DEE.3030200@gooseman.cz \
--to=jakub@gooseman.cz \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.