* Re: Reassembling RAID1 after good drive was offline [newbie]
From: Aryeh Leib Taurog @ 2015-01-02 14:02 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54A67A8C.8080803@youngman.org.uk>
On Fri, Jan 02, 2015 at 11:02, Anthonys Lists wrote:
>On 01/01/2015 20:54, Robert L Mathews wrote:
>>> >Personalities : [raid1]
>>> >md126 : active (auto-read-only) raid1 sdc2[0]
>>> > 943587136 blocks super 1.2 [2/1] [U_]
>>> >
>>> >md127 : active (auto-read-only) raid1 sdd2[1]
>>> > 943587136 blocks super 1.2 [2/1] [_U]
>> Still sdc2 and sdd2, although now in two arrays.
>>
>>
> What version is mdadm? iirc there is a known bug in older versions
> which does this. (Splits one array into two arrays.)
$ mdadm --version
mdadm - v3.2.5 - 18th May 2012
It's the mdadm packaged with Debian wheezy
^ permalink raw reply
* Re: Reassembling RAID1 after good drive was offline [newbie]
From: Aryeh Leib Taurog @ 2015-01-02 13:01 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54A5B409.8040501@tigertech.com>
On Thu, Jan 01, 2015 at 8:54 PM, Robert L Mathews wrote:
>> I recently made a RAID1 array from a couple of extra usb drives:
>> $ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2
>
> These are sdc2 and sdd2. Okay.
>
>> Personalities : [raid1] md126 : active (auto-read-only) raid1
>> sdc2[0]
>> 943587136 blocks super 1.2 [2/1] [U_]
>>
>> md127 : active (auto-read-only) raid1 sdd2[1]
>> 943587136 blocks super 1.2 [2/1] [_U]
>
> Still sdc2 and sdd2, although now in two arrays.
>
>> AFAIK both drives are healthy, but since that happened, it refuses
>> to assemble them both in the array:
>> $ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
>> mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed
>
> Now you're working on sdb2 and sdd2. Is that intentional? Did sdc2
> become sdb2 after a restart or something?
Yes. They're just two of three external usb disks. The lettering
depends on the order in which I power up the computer and the drives.
I believe I read that mdadm works even when the device names change.
It did seem to be working before this issue arose.
>> Is there any way to put the array back together without having to
>> resync?
>
> You should collect more data about what array each partition thinks
> it's a member of, etc., before you try anything else. People can
> probably help more if you report the output of these to the list:
>
> mdadm --detail /dev/md*
I had to assemble the array, but as above, it only includes one device.
While doing this I discovered one of the usb cables is flaky, which
explains why the device (sdc below) wasn't always coming on line.
$ mdadm --assemble --force /dev/md/backup /dev/sd[cd]2
mdadm: ignoring /dev/sdc2 as it reports /dev/sdd2 as failed
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ mdadm --detail /dev/md/*
/dev/md/backup:
Version : 1.2
Creation Time : Wed Dec 17 22:39:10 2014
Raid Level : raid1
Array Size : 943587136 (899.87 GiB 966.23 GB)
Used Dev Size : 943587136 (899.87 GiB 966.23 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Update Time : Tue Dec 30 08:06:51 2014
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : deb76:backup (local to host deb76)
UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
Events : 382
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 50 1 active sync /dev/sdd2
> mdadm --examine /dev/sd*
I won't bore you with sda and sdb, there's nothing raid-related
on them.
$ mdadm --examine /dev/sd[cd]{,2}
/dev/sdc:
MBR Magic : aa55
Partition[0] : 16777216 sectors at 16384 (type 83)
Partition[1] : 1887436800 sectors at 16793600 (type da)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 16777216 sectors at 16384 (type 83)
Partition[1] : 1887436800 sectors at 16793600 (type da)
/dev/sdc2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
Name : deb76:backup (local to host deb76)
Creation Time : Wed Dec 17 22:39:10 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1887174656 (899.88 GiB 966.23 GB)
Array Size : 943587136 (899.87 GiB 966.23 GB)
Used Dev Size : 1887174272 (899.87 GiB 966.23 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 0347f019:c7911229:bde3bb2e:847aeebc
Update Time : Sun Dec 28 21:20:46 2014
Checksum : 7cd44677 - correct
Events : 38
Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing)
/dev/sdd2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ee0bd35a:727132cc:b4230313:69d3cbd7
Name : deb76:backup (local to host deb76)
Creation Time : Wed Dec 17 22:39:10 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1887174656 (899.88 GiB 966.23 GB)
Array Size : 943587136 (899.87 GiB 966.23 GB)
Used Dev Size : 1887174272 (899.87 GiB 966.23 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d2c29953:34be093c:00279411:674cd53e
Update Time : Tue Dec 30 08:06:51 2014
Checksum : 2e39ee1c - correct
Events : 382
Device Role : Active device 1
Array State : .A ('A' == active, '.' == missing)
^ permalink raw reply
* Re: Reassembling RAID1 after good drive was offline [newbie]
From: Anthonys Lists @ 2015-01-02 11:01 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54A5B409.8040501@tigertech.com>
On 01/01/2015 20:54, Robert L Mathews wrote:
>> Personalities : [raid1]
>> >md126 : active (auto-read-only) raid1 sdc2[0]
>> > 943587136 blocks super 1.2 [2/1] [U_]
>> >
>> >md127 : active (auto-read-only) raid1 sdd2[1]
>> > 943587136 blocks super 1.2 [2/1] [_U]
> Still sdc2 and sdd2, although now in two arrays.
>
>
What version is mdadm? iirc there is a known bug in older versions which
does this. (Splits one array into two arrays.)
Cheers,
Wol
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: Can Jeuleers @ 2015-01-02 7:59 UTC (permalink / raw)
To: John Stoffel, sylvain.depuille; +Cc: linux-raid
In-Reply-To: <21670.6532.641717.517799@quad.stoffel.home>
On 02/01/15 05:07, John Stoffel wrote:
> My real suggestion to you is to now collect all the information you
> can on the system, including drive serial numbers, and how they map to
> each device and member of the RAID array. Sometimes when a system
> reboots, or you add new disks, things get re-arranged and you don't
> notice at first. And that leads you down the wrong path quite
> quickly.
Sylvain, a nice script for documenting device serial numbers,
partitions, RAID memberships etc can be found here:
https://github.com/pturmel/lsdrv/raw/master/lsdrv
Written by Phil Turmel; a frequent contributor to this list.
Jan
^ permalink raw reply
* RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Manibalan P @ 2015-01-02 6:38 UTC (permalink / raw)
To: Pasi Kärkkäinen; +Cc: neilb@suse.de, linux-raid
In-Reply-To: <20141231164800.GL19091@reaktio.net>
Dear Pasi,
I have add the bug in https://bugzilla.redhat.com/show_bug.cgi?id=1178080
Thanks,
Manibalan.
-----Original Message-----
From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
Sent: Wednesday, December 31, 2014 10:18 PM
To: Manibalan P
Cc: neilb@suse.de; linux-raid
Subject: Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
On Tue, Dec 30, 2014 at 11:06:47AM +0000, Manibalan P wrote:
> Dear Neil,
>
Hello,
> Few this for you kind attention,
> 1. I tried the same test with FC11 (2.6.32 kernel before MD code
> change). And the issue is not there 2. But with Centos 6.4 (2.6.32 kernel after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Also, a bug has been raise with RHEL regarding this issue. Please find the bug link "https://access.redhat.com/support/cases/#/case/01320319"
>
That support case URL can only be accessed by you and Redhat. Do you happen to have a public bugzilla link?
Thanks,
-- Pasi
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 24, 2014 12:15 PM
> To: neilb@suse.de; 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
>
> Dear Neil,
>
> Few this for you kind attention,
> 1. I tried the same tesst with FC11 (2.6 kernel before MD code change). And the issue is not there 2. But with Centos 6.4 (2.6 after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Thursday, December 18, 2014 11:38 AM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear neil,
>
> I also compiled the latest 3.18 kernel on CentOS 6.4 with GIT MD pull patches form 3.19, that also ran in to the same issue after removing a drive during resync.
>
> Dec 17 19:07:32 ITX002590129362 kernel: Linux version 3.18.0 (root@mycentos6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Dec 17 15:59:09 EST 2014 Dec 17 19:07:32 ITX002590129362 kernel: Command line: ro root=/dev/md255 rd_NO_LVM rd_NO_DM rhgb quiet md_mod.start_ro=1 nmi_watchdog=1 md_mod.start_dirty_degraded=1 ??? Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sda6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdb6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdc6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdh6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdi6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdj6> Dec 17 19:10:15 ITX002590129362 kernel: async_tx: api initialized (async) Dec 17 19:10:15 ITX002590129362 kernel: xor: measuring software checksum speed
> Dec 17 19:10:15 ITX002590129362 kernel: prefetch64-sse: 10048.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: generic_sse: 8824.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: xor: using function: prefetch64-sse (10048.000 MB/sec)
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x1 5921 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x2 6933 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x4 7476 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: using algorithm sse2x4
> (7476 MB/s) Dec 17 19:10:15 ITX002590129362 kernel: raid6: using
> ssse3x2 recovery algorithm Dec 17 19:10:15 ITX002590129362 kernel: md:
> raid6 personality registered for level 6 Dec 17 19:10:15
> ITX002590129362 kernel: md: raid5 personality registered for level 5
> Dec 17 19:10:15 ITX002590129362 kernel: md: raid4 personality
> registered for level 4 Dec 17 19:10:15 ITX002590129362 kernel:
> md/raid:md0: not clean -- starting background reconstruction Dec 17
> 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdj6 operational
> as raid disk 5 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> device sdi6 operational as raid disk 4 Dec 17 19:10:15 ITX002590129362
> kernel: md/raid:md0: device sdh6 operational as raid disk 3 Dec 17
> 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdc6 operational
> as raid disk 2 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0:
> device sdb6 operational as raid disk 1 Dec 17 19:10:15 ITX002590129362
> kernel: md/ra
> id:md0: device sda6 operational as raid disk 0 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: allocated 0kB Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: raid level 5 active with 6 out of 6 devices, algorithm 2 Dec 17 19:10:15 ITX002590129362 kernel: md0: detected capacity change from 0 to 2361059573760 Dec 17 19:10:15 ITX002590129362 kernel: md0: unknown partition table Dec 17 19:10:35 ITX002590129362 kernel: md: md0 switched to read-write mode.
> Dec 17 19:10:35 ITX002590129362 kernel: md: resync of RAID array md0 Dec 17 19:10:35 ITX002590129362 kernel: md: minimum _guaranteed_ speed: 10000 KB/sec/disk.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using maximum available idle IO bandwidth (but not more than 30000 KB/sec) for resync.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using 128k window, over a total of 461144448k.
> ???
> Started IOs using fio tool.
>
> ./fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> --blockalign=512
>
> ???
> Removed a drive form the system..
>
> Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) ..
> Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 02 69 03 70 00 00 10 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 40436592 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 51 b3 d0 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 206681040 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 3a f3 40 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 205189952 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK ??? Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb 00 00 00 08 00 Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:25 ITX002590129362 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb f0 00 00 10 00 Dec 17 19:13:25 ITX002590129362 aghswap: devpath [0:0:7:0] action [remove] devtype [scsi_disk] Dec 17 19:13:25 ITX002590129362 aghswap: MHSA: Sent event 0 0 7 0 remove scsi_disk Dec 17 19:13:25 ITX002590129362 kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x500605ba0101e305) Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Disk failure on sdh6, disabling device.
> Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Operation continuing on 5 devices.
> Dec 17 19:13:25 ITX002590129362 kernel: md: md0: resync interrupted.
> Dec 17 19:13:25 ITX002590129362 kernel: md: checkpointing resync of md0.
> ..
> Log messages after enabling debufgs on raid5.c, it is getting repeated continuously.
>
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0
> bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0 handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328 for sector 273480328, rmw=2 rcw=1 handling stripe 65238568, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081a956b00 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=65238568 for sector 65238568, rmw=2 rcw=1 handling stripe 713868672, state=0x2041 cnt=1, pd_idx=4, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x10 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081f020100 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=713868672 for sector 713868672, rmw=2 rcw=1 handling stripe 729622496, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622496 for sector 729622496, rmw=2 rcw=1 handling stripe 729622504, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622504 for sector 729622504, rmw=2 rcw=1 handling stripe 245773680, state=0x2041 cnt=1, pd_idx=0, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081cab7a00 written (null)
> check 0: state 0x10 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=245773680 for sector 245773680, rmw=2 rcw=1 handling stripe 867965560, state=0x2041 cnt=1, pd_idx=1, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff880802b2bf00 written (null)
> check 1: state 0x10 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=867965560 for sector 867965560, rmw=2 rcw=1 handling stripe 550162280, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x18 read (null) write ffff880802b08800 written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=550162280 for
> sector 550162280, rmw=2 rcw=1
>
>
> Thanks,
> Manibalan
>
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:11 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> The same Issue is reproducible in the latest upstream kernel also.
>
> Tested in "3.17.6" latest stable upstream kernel and find the same issue.
>
> [root@root ~]# modinfo raid456
> filename: /lib/modules/3.17.6/kernel/drivers/md/raid456.ko
> alias: raid6
> alias: raid5
> alias: md-level-6
> alias: md-raid6
> alias: md-personality-8
> alias: md-level-4
> alias: md-level-5
> alias: md-raid4
> alias: md-raid5
> alias: md-personality-4
> description: RAID4/5/6 (striping with parity) personality for MD
> license: GPL
> srcversion: 0EEF680023FDC7410F7989A
> depends: async_raid6_recov,async_pq,async_tx,async_memcpy,async_xor
> intree: Y
> vermagic: 3.17.6 SMP mod_unload modversions
> parm: devices_handle_discard_safely:Set to Y if all devices in each array reliably return zeroes on reads from discarded regions (bool)
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:01 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status
> resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> We are facing IO struck issue with raid5 in the following scenario.
> (please see the attachment for the complete information) In RAID5
> array, if a drive is removed while initialization and the same time if
> IO is happening to that md. Then IO is getting struck, and md_raid5
> thread is using 100 % of CPU. Also the md state showing as
> resync=PENDING
>
> Kernel : Issue found in the following kernels RHEL 6.5
> (2.6.32-431.el6.x86_64) CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
>
> Steps to Reproduce the issue:
>
> 1. Created a raid 5 md with 4 drives using the below mdadm command.
> mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6
> /dev/sdd6 /dev/sde6
>
> 2. Make the md writable
> mdadm ???readwrite /dev/md0
>
> 3. Now md will start initialization
>
> 4. Run FIO Tool, the the below said configuration /usr/bin/fio
> --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1
> --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1
> --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio
> --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000
> --blockalign=512
>
> 4. During MD initialzing, remove a drive(either using MDADM set
> faulty/remove or remove manually)
>
> 5. Now the IO will struck, and cat /proc/mdstat shows states with
> resync=PENDING
> ----------------------------------------------------------------------
> ----------------------- top - output show, md_raid5 using 100% cpu
>
> top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
> ----------------------------------------------------------------------
> -----------------------
> dmesg - show the stack trace
>
> INFO: task fio:2715 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000a 0 2715 2654 0x00000080
> ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
> ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff8140fa39>] ?
> md_wakeup_thread+0x39/0x70 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffffa0308f66>] ?
> make_request+0x306/0xc6c [raid456] [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81122283>] ?
> mempool_alloc+0x63/0x140 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0 [<ffffffff811c8196>]
> __blockdev_direct_IO_newtrunc+0x346/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2717 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000004 0 2717 2654 0x00000080
> ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
> ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2718 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000005 0 2718 2654 0x00000080
> ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
> ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2719 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000001 0 2719 2654 0x00000080
> ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
> ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2720 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000008 0 2720 2654 0x00000080
> ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
> ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2721 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2721 2654 0x00000080
> ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
> ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2722 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2722 2654 0x00000080
> ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
> ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2723 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000006 0 2723 2654 0x00000080
> ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
> ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2724 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000b 0 2724 2654 0x00000080
> ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
> ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
> INFO: task fio:2725 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000003 0 2725 2654 0x00000080
> ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
> ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ?
> md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>]
> get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ?
> default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ?
> prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>]
> make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ?
> autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ?
> mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>]
> md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ?
> bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ?
> __bio_add_page+0x110/0x230 [<ffffffff81266c50>]
> generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ?
> do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120
> [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>]
> __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>]
> blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ?
> blkdev_get_block+0x0/0x20 [<ffffffff81120552>]
> generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>]
> blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ?
> blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>]
> aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>]
> aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920
> [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>]
> system_call_fastpath+0x16/0x1b
>
> [root@root ~]# cat /proc/2690/stack
> [<ffffffff810686da>] __cond_resched+0x2a/0x40 [<ffffffffa030361c>]
> ops_run_io+0x2c/0x920 [raid456] [<ffffffffa03052cc>]
> handle_stripe+0x9cc/0x2980 [raid456] [<ffffffffa03078a4>]
> raid5d+0x624/0x850 [raid456] [<ffffffff81416f05>]
> md_thread+0x115/0x150 [<ffffffff8109aef6>] kthread+0x96/0xa0
> [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffffffffffff>]
> 0xffffffffffffffff
>
> [root@root ~]# cat /proc/2690/stat
> 2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0
> 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17
> 2 0 0 6855 0 0 [root@root ~]# cat /proc/2690/statm
> 0 0 0 0 0 0 0
> [root@root ~]# cat /proc/2690/stat
> stat statm status
> [root@root ~]# cat /proc/2690/status
> Name: md0_raid5
> State: R (running)
> Tgid: 2690
> Pid: 2690
> PPid: 2
> TracerPid: 0
> Uid: 0 0 0 0
> Gid: 0 0 0 0
> Utrace: 0
> FDSize: 64
> Groups:
> Threads: 1
> SigQ: 2/128402
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: fffffffffffffeff
> SigCgt: 0000000000000100
> CapInh: 0000000000000000
> CapPrm: ffffffffffffffff
> CapEff: fffffffffffffeff
> CapBnd: ffffffffffffffff
> Cpus_allowed: ffffff
> Cpus_allowed_list: 0-23
> Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
> Mems_allowed_list: 0-1
> voluntary_ctxt_switches: 5411612
> nonvoluntary_ctxt_switches: 257032
>
>
> Thanks,
> Manibalan.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2015-01-02 4:07 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <1476942476.18574241.1420136514556.JavaMail.zimbra@laposte.net>
sylvain> Happy New Year and received all my best wishes for 2015. i'm
sylvain> back to home.
Me too, but heading to bed soon, been a busy day here.
sylvain> Now, the failed raid is at 700km (~400 miles) :-( I should
sylvain> work only by phone and ssh to repair the Raid.
It sucks having to work at a distance like this.
sylvain> We have changed the right disk and the raid is rebuilding. I
sylvain> wille write you when it's finished. Many many thank's for
sylvain> your help and your patience.
My real suggestion to you is to now collect all the information you
can on the system, including drive serial numbers, and how they map to
each device and member of the RAID array. Sometimes when a system
reboots, or you add new disks, things get re-arranged and you don't
notice at first. And that leads you down the wrong path quite
quickly.
sylvain> I have a question to migrate Raid 5 to Raid 6 :
sylvain> This site
sylvain> http://www.ewams.net/?date=2013/05/02&view=Converting_RAID5_to_RAID6_in_mdadm
sylvain> describe the right cookbook? I don't want re-do this bad
sylvain> Story.
It looks right to me. But I would also suggest that you look through
the archives of this list, and make sure you have a newer version of
the mdadm tool before you try to do this.
I would also suggest that you add two new disks, one to be used as a
hot spare, and the other used to grow the array into a RAID6.
Then go through and make sure that all your drive timeouts are set
properly on system boot, so that if a disk hangs when trying to
re-read a bad block, it won't cause it to be kicked out of the array
too quickly. Again, check the recent archives (last three months of
2014) of the list for examples and discussions of this issue.
Good luck!
John
^ permalink raw reply
* Re: Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2015-01-02 4:01 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <1490650580.18792702.1420146376741.JavaMail.zimbra@laposte.net>
sylvain> First good news of the year : the dirty Raid is online.
Awesome news! A great start to the New Year. And a happy new year to
you all.
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> md2 : active raid5 sdc1[3] sdd1[2] sde1[4] sdb1[5]
sylvain> 2930276352 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
sylvain> bitmap: 0/8 pages [0KB], 65536KB chunk
sylvain> Many thank's for your help.
sylvain> Now, i have unmounted all filesystems and do e2fsck on all.
sylvain> With ext4, it's running very quickly, and no errors found. Is it reliable?
sylvain> But it's may be not your part? ;-)
You can force ext4 to do a more thorough check if you like with:
fsck.ext4 -f /dev/....
and it will force a check, even if everything looks to be ok.
sylvain> Thank's in advance
sylvain> Best Regards
sylvain> Sylvain Depuille (Happy).
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "sylvain depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
sylvain> Envoyé: Mercredi 31 Décembre 2014 20:11:32
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
>>>>> "sylvain" == sylvain depuille <sylvain.depuille@laposte.net> writes:
sylvain> Hi John,
sylvain> I do the summary of disk and states :
sylvain> /dev/sdb : 1TB : OK
sylvain> /dev/sdc : 1TB : ??
sylvain> /dev/sdd : 2TB : ddrescued
sylvain> /dev/sde : 3TB : Not synchronized
sylvain> Something is wrong in this list. It seems that i have remove
sylvain> a wrong disk.
sylvain> It might be that is the problem. One thing to do is to run 'smartctl'
sylvain> on all the disks and get the UUIDs/serial numbers so you can confirm
sylvain> which disk is which.
sylvain> The 'lsscsi' is also useful to help make sure you know which drives
sylvain> are mapped to which drive letter, since they can move around. But if
sylvain> you do get a chance, try what I suggested, which is to assemble with
sylvain> just three of the drives. Maybe don't even hook up /dev/sdc1 at all
sylvain> to the system.
sylvain> I should investigate with my friends tomorrow evening for us,
sylvain> afternoon for you. I will on the road tomorrow. And it is
sylvain> may be free for you?
sylvain> I won't be around tomorrow afternoon, January 1st, I've got a birthday
sylvain> party to attend. I'll try to be online at some point, but it's not
sylvain> likely.
sylvain> I wish you a good eve for this evening, and see you next year ;-).
sylvain> Happy New Year to you and your friends and family as well! Enjoy! I
sylvain> will talk with you in the new year for sure.
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>
sylvain> Envoyé: Mercredi 31 Décembre 2014 18:27:03
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> But the missing disk is the rescued disk!
sylvain> I understand that, but we're looking to get the RAID5 up and working
sylvain> again, but the event count looks wrong on that disk, so it shouldn't
sylvain> be used to re-assemble the array.
sylvain> There is going to be some data loss here, but let's minimize it and
sylvain> get the array going again.
Sylvain> And the 3TB should be rebuild, because i have change It just
Sylvain> before the issue :-(sorry iPhone in French)
sylvain> I think it's ok, since the other three disks have matching event
sylvain> counts. It's the /dev/sdc1 that's behind the rest, so it needs to be
sylvain> excluded at first.
sylvain> Try it out and let me know.
Sylvain> I haven't mailings list in copy because html :-(
Sylvain> If necessary i Will revend the mail from House!
Sylvain> Best Regards
Sylvain> Sylvain Depuillr
Sylvain> Envoyé depuis un telephone portable
>>>> Le 31 déc. 2014 à 17:08, John Stoffel <john@stoffel.org> a écrit :
>>>>
>>>>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
>>>>
>>>> It's a total pain at times. I'm still reading email with plain ascii
>>>> most of the time, I don't care for HTML email generally since it just
>>>> gets in the way.
>>>>
sylvain> Back to the issue :
>>>>
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
>>>>
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
>>>>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
>>>>
>>>> So if I look at the event counts, there's a mismatch between
>>>> /dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
>>>>
>>>> mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
>>>>
>>>> and see what happens then. With three out of the four devices, the
>>>> RAID5 array should start up, and then you can add in /dev/sdc1 to
>>>> rebuild the parity.
>>>>
>>>> John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2015-01-01 21:06 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21668.19044.575022.311461@quad.stoffel.home>
Hello,
First good news of the year : the dirty Raid is online.
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid5 sdc1[3] sdd1[2] sde1[4] sdb1[5]
2930276352 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/8 pages [0KB], 65536KB chunk
Many thank's for your help.
Now, i have unmounted all filesystems and do e2fsck on all.
With ext4, it's running very quickly, and no errors found. Is it reliable?
But it's may be not your part? ;-)
Thank's in advance
Best Regards
Sylvain Depuille (Happy).
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "sylvain depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
Envoyé: Mercredi 31 Décembre 2014 20:11:32
Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
>>>>> "sylvain" == sylvain depuille <sylvain.depuille@laposte.net> writes:
sylvain> Hi John,
sylvain> I do the summary of disk and states :
sylvain> /dev/sdb : 1TB : OK
sylvain> /dev/sdc : 1TB : ??
sylvain> /dev/sdd : 2TB : ddrescued
sylvain> /dev/sde : 3TB : Not synchronized
sylvain> Something is wrong in this list. It seems that i have remove
sylvain> a wrong disk.
It might be that is the problem. One thing to do is to run 'smartctl'
on all the disks and get the UUIDs/serial numbers so you can confirm
which disk is which.
The 'lsscsi' is also useful to help make sure you know which drives
are mapped to which drive letter, since they can move around. But if
you do get a chance, try what I suggested, which is to assemble with
just three of the drives. Maybe don't even hook up /dev/sdc1 at all
to the system.
sylvain> I should investigate with my friends tomorrow evening for us,
sylvain> afternoon for you. I will on the road tomorrow. And it is
sylvain> may be free for you?
I won't be around tomorrow afternoon, January 1st, I've got a birthday
party to attend. I'll try to be online at some point, but it's not
likely.
sylvain> I wish you a good eve for this evening, and see you next year ;-).
Happy New Year to you and your friends and family as well! Enjoy! I
will talk with you in the new year for sure.
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>
sylvain> Envoyé: Mercredi 31 Décembre 2014 18:27:03
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> But the missing disk is the rescued disk!
sylvain> I understand that, but we're looking to get the RAID5 up and working
sylvain> again, but the event count looks wrong on that disk, so it shouldn't
sylvain> be used to re-assemble the array.
sylvain> There is going to be some data loss here, but let's minimize it and
sylvain> get the array going again.
Sylvain> And the 3TB should be rebuild, because i have change It just
Sylvain> before the issue :-(sorry iPhone in French)
sylvain> I think it's ok, since the other three disks have matching event
sylvain> counts. It's the /dev/sdc1 that's behind the rest, so it needs to be
sylvain> excluded at first.
sylvain> Try it out and let me know.
Sylvain> I haven't mailings list in copy because html :-(
Sylvain> If necessary i Will revend the mail from House!
Sylvain> Best Regards
Sylvain> Sylvain Depuillr
Sylvain> Envoyé depuis un telephone portable
>>> Le 31 déc. 2014 à 17:08, John Stoffel <john@stoffel.org> a écrit :
>>>
>>>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
>>>
>>> It's a total pain at times. I'm still reading email with plain ascii
>>> most of the time, I don't care for HTML email generally since it just
>>> gets in the way.
>>>
sylvain> Back to the issue :
>>>
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
>>>
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
>>>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
>>>
>>> So if I look at the event counts, there's a mismatch between
>>> /dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
>>>
>>> mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
>>>
>>> and see what happens then. With three out of the four devices, the
>>> RAID5 array should start up, and then you can add in /dev/sdc1 to
>>> rebuild the parity.
>>>
>>> John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Reassembling RAID1 after good drive was offline [newbie]
From: Robert L Mathews @ 2015-01-01 20:54 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150101174048.GB4840@deb76.aryehleib.com>
On 1/1/15 9:40 AM, Aryeh Leib Taurog wrote:
> I recently made a RAID1 array from a couple of extra usb drives:
>
> $ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2
These are sdc2 and sdd2. Okay.
> Personalities : [raid1]
> md126 : active (auto-read-only) raid1 sdc2[0]
> 943587136 blocks super 1.2 [2/1] [U_]
>
> md127 : active (auto-read-only) raid1 sdd2[1]
> 943587136 blocks super 1.2 [2/1] [_U]
Still sdc2 and sdd2, although now in two arrays.
> AFAIK both drives are healthy, but since that happened, it refuses to
> assemble them both in the array:
>
> $ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
> mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed
Now you're working on sdb2 and sdd2. Is that intentional? Did sdc2
become sdb2 after a restart or something?
> Is there any way to put the array back together without having to
> resync?
You should collect more data about what array each partition thinks it's
a member of, etc., before you try anything else. People can probably
help more if you report the output of these to the list:
mdadm --detail /dev/md*
mdadm --examine /dev/sd*
--
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2015-01-01 18:21 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21668.19044.575022.311461@quad.stoffel.home>
Hello,
Happy New Year and received all my best wishes for 2015.
i'm back to home.
Now, the failed raid is at 700km (~400 miles) :-( I should work only by phone and ssh to repair the Raid.
We have changed the right disk and the raid is rebuilding. I wille write you when it's finished.
Many many thank's for your help and your patience.
I have a question to migrate Raid 5 to Raid 6 :
This site http://www.ewams.net/?date=2013/05/02&view=Converting_RAID5_to_RAID6_in_mdadm describe the right cookbook?
I don't want re-do this bad Story.
Thank's in advance.
Best Regards.
----------------------------------
Sylvain Depuille (Happy at this time).
sylvain.depuille@laposte.net
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "sylvain depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
Envoyé: Mercredi 31 Décembre 2014 20:11:32
Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
>>>>> "sylvain" == sylvain depuille <sylvain.depuille@laposte.net> writes:
sylvain> Hi John,
sylvain> I do the summary of disk and states :
sylvain> /dev/sdb : 1TB : OK
sylvain> /dev/sdc : 1TB : ??
sylvain> /dev/sdd : 2TB : ddrescued
sylvain> /dev/sde : 3TB : Not synchronized
sylvain> Something is wrong in this list. It seems that i have remove
sylvain> a wrong disk.
It might be that is the problem. One thing to do is to run 'smartctl'
on all the disks and get the UUIDs/serial numbers so you can confirm
which disk is which.
The 'lsscsi' is also useful to help make sure you know which drives
are mapped to which drive letter, since they can move around. But if
you do get a chance, try what I suggested, which is to assemble with
just three of the drives. Maybe don't even hook up /dev/sdc1 at all
to the system.
sylvain> I should investigate with my friends tomorrow evening for us,
sylvain> afternoon for you. I will on the road tomorrow. And it is
sylvain> may be free for you?
I won't be around tomorrow afternoon, January 1st, I've got a birthday
party to attend. I'll try to be online at some point, but it's not
likely.
sylvain> I wish you a good eve for this evening, and see you next year ;-).
Happy New Year to you and your friends and family as well! Enjoy! I
will talk with you in the new year for sure.
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>
sylvain> Envoyé: Mercredi 31 Décembre 2014 18:27:03
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> But the missing disk is the rescued disk!
sylvain> I understand that, but we're looking to get the RAID5 up and working
sylvain> again, but the event count looks wrong on that disk, so it shouldn't
sylvain> be used to re-assemble the array.
sylvain> There is going to be some data loss here, but let's minimize it and
sylvain> get the array going again.
Sylvain> And the 3TB should be rebuild, because i have change It just
Sylvain> before the issue :-(sorry iPhone in French)
sylvain> I think it's ok, since the other three disks have matching event
sylvain> counts. It's the /dev/sdc1 that's behind the rest, so it needs to be
sylvain> excluded at first.
sylvain> Try it out and let me know.
Sylvain> I haven't mailings list in copy because html :-(
Sylvain> If necessary i Will revend the mail from House!
Sylvain> Best Regards
Sylvain> Sylvain Depuillr
Sylvain> Envoyé depuis un telephone portable
>>> Le 31 déc. 2014 à 17:08, John Stoffel <john@stoffel.org> a écrit :
>>>
>>>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
>>>
>>> It's a total pain at times. I'm still reading email with plain ascii
>>> most of the time, I don't care for HTML email generally since it just
>>> gets in the way.
>>>
sylvain> Back to the issue :
>>>
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
>>>
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
>>>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
>>>
>>> So if I look at the event counts, there's a mismatch between
>>> /dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
>>>
>>> mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
>>>
>>> and see what happens then. With three out of the four devices, the
>>> RAID5 array should start up, and then you can add in /dev/sdc1 to
>>> rebuild the parity.
>>>
>>> John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] md: dm-thin-metadata: Remove unused function
From: Rickard Strandqvist @ 2015-01-01 17:52 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer
Cc: Rickard Strandqvist, dm-devel, Neil Brown, linux-raid,
linux-kernel
Remove the function dm_pool_get_data_block_size() that is not used anywhere.
This was partially found by using a static code analysis program called cppcheck.
Signed-off-by: Rickard Strandqvist <rickard_strandqvist@spectrumdigital.se>
---
drivers/md/dm-thin-metadata.c | 9 ---------
drivers/md/dm-thin-metadata.h | 2 --
2 files changed, 11 deletions(-)
diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c
index e9d33ad..c7914a5 100644
--- a/drivers/md/dm-thin-metadata.c
+++ b/drivers/md/dm-thin-metadata.c
@@ -1639,15 +1639,6 @@ int dm_pool_get_metadata_dev_size(struct dm_pool_metadata *pmd,
return r;
}
-int dm_pool_get_data_block_size(struct dm_pool_metadata *pmd, sector_t *result)
-{
- down_read(&pmd->root_lock);
- *result = pmd->data_block_size;
- up_read(&pmd->root_lock);
-
- return 0;
-}
-
int dm_pool_get_data_dev_size(struct dm_pool_metadata *pmd, dm_block_t *result)
{
int r = -EINVAL;
diff --git a/drivers/md/dm-thin-metadata.h b/drivers/md/dm-thin-metadata.h
index e3c857d..6258c14 100644
--- a/drivers/md/dm-thin-metadata.h
+++ b/drivers/md/dm-thin-metadata.h
@@ -182,8 +182,6 @@ int dm_pool_get_free_metadata_block_count(struct dm_pool_metadata *pmd,
int dm_pool_get_metadata_dev_size(struct dm_pool_metadata *pmd,
dm_block_t *result);
-int dm_pool_get_data_block_size(struct dm_pool_metadata *pmd, sector_t *result);
-
int dm_pool_get_data_dev_size(struct dm_pool_metadata *pmd, dm_block_t *result);
int dm_pool_block_is_used(struct dm_pool_metadata *pmd, dm_block_t b, bool *result);
--
1.7.10.4
^ permalink raw reply related
* Reassembling RAID1 after good drive was offline [newbie]
From: Aryeh Leib Taurog @ 2015-01-01 17:40 UTC (permalink / raw)
To: linux-raid
I recently made a RAID1 array from a couple of extra usb drives:
$ mdadm --create --metadata 1.2 --verbose /dev/md/backup --level=mirror -n2 /dev/sd[cd]2
I'm using it for my backups. I've run into a small problem though.
One of the drives for some reason hadn't come online when I tried to
assemble the array. Trying --assemble a second time after it came online
apparently wasn't the right thing to do:
$ mdadm --assemble --scan
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ mdadm --assemble --scan
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
$ cat /proc/mdstat
Personalities : [raid1]
md126 : active (auto-read-only) raid1 sdc2[0]
943587136 blocks super 1.2 [2/1] [U_]
md127 : active (auto-read-only) raid1 sdd2[1]
943587136 blocks super 1.2 [2/1] [_U]
AFAIK both drives are healthy, but since that happened, it refuses to
assemble them both in the array:
$ mdadm --assemble --force /dev/md/backup /dev/sd[db]2
mdadm: ignoring /dev/sdb2 as it reports /dev/sdd2 as failed
mdadm: /dev/md/backup has been started with 1 drive (out of 2).
I haven't mounted the volume, so I believe the drives are in sync.
Is there any way to put the array back together without having to
resync? It looks like I want something like --assume-clean, but that
flag isn't valid in assemble mode.
Please cc responses to me.
Thanks,
Aryeh Leib Taurog
^ permalink raw reply
* Re: Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2014-12-31 19:11 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <480970436.16926591.1420048769566.JavaMail.zimbra@laposte.net>
>>>>> "sylvain" == sylvain depuille <sylvain.depuille@laposte.net> writes:
sylvain> Hi John,
sylvain> I do the summary of disk and states :
sylvain> /dev/sdb : 1TB : OK
sylvain> /dev/sdc : 1TB : ??
sylvain> /dev/sdd : 2TB : ddrescued
sylvain> /dev/sde : 3TB : Not synchronized
sylvain> Something is wrong in this list. It seems that i have remove
sylvain> a wrong disk.
It might be that is the problem. One thing to do is to run 'smartctl'
on all the disks and get the UUIDs/serial numbers so you can confirm
which disk is which.
The 'lsscsi' is also useful to help make sure you know which drives
are mapped to which drive letter, since they can move around. But if
you do get a chance, try what I suggested, which is to assemble with
just three of the drives. Maybe don't even hook up /dev/sdc1 at all
to the system.
sylvain> I should investigate with my friends tomorrow evening for us,
sylvain> afternoon for you. I will on the road tomorrow. And it is
sylvain> may be free for you?
I won't be around tomorrow afternoon, January 1st, I've got a birthday
party to attend. I'll try to be online at some point, but it's not
likely.
sylvain> I wish you a good eve for this evening, and see you next year ;-).
Happy New Year to you and your friends and family as well! Enjoy! I
will talk with you in the new year for sure.
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>
sylvain> Envoyé: Mercredi 31 Décembre 2014 18:27:03
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> But the missing disk is the rescued disk!
sylvain> I understand that, but we're looking to get the RAID5 up and working
sylvain> again, but the event count looks wrong on that disk, so it shouldn't
sylvain> be used to re-assemble the array.
sylvain> There is going to be some data loss here, but let's minimize it and
sylvain> get the array going again.
Sylvain> And the 3TB should be rebuild, because i have change It just
Sylvain> before the issue :-(sorry iPhone in French)
sylvain> I think it's ok, since the other three disks have matching event
sylvain> counts. It's the /dev/sdc1 that's behind the rest, so it needs to be
sylvain> excluded at first.
sylvain> Try it out and let me know.
Sylvain> I haven't mailings list in copy because html :-(
Sylvain> If necessary i Will revend the mail from House!
Sylvain> Best Regards
Sylvain> Sylvain Depuillr
Sylvain> Envoyé depuis un telephone portable
>>> Le 31 déc. 2014 à 17:08, John Stoffel <john@stoffel.org> a écrit :
>>>
>>>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
>>>
>>> It's a total pain at times. I'm still reading email with plain ascii
>>> most of the time, I don't care for HTML email generally since it just
>>> gets in the way.
>>>
sylvain> Back to the issue :
>>>
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
>>>
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
>>>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
>>>
>>> So if I look at the event counts, there's a mismatch between
>>> /dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
>>>
>>> mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
>>>
>>> and see what happens then. With three out of the four devices, the
>>> RAID5 array should start up, and then you can add in /dev/sdc1 to
>>> rebuild the parity.
>>>
>>> John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2014-12-31 17:59 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21668.12775.212408.482021@quad.stoffel.home>
Hi John,
I do the summary of disk and states :
/dev/sdb : 1TB : OK
/dev/sdc : 1TB : ??
/dev/sdd : 2TB : ddrescued
/dev/sde : 3TB : Not synchronized
Something is wrong in this list.
It seems that i have remove a wrong disk.
I should investigate with my friends tomorrow evening for us, afternoon for you.
I will on the road tomorrow.
And it is may be free for you?
I wish you a good eve for this evening, and see you next year ;-).
Best Regards
Sylvain Depuille
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>
Envoyé: Mercredi 31 Décembre 2014 18:27:03
Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> But the missing disk is the rescued disk!
I understand that, but we're looking to get the RAID5 up and working
again, but the event count looks wrong on that disk, so it shouldn't
be used to re-assemble the array.
There is going to be some data loss here, but let's minimize it and
get the array going again.
Sylvain> And the 3TB should be rebuild, because i have change It just
Sylvain> before the issue :-(sorry iPhone in French)
I think it's ok, since the other three disks have matching event
counts. It's the /dev/sdc1 that's behind the rest, so it needs to be
excluded at first.
Try it out and let me know.
Sylvain> I haven't mailings list in copy because html :-(
Sylvain> If necessary i Will revend the mail from House!
Sylvain> Best Regards
Sylvain> Sylvain Depuillr
Sylvain> Envoyé depuis un telephone portable
>> Le 31 déc. 2014 à 17:08, John Stoffel <john@stoffel.org> a écrit :
>>
>>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
>>
>> It's a total pain at times. I'm still reading email with plain ascii
>> most of the time, I don't care for HTML email generally since it just
>> gets in the way.
>>
sylvain> Back to the issue :
>>
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
>>
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
>>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
>>
>> So if I look at the event counts, there's a mismatch between
>> /dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
>>
>> mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
>>
>> and see what happens then. With three out of the four devices, the
>> RAID5 array should start up, and then you can add in /dev/sdc1 to
>> rebuild the parity.
>>
>> John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Pasi Kärkkäinen @ 2014-12-31 16:48 UTC (permalink / raw)
To: Manibalan P; +Cc: neilb@suse.de, linux-raid
In-Reply-To: <CD8664C5675EDF49A5E76D7DB099D7B32611BB56@VENUS1.in.megatrends.com>
On Tue, Dec 30, 2014 at 11:06:47AM +0000, Manibalan P wrote:
> Dear Neil,
>
Hello,
> Few this for you kind attention,
> 1. I tried the same test with FC11 (2.6.32 kernel before MD code change). And the issue is not there
> 2. But with Centos 6.4 (2.6.32 kernel after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Also, a bug has been raise with RHEL regarding this issue. Please find the bug link "https://access.redhat.com/support/cases/#/case/01320319"
>
That support case URL can only be accessed by you and Redhat. Do you happen to have a public bugzilla link?
Thanks,
-- Pasi
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 24, 2014 12:15 PM
> To: neilb@suse.de; 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
>
> Dear Neil,
>
> Few this for you kind attention,
> 1. I tried the same tesst with FC11 (2.6 kernel before MD code change). And the issue is not there 2. But with Centos 6.4 (2.6 after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Thursday, December 18, 2014 11:38 AM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> Dear neil,
>
> I also compiled the latest 3.18 kernel on CentOS 6.4 with GIT MD pull patches form 3.19, that also ran in to the same issue after removing a drive during resync.
>
> Dec 17 19:07:32 ITX002590129362 kernel: Linux version 3.18.0 (root@mycentos6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Dec 17 15:59:09 EST 2014 Dec 17 19:07:32 ITX002590129362 kernel: Command line: ro root=/dev/md255 rd_NO_LVM rd_NO_DM rhgb quiet md_mod.start_ro=1 nmi_watchdog=1 md_mod.start_dirty_degraded=1 ??? Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sda6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdb6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdc6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdh6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdi6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdj6> Dec 17 19:10:15 ITX002590129362 kernel: async_tx: api initialized (async) Dec 17 19:10:15 ITX002590129362 kernel: xor: measuring
software checksum speed
> Dec 17 19:10:15 ITX002590129362 kernel: prefetch64-sse: 10048.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: generic_sse: 8824.000 MB/sec
> Dec 17 19:10:15 ITX002590129362 kernel: xor: using function: prefetch64-sse (10048.000 MB/sec)
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x1 5921 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x2 6933 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x4 7476 MB/s
> Dec 17 19:10:15 ITX002590129362 kernel: raid6: using algorithm sse2x4 (7476 MB/s) Dec 17 19:10:15 ITX002590129362 kernel: raid6: using ssse3x2 recovery algorithm Dec 17 19:10:15 ITX002590129362 kernel: md: raid6 personality registered for level 6 Dec 17 19:10:15 ITX002590129362 kernel: md: raid5 personality registered for level 5 Dec 17 19:10:15 ITX002590129362 kernel: md: raid4 personality registered for level 4 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: not clean -- starting background reconstruction Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdj6 operational as raid disk 5 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdi6 operational as raid disk 4 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdh6 operational as raid disk 3 Dec
17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdc6 operational as raid disk 2 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdb6 operational as raid disk 1 Dec 17 19:10:15 ITX002590129362 kernel: md/ra
> id:md0: device sda6 operational as raid disk 0 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: allocated 0kB Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: raid level 5 active with 6 out of 6 devices, algorithm 2 Dec 17 19:10:15 ITX002590129362 kernel: md0: detected capacity change from 0 to 2361059573760 Dec 17 19:10:15 ITX002590129362 kernel: md0: unknown partition table Dec 17 19:10:35 ITX002590129362 kernel: md: md0 switched to read-write mode.
> Dec 17 19:10:35 ITX002590129362 kernel: md: resync of RAID array md0 Dec 17 19:10:35 ITX002590129362 kernel: md: minimum _guaranteed_ speed: 10000 KB/sec/disk.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using maximum available idle IO bandwidth (but not more than 30000 KB/sec) for resync.
> Dec 17 19:10:35 ITX002590129362 kernel: md: using 128k window, over a total of 461144448k.
> ???
> Started IOs using fio tool.
>
> ./fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
>
> ???
> Removed a drive form the system..
>
> Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) ..
> Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 02 69 03 70 00 00 10 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 40436592 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 51 b3 d0 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 206681040 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 3a f3 40 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 205189952 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK ??? Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb 00 00 00 08 00 Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:25 ITX002590129362 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
> Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb f0 00 00 10 00 Dec 17 19:13:25 ITX002590129362 aghswap: devpath [0:0:7:0] action [remove] devtype [scsi_disk] Dec 17 19:13:25 ITX002590129362 aghswap: MHSA: Sent event 0 0 7 0 remove scsi_disk Dec 17 19:13:25 ITX002590129362 kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x500605ba0101e305) Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Disk failure on sdh6, disabling device.
> Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Operation continuing on 5 devices.
> Dec 17 19:13:25 ITX002590129362 kernel: md: md0: resync interrupted.
> Dec 17 19:13:25 ITX002590129362 kernel: md: checkpointing resync of md0.
> ..
> Log messages after enabling debufgs on raid5.c, it is getting repeated continuously.
>
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
> __get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0 handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328 for sector 273480328, rmw=2 rcw=1 handling stripe 65238568, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081a956b00 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=65238568 for sector 65238568, rmw=2 rcw=1 handling stripe 713868672, state=0x2041 cnt=1, pd_idx=4, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x10 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff88081f020100 written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=713868672 for sector 713868672, rmw=2 rcw=1 handling stripe 729622496, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622496 for sector 729622496, rmw=2 rcw=1 handling stripe 729622504, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622504 for sector 729622504, rmw=2 rcw=1 handling stripe 245773680, state=0x2041 cnt=1, pd_idx=0, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x18 read (null) write ffff88081cab7a00 written (null)
> check 0: state 0x10 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=245773680 for sector 245773680, rmw=2 rcw=1 handling stripe 867965560, state=0x2041 cnt=1, pd_idx=1, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x18 read (null) write ffff880802b2bf00 written (null)
> check 1: state 0x10 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=867965560 for sector 867965560, rmw=2 rcw=1 handling stripe 550162280, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
> check 5: state 0x11 read (null) write (null) written (null)
> check 4: state 0x18 read (null) write ffff880802b08800 written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x10 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x11 read (null) write (null) written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=550162280 for sector 550162280, rmw=2 rcw=1
>
>
> Thanks,
> Manibalan
>
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:11 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> The same Issue is reproducible in the latest upstream kernel also.
>
> Tested in "3.17.6" latest stable upstream kernel and find the same issue.
>
> [root@root ~]# modinfo raid456
> filename: /lib/modules/3.17.6/kernel/drivers/md/raid456.ko
> alias: raid6
> alias: raid5
> alias: md-level-6
> alias: md-raid6
> alias: md-personality-8
> alias: md-level-4
> alias: md-level-5
> alias: md-raid4
> alias: md-raid5
> alias: md-personality-4
> description: RAID4/5/6 (striping with parity) personality for MD
> license: GPL
> srcversion: 0EEF680023FDC7410F7989A
> depends: async_raid6_recov,async_pq,async_tx,async_memcpy,async_xor
> intree: Y
> vermagic: 3.17.6 SMP mod_unload modversions
> parm: devices_handle_discard_safely:Set to Y if all devices in each array reliably return zeroes on reads from discarded regions (bool)
>
> Thanks,
> Manibalan.
>
> -----Original Message-----
> From: Manibalan P
> Sent: Wednesday, December 17, 2014 12:01 PM
> To: 'linux-raid'
> Cc: 'NeilBrown'
> Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
>
> Dear Neil,
>
> We are facing IO struck issue with raid5 in the following scenario. (please see the attachment for the complete information) In RAID5 array, if a drive is removed while initialization and the same time if IO is happening to that md. Then IO is getting struck, and md_raid5 thread is using 100 % of CPU. Also the md state showing as resync=PENDING
>
> Kernel : Issue found in the following kernels RHEL 6.5 (2.6.32-431.el6.x86_64) CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
>
> Steps to Reproduce the issue:
>
> 1. Created a raid 5 md with 4 drives using the below mdadm command.
> mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6
>
> 2. Make the md writable
> mdadm ???readwrite /dev/md0
>
> 3. Now md will start initialization
>
> 4. Run FIO Tool, the the below said configuration /usr/bin/fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
>
> 4. During MD initialzing, remove a drive(either using MDADM set faulty/remove or remove manually)
>
> 5. Now the IO will struck, and cat /proc/mdstat shows states with resync=PENDING
> ---------------------------------------------------------------------------------------------
> top - output show, md_raid5 using 100% cpu
>
> top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
> ---------------------------------------------------------------------------------------------
> dmesg - show the stack trace
>
> INFO: task fio:2715 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000a 0 2715 2654 0x00000080
> ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
> ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8140fa39>] ? md_wakeup_thread+0x39/0x70 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffffa0308f66>] ? make_request+0x306/0xc6c [raid456] [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81122283>] ? mempool_alloc+0x63/0x140 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0 [<ffffffff811c8196>] __blockdev_direct_IO_newtrunc+0x346/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2717 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000004 0 2717 2654 0x00000080
> ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
> ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2718 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000005 0 2718 2654 0x00000080
> ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
> ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2719 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000001 0 2719 2654 0x00000080
> ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
> ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2720 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000008 0 2720 2654 0x00000080
> ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
> ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2721 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2721 2654 0x00000080
> ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
> ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2722 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000000 0 2722 2654 0x00000080
> ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
> ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2723 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000006 0 2723 2654 0x00000080
> ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
> ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2724 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 000000000000000b 0 2724 2654 0x00000080
> ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
> ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> INFO: task fio:2725 blocked for more than 120 seconds.
> Not tainted 2.6.32-431.el6.x86_64 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> fio D 0000000000000003 0 2725 2654 0x00000080
> ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
> ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
> ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8 Call Trace:
> [<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
> [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
> [<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
> [<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>
> [root@root ~]# cat /proc/2690/stack
> [<ffffffff810686da>] __cond_resched+0x2a/0x40 [<ffffffffa030361c>] ops_run_io+0x2c/0x920 [raid456] [<ffffffffa03052cc>] handle_stripe+0x9cc/0x2980 [raid456] [<ffffffffa03078a4>] raid5d+0x624/0x850 [raid456] [<ffffffff81416f05>] md_thread+0x115/0x150 [<ffffffff8109aef6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffffffffffff>] 0xffffffffffffffff
>
> [root@root ~]# cat /proc/2690/stat
> 2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17 2 0 0 6855 0 0 [root@root ~]# cat /proc/2690/statm
> 0 0 0 0 0 0 0
> [root@root ~]# cat /proc/2690/stat
> stat statm status
> [root@root ~]# cat /proc/2690/status
> Name: md0_raid5
> State: R (running)
> Tgid: 2690
> Pid: 2690
> PPid: 2
> TracerPid: 0
> Uid: 0 0 0 0
> Gid: 0 0 0 0
> Utrace: 0
> FDSize: 64
> Groups:
> Threads: 1
> SigQ: 2/128402
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: fffffffffffffeff
> SigCgt: 0000000000000100
> CapInh: 0000000000000000
> CapPrm: ffffffffffffffff
> CapEff: fffffffffffffeff
> CapBnd: ffffffffffffffff
> Cpus_allowed: ffffff
> Cpus_allowed_list: 0-23
> Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
> Mems_allowed_list: 0-1
> voluntary_ctxt_switches: 5411612
> nonvoluntary_ctxt_switches: 257032
>
>
> Thanks,
> Manibalan.
^ permalink raw reply
* Re: Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2014-12-31 16:08 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <2050225023.16668598.1420037480502.JavaMail.zimbra@laposte.net>
sylvain> It's my fault too, because the mailing list doesn't accept
sylvain> HTML format. And my provider doesn't offer plain text option
sylvain> with the mobile site :-(
It's a total pain at times. I'm still reading email with plain ascii
most of the time, I don't care for HTML email generally since it just
gets in the way.
sylvain> Back to the issue :
sylvain> I have stop the raid.
sylvain> mdadm --stop /dev/md2
sylvain> mdadm: stopped /dev/md2
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> unused devices: <none>
sylvain> And when i try to re-assemble, the command is aborting :
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
So if I look at the event counts, there's a mismatch between
/dev/sd[bde]1 and /dev/sdc1. So let's try to assemble without /dev/sdc1:
mdadm -vv --assemble --force /dev/md2 /dev/sdb1 missing /dev/sdd1 /dev/sde1
and see what happens then. With three out of the four devices, the
RAID5 array should start up, and then you can add in /dev/sdc1 to
rebuild the parity.
John
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2014-12-31 14:51 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21668.1236.130012.995660@quad.stoffel.home>
Hello,
It's my fault too, because the mailing list doesn't accept HTML format.
And my provider doesn't offer plain text option with the mobile site :-(
Back to the issue :
I have stop the raid.
mdadm --stop /dev/md2
mdadm: stopped /dev/md2
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
unused devices: <none>
And when i try to re-assemble, the command is aborting :
mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
mdadm: /dev/md2 assembled from 2 drives and 1 spare - not enough to start the array.
The command wan't re-acquire the sdc1 device :
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : inactive sde1[4](S) sdd1[2](S) sdb1[5](S)
4101001112 blocks super 1.2
unused devices: <none>
Many thank's for your help.
Best Regards
Sylvain Depuille
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "sylvain depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>
Envoyé: Mercredi 31 Décembre 2014 15:14:44
Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
sylvain> I have remove the 1TB burny disk, and replace it by the 2TB
sylvain> ddrescued of the burny disk.
Great,
sylvain> But i can't re-assemble the raid.
sylvain> mdadm --assemble --force /dev/md2 /dev/sd[bcde]1
sylvain> mdadm: /dev/sdb1 is busy - skipping
sylvain> mdadm: /dev/sdd1 is busy - skipping
sylvain> mdadm: /dev/sde1 is busy - skipping
sylvain> mdadm: Merging with already-assembled /dev/md/2
sylvain> mdadm: /dev/md/2 assembled from 2 drives and 1 spare - not enough to start the array.
I think you first need to stop the array, to make sure all the devices
aren't in use. Have you looked through the archives of this list for
previous examples?
So you should be able to do:
> mdadm --stop md2
> mdadm --assemble --force /dev/md2 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
and send the output. You should also be replying to the mailing list
as well, which I just noticed you haven't. My fault too!
sylvain> cat /proc/mdstat :
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> md2 : inactive sdd1[2](S) sde1[4](S) sdb1[5](S)
sylvain> 4101001112 blocks super 1.2
sylvain> unused devices: <none>
sylvain> The result of command mdadm -E /dev/sd[bcde]1 > mdadm-E-new.log is
sylvain> /dev/sdb1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953521072 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1960 sectors, after=3504 sectors
sylvain> State : clean
sylvain> Device UUID : 8506e09c:b87a44ed:7b4ee314:777ce89c
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:34 2014
sylvain> Bad Block Log : 512 entries available at offset 72 sectors
sylvain> Checksum : bad52d25 - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 0
sylvain> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sdc1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1968 sectors, after=386 sectors
sylvain> State : clean
sylvain> Device UUID : 44002aad:d3e17729:a93854eb:4139972e
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:22 2014
sylvain> Checksum : 6f69285d - correct
sylvain> Events : 167431
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 1
sylvain> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sdd1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1968 sectors, after=1953507504 sectors
sylvain> State : clean
sylvain> Device UUID : 44002aad:d3e17729:a93854eb:4139972e
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:22 2014
sylvain> Checksum : 6f692876 - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 1
sylvain> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sde1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x9
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 4294963199 (2048.00 GiB 2199.02 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1960 sectors, after=2341445631 sectors
sylvain> State : clean
sylvain> Device UUID : 0ebce28d:1a792d55:76a86538:12cc94dd
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:34 2014
sylvain> Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
sylvain> Checksum : 3801cfa - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : spare
sylvain> Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
sylvain> Could you help me to re-assemble safely the raid array?
sylvain> Thank's in advance
sylvain> Best Regards
sylvain> Sylvain Depuille (always in trouble).
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "Sylvain Depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>
sylvain> Envoyé: Mardi 30 Décembre 2014 22:07:04
sylvain> Objet: Re: Re: Re : Re: Big trouble during reassemble a Raid5
Sylvain> I'm front of the pc! The Pass1 of The ddrescue is Not
Sylvain> finished! Sorry for The false News!
sylvain> No problem. Let it finish before you make any other attempts to
sylvain> re-assemble the array.
Sylvain> Of the 5 pass tale the same Time, the command Will finish in 20 or 24 jours.
Sylvain> Thank's in advance
Sylvain> Best Regards
Sylvain> Envoyé depuis un telephone portable
>>> Le 29 déc. 2014 à 21:36, John Stoffel <john@stoffel.org> a écrit :
>>>
>>>
sylvain> Hi john, thanks for your answer! I have change a 1TB disk to
sylvain> growing the raid with 3TB disk. if i can re-insert the old
sylvain> 1TB disk in place of 3TB disk, only some log and history are
sylvain> corrupted. i think that is the best way to relaunch the raid
sylvain> without data loss. But i dont known how change the timestamp
sylvain> of the one raid disk. Have you a magic command to change a
sylvain> timestamp of a raid partition, and how known the timestamp of
sylvain> the other disk of the raid? After' raid relaunch, i can
sylvain> change the burn disk by a 3TB new one. To do the ddrescue, i
sylvain> have a 2TB disk spare! Its not the same geometry, is it
sylvain> possible? thanks in advance for your help
>>>
>>> Sylvain,
>>>
>>> Always glad to help here. I'm going to try and understand what you
>>> wrote and do my best to reply.
>>>
>>> Is the 1Tb disk the bad disk? And if you re-insert it and re-start
>>> the RAID5 array, you only have some minor lost files? If so, I would
>>> probably just copy all the data off the RAID5 onto the single 3Tb disk
>>> as a quick and dirty backup, then I'd use 'dd_rescue' to copy the bad
>>> 1Tb disk onto the new 2Tb disk.
>>>
>>> All you would have to do is make a partition on the 2tb disk which is
>>> the same size (or a little bigger) than the partition on the 1tb disk,
>>> then copy the partition over like this:
>>>
>>> ddrescue /dev/sd[BAD DISK LETTER HERE]1 /dev/sd[2TB disk letter]1 \
>>> /tmp/rescue.log
>>>
>>> So say the bad disk is sdc, and the good 2tb is sdf, you would do:
>>>
>>> ddrescue /dev/sdc1 /dev/sdf1 /tmp/rescue.log
>>>
>>> and let it go. Then you would assemble the array using the NEW 2tb
>>> disk. Ideally you would remove the bad 1tb disk from the system when
>>> trying to do this.
>>>
>>> But you really do need send us the output of the following commands:
>>>
>>> cat /proc/mdstat
>>> cat /proc/partitions
>>> mdadm --detail /dev/md#
>>>
>>> do the above for the RADI5 array.
>>>
>>> mdadm --examine /dev/sd#1
>>>
>>> for each disk in the RAID5 array.
>>>
>>> And we can give you better advice.
>>>
>>> Good luck!
>>>
>>>
sylvain> ---------------------------------- Sylvain Depuille
sylvain> sylvain.depuille@laposte.net ----- Mail d'origine ----- De:
sylvain> John Stoffel <john@stoffel.org> À: sylvain depuille
sylvain> <sylvain.depuille@laposte.net> Cc: linux-raid@vger.kernel.org
sylvain> Envoyé: Mon, 29 Dec 2014 19:32:04 +0100 (CET) Objet: Re: Big
sylvain> trouble during reassemble a Raid5
>>>
sylvain> Sylvain, I would recommend that you buy a replacement disk
sylvain> for the one throwing errors and then run dd_rescue to copy as
sylvain> much data from the dying disk to the replacement. Then, and
sylvain> only then, do you try to reassemble the array with the
sylvain> --force option. That disk is dying, and dying quickly. Can
sylvain> you also post the output of mdadm -E /dev/sd[bcde]1 for each
sylvain> disk, even the dying one, so we can look at the counts and
sylvain> give you some more advice. Also, the output of the mdadm
sylvain> --assemble --force /dev/md2 /dev/sd[bcde]1 would also be
sylvain> good. The more info the better. Good luck! John
>>>
sylvain> i'm sorry to ask this questions but the raid 5 with 4 disk is
sylvain> in big trouble during re-assemble. 2 disks are out of order.
sylvain> I have change a disk of the raid 5 (sde) to growing the raid.
sylvain> But a second disk (sdc) have too many bad sector during the
sylvain> re-assemble, and shutdown the re-assemble. "mdadm --assemble
sylvain> --force /dev/md2 /dev/sd[bcde]1" I have try to correct bad
sylvain> sectors with badblocks, but it's finished by no more spare
sylvain> sectors and the disk still have some bad sector. badblocks -b
sylvain> 512 -o badblocks-sdc.txt -v -n /dev/sdc 1140170000 1140169336
sylvain> 1140169400 1140169401 1140169402 1140169403 1140169404
sylvain> 1140169405 1140169406 1140169407 1140169416 1140169417
sylvain> 1140169418 1140169419 1140169420 1140169421 1140169422
sylvain> 1140169423
>>>
sylvain> For information the mdadm examine return : cat mdadm-exam.txt
sylvain> /dev/sdb: MBR Magic : aa55 Partition[0] : 1953523120 sectors
sylvain> at 2048 (type fd) /dev/sdc: MBR Magic : aa55 Partition[0] :
sylvain> 1953520002 sectors at 63 (type fd) /dev/sdd: MBR Magic : aa55
sylvain> Partition[0] : 1953520002 sectors at 63 (type fd) /dev/sde:
sylvain> MBR Magic : aa55 Partition[0] : 4294965247 sectors at 2048
sylvain> (type fd) I have 2 way to solve the issue. The first, is to
sylvain> have special command to pass bad sector during re-assemble as
sylvain> "mdadm --assemble --force /dev/md2 /dev/sd[bcde]1" The second
sylvain> is change the disk sde with the old good one, but some datas
sylvain> have been changed on the raid since i have remove it. But
sylvain> these datas are not important. It's only logs and history
sylvain> activity. What can i do to recover a maximum datas without
sylvain> too many risk? Thank's in advance Best Regards
sylvain> ---------------------------------- Sylvain Depuille (in
sylvain> trouble) sylvain.depuille@laposte.net -- To unsubscribe from
sylvain> this list: send the line "unsubscribe linux-raid" in the body
sylvain> of a message to majordomo@vger.kernel.org More majordomo info
sylvain> at http://vger.kernel.org/majordomo-info.html
>>>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2014-12-30 21:06 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <1916328338.14439065.1419932231303.JavaMail.zimbra@laposte.net>
>>>>> "sylvain" == sylvain depuille <sylvain.depuille@laposte.net> writes:
sylvain> Hi John,
sylvain> I'm sorry! I'm French and my English is poor!
No problems, your english is much better than I could ever do in
French!
sylvain> Yesterday, i have try to you explain what is the history of
sylvain> the issue. The Raid was fully operational before i change
sylvain> the disk sde by a 3TB disk to grow the Raid, by changing all
sylvain> disks of the raid. But when i have do the re-assemble the
sylvain> disk sdc goes burny :-(
sylvain> I have keep the old sde 1TB disk in state.
sylvain> The result of your asked command :
sylvain> cat /proc/mdstat
sylvain> Personalities : [raid1] [raid6] [raid5] [raid4]
sylvain> md2 : inactive sdc1[2](S) sdd1[3](S) sde1[4](S) sdb1[5](S)
sylvain> 5077760089 blocks super 1.2
Ok, this looks like the raid won't assemble, which is probably fine
for now.
sylvain> major minor #blocks name
sylvain> 8 0 125034840 sda
sylvain> 8 1 131072 sda1
sylvain> 8 2 62914560 sda2
sylvain> 8 3 61865984 sda3
sylvain> 8 32 976762584 sdc
sylvain> 8 33 976760001 sdc1
sylvain> 8 48 976762584 sdd
sylvain> 8 49 976760001 sdd1
sylvain> 8 16 976762584 sdb
sylvain> 8 17 976761560 sdb1
sylvain> 8 64 2930266584 sde
sylvain> 8 65 2147482623 sde1
Did the reshape or replacement of /dev/sde 1tb work fine to the 3tb
disk? It's not clear to me, and I want to be safe. But I think you
should be able to do the --assemble --force command once the dd_rescue
has finished from the bad disk onto the new disk.
Then I would also seriously think about moving to RAID6 for your data
as well. It's cheap insurance when you have lots of important data
and such to keep.
sylvain> mdadm --detail /dev/md2
sylvain> /dev/md2:
sylvain> Version : 1.2
sylvain> Raid Level : raid0
sylvain> Total Devices : 4
sylvain> Persistence : Superblock is persistent
sylvain> State : inactive
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Events : 167456
sylvain> Number Major Minor RaidDevice
sylvain> - 8 17 - /dev/sdb1
sylvain> - 8 33 - /dev/sdc1
sylvain> - 8 49 - /dev/sdd1
sylvain> - 8 65 - /dev/sde1
sylvain> mdadm -E /dev/sd[bcde]1
sylvain> /dev/sdb1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953521072 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1960 sectors, after=3504 sectors
sylvain> State : clean
sylvain> Device UUID : 8506e09c:b87a44ed:7b4ee314:777ce89c
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:34 2014
sylvain> Bad Block Log : 512 entries available at offset 72 sectors
sylvain> Checksum : bad62d22 - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 0
sylvain> Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sdc1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1968 sectors, after=386 sectors
sylvain> State : clean
sylvain> Device UUID : 44002aad:d3e17729:a93854eb:4139972e
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:22 2014
sylvain> Checksum : 6f69285d - correct
sylvain> Events : 167431
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 1
sylvain> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sdd1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x1
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1968 sectors, after=386 sectors
sylvain> State : clean
sylvain> Device UUID : 5cff6f7f:ea6b89b6:28e4e8b3:7a2b5a7f
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:34 2014
sylvain> Checksum : e81a4f63 - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : Active device 2
sylvain> Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
sylvain> /dev/sde1:
sylvain> Magic : a92b4efc
sylvain> Version : 1.2
sylvain> Feature Map : 0x9
sylvain> Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
sylvain> Name : le-bohec:2 (local to host le-bohec)
sylvain> Creation Time : Tue Apr 9 17:56:19 2013
sylvain> Raid Level : raid5
sylvain> Raid Devices : 4
sylvain> Avail Dev Size : 4294963199 (2048.00 GiB 2199.02 GB)
sylvain> Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
sylvain> Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
sylvain> Data Offset : 2048 sectors
sylvain> Super Offset : 8 sectors
sylvain> Unused Space : before=1960 sectors, after=2341445631 sectors
sylvain> State : clean
sylvain> Device UUID : 0ebce28d:1a792d55:76a86538:12cc94dd
sylvain> Internal Bitmap : 8 sectors from superblock
sylvain> Update Time : Sat Dec 27 22:08:34 2014
sylvain> Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
sylvain> Checksum : 3801cfa - correct
sylvain> Events : 167456
sylvain> Layout : left-symmetric
sylvain> Chunk Size : 512K
sylvain> Device Role : spare
sylvain> Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
sylvain> Now, i have installed ddrescue on the system, and i go to my
sylvain> friends to add the 2TB disk in the tower, and launch the
sylvain> command.
sylvain> Many thank's for your help and your patience.
sylvain> Best Regards
sylvain> Sylvain Depuille
sylvain> ----- Mail original -----
sylvain> De: "John Stoffel" <john@stoffel.org>
sylvain> À: "sylvain depuille" <sylvain.depuille@laposte.net>
sylvain> Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
sylvain> Envoyé: Lundi 29 Décembre 2014 21:36:23
sylvain> Objet: Re: Re : Re: Big trouble during reassemble a Raid5
sylvain> Hi john, thanks for your answer! I have change a 1TB disk to
sylvain> growing the raid with 3TB disk. if i can re-insert the old
sylvain> 1TB disk in place of 3TB disk, only some log and history are
sylvain> corrupted. i think that is the best way to relaunch the raid
sylvain> without data loss. But i dont known how change the timestamp
sylvain> of the one raid disk. Have you a magic command to change a
sylvain> timestamp of a raid partition, and how known the timestamp of
sylvain> the other disk of the raid? After' raid relaunch, i can
sylvain> change the burn disk by a 3TB new one. To do the ddrescue, i
sylvain> have a 2TB disk spare! Its not the same geometry, is it
sylvain> possible? thanks in advance for your help
sylvain> Sylvain,
sylvain> Always glad to help here. I'm going to try and understand what you
sylvain> wrote and do my best to reply.
sylvain> Is the 1Tb disk the bad disk? And if you re-insert it and re-start
sylvain> the RAID5 array, you only have some minor lost files? If so, I would
sylvain> probably just copy all the data off the RAID5 onto the single 3Tb disk
sylvain> as a quick and dirty backup, then I'd use 'dd_rescue' to copy the bad
sylvain> 1Tb disk onto the new 2Tb disk.
sylvain> All you would have to do is make a partition on the 2tb disk which is
sylvain> the same size (or a little bigger) than the partition on the 1tb disk,
sylvain> then copy the partition over like this:
sylvain> ddrescue /dev/sd[BAD DISK LETTER HERE]1 /dev/sd[2TB disk letter]1 \
sylvain> /tmp/rescue.log
sylvain> So say the bad disk is sdc, and the good 2tb is sdf, you would do:
sylvain> ddrescue /dev/sdc1 /dev/sdf1 /tmp/rescue.log
sylvain> and let it go. Then you would assemble the array using the NEW 2tb
sylvain> disk. Ideally you would remove the bad 1tb disk from the system when
sylvain> trying to do this.
sylvain> But you really do need send us the output of the following commands:
sylvain> cat /proc/mdstat
sylvain> cat /proc/partitions
sylvain> mdadm --detail /dev/md#
sylvain> do the above for the RADI5 array.
sylvain> mdadm --examine /dev/sd#1
sylvain> for each disk in the RAID5 array.
sylvain> And we can give you better advice.
sylvain> Good luck!
sylvain> ---------------------------------- Sylvain Depuille
sylvain> sylvain.depuille@laposte.net ----- Mail d'origine ----- De:
sylvain> John Stoffel <john@stoffel.org> À: sylvain depuille
sylvain> <sylvain.depuille@laposte.net> Cc: linux-raid@vger.kernel.org
sylvain> Envoyé: Mon, 29 Dec 2014 19:32:04 +0100 (CET) Objet: Re: Big
sylvain> trouble during reassemble a Raid5
sylvain> Sylvain, I would recommend that you buy a replacement disk
sylvain> for the one throwing errors and then run dd_rescue to copy as
sylvain> much data from the dying disk to the replacement. Then, and
sylvain> only then, do you try to reassemble the array with the
sylvain> --force option. That disk is dying, and dying quickly. Can
sylvain> you also post the output of mdadm -E /dev/sd[bcde]1 for each
sylvain> disk, even the dying one, so we can look at the counts and
sylvain> give you some more advice. Also, the output of the mdadm
sylvain> --assemble --force /dev/md2 /dev/sd[bcde]1 would also be
sylvain> good. The more info the better. Good luck! John
sylvain> i'm sorry to ask this questions but the raid 5 with 4 disk is
sylvain> in big trouble during re-assemble. 2 disks are out of order.
sylvain> I have change a disk of the raid 5 (sde) to growing the raid.
sylvain> But a second disk (sdc) have too many bad sector during the
sylvain> re-assemble, and shutdown the re-assemble. "mdadm --assemble
sylvain> --force /dev/md2 /dev/sd[bcde]1" I have try to correct bad
sylvain> sectors with badblocks, but it's finished by no more spare
sylvain> sectors and the disk still have some bad sector. badblocks -b
sylvain> 512 -o badblocks-sdc.txt -v -n /dev/sdc 1140170000 1140169336
sylvain> 1140169400 1140169401 1140169402 1140169403 1140169404
sylvain> 1140169405 1140169406 1140169407 1140169416 1140169417
sylvain> 1140169418 1140169419 1140169420 1140169421 1140169422
sylvain> 1140169423
sylvain> For information the mdadm examine return : cat mdadm-exam.txt
sylvain> /dev/sdb: MBR Magic : aa55 Partition[0] : 1953523120 sectors
sylvain> at 2048 (type fd) /dev/sdc: MBR Magic : aa55 Partition[0] :
sylvain> 1953520002 sectors at 63 (type fd) /dev/sdd: MBR Magic : aa55
sylvain> Partition[0] : 1953520002 sectors at 63 (type fd) /dev/sde:
sylvain> MBR Magic : aa55 Partition[0] : 4294965247 sectors at 2048
sylvain> (type fd) I have 2 way to solve the issue. The first, is to
sylvain> have special command to pass bad sector during re-assemble as
sylvain> "mdadm --assemble --force /dev/md2 /dev/sd[bcde]1" The second
sylvain> is change the disk sde with the old good one, but some datas
sylvain> have been changed on the raid since i have remove it. But
sylvain> these datas are not important. It's only logs and history
sylvain> activity. What can i do to recover a maximum datas without
sylvain> too many risk? Thank's in advance Best Regards
sylvain> ---------------------------------- Sylvain Depuille (in
sylvain> trouble) sylvain.depuille@laposte.net -- To unsubscribe from
sylvain> this list: send the line "unsubscribe linux-raid" in the body
sylvain> of a message to majordomo@vger.kernel.org More majordomo info
sylvain> at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2014-12-30 12:44 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21665.47943.879001.195325@quad.stoffel.home>
Hello John,
the ddrescue command is finished.
The log file :
# Rescue Logfile. Created by GNU ddrescue version 1.18.1
# Command line: ddrescue --force /dev/sdc1 /dev/sdf1 rescue.log
# Start time: 2014-12-30 11:28:57
# Current time: 2014-12-30 13:40:29
# Copying non-tried blocks... Pass 1 (forwards)
# current_pos current_status
0x89EE5A0000 ?
# pos size status
0x00000000 0x87EB34F000 +
0x87EB34F000 0x00001000 *
0x87EB350000 0x00001000 +
0x87EB351000 0x0001F000 *
0x87EB370000 0x0015B000 +
0x87EB4CB000 0x00005000 *
0x87EB4D0000 0x0BA39000 +
0x87F6F09000 0x00007000 *
0x87F6F10000 0x1F76A0000 +
0x89EE5B0000 0x5EF2580400 ?
Now, can i change the sdc by the sdf disk?
Thank's in advance for your help.
Best Regards
Sylvain Depuille
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "sylvain depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
Envoyé: Lundi 29 Décembre 2014 21:36:23
Objet: Re: Re : Re: Big trouble during reassemble a Raid5
sylvain> Hi john, thanks for your answer! I have change a 1TB disk to
sylvain> growing the raid with 3TB disk. if i can re-insert the old
sylvain> 1TB disk in place of 3TB disk, only some log and history are
sylvain> corrupted. i think that is the best way to relaunch the raid
sylvain> without data loss. But i dont known how change the timestamp
sylvain> of the one raid disk. Have you a magic command to change a
sylvain> timestamp of a raid partition, and how known the timestamp of
sylvain> the other disk of the raid? After' raid relaunch, i can
sylvain> change the burn disk by a 3TB new one. To do the ddrescue, i
sylvain> have a 2TB disk spare! Its not the same geometry, is it
sylvain> possible? thanks in advance for your help
Sylvain,
Always glad to help here. I'm going to try and understand what you
wrote and do my best to reply.
Is the 1Tb disk the bad disk? And if you re-insert it and re-start
the RAID5 array, you only have some minor lost files? If so, I would
probably just copy all the data off the RAID5 onto the single 3Tb disk
as a quick and dirty backup, then I'd use 'dd_rescue' to copy the bad
1Tb disk onto the new 2Tb disk.
All you would have to do is make a partition on the 2tb disk which is
the same size (or a little bigger) than the partition on the 1tb disk,
then copy the partition over like this:
ddrescue /dev/sd[BAD DISK LETTER HERE]1 /dev/sd[2TB disk letter]1 \
/tmp/rescue.log
So say the bad disk is sdc, and the good 2tb is sdf, you would do:
ddrescue /dev/sdc1 /dev/sdf1 /tmp/rescue.log
and let it go. Then you would assemble the array using the NEW 2tb
disk. Ideally you would remove the bad 1tb disk from the system when
trying to do this.
But you really do need send us the output of the following commands:
cat /proc/mdstat
cat /proc/partitions
mdadm --detail /dev/md#
do the above for the RADI5 array.
mdadm --examine /dev/sd#1
for each disk in the RAID5 array.
And we can give you better advice.
Good luck!
sylvain> ---------------------------------- Sylvain Depuille
sylvain> sylvain.depuille@laposte.net ----- Mail d'origine ----- De:
sylvain> John Stoffel <john@stoffel.org> À: sylvain depuille
sylvain> <sylvain.depuille@laposte.net> Cc: linux-raid@vger.kernel.org
sylvain> Envoyé: Mon, 29 Dec 2014 19:32:04 +0100 (CET) Objet: Re: Big
sylvain> trouble during reassemble a Raid5
sylvain> Sylvain, I would recommend that you buy a replacement disk
sylvain> for the one throwing errors and then run dd_rescue to copy as
sylvain> much data from the dying disk to the replacement. Then, and
sylvain> only then, do you try to reassemble the array with the
sylvain> --force option. That disk is dying, and dying quickly. Can
sylvain> you also post the output of mdadm -E /dev/sd[bcde]1 for each
sylvain> disk, even the dying one, so we can look at the counts and
sylvain> give you some more advice. Also, the output of the mdadm
sylvain> --assemble --force /dev/md2 /dev/sd[bcde]1 would also be
sylvain> good. The more info the better. Good luck! John
sylvain> i'm sorry to ask this questions but the raid 5 with 4 disk is
sylvain> in big trouble during re-assemble. 2 disks are out of order.
sylvain> I have change a disk of the raid 5 (sde) to growing the raid.
sylvain> But a second disk (sdc) have too many bad sector during the
sylvain> re-assemble, and shutdown the re-assemble. "mdadm --assemble
sylvain> --force /dev/md2 /dev/sd[bcde]1" I have try to correct bad
sylvain> sectors with badblocks, but it's finished by no more spare
sylvain> sectors and the disk still have some bad sector. badblocks -b
sylvain> 512 -o badblocks-sdc.txt -v -n /dev/sdc 1140170000 1140169336
sylvain> 1140169400 1140169401 1140169402 1140169403 1140169404
sylvain> 1140169405 1140169406 1140169407 1140169416 1140169417
sylvain> 1140169418 1140169419 1140169420 1140169421 1140169422
sylvain> 1140169423
sylvain> For information the mdadm examine return : cat mdadm-exam.txt
sylvain> /dev/sdb: MBR Magic : aa55 Partition[0] : 1953523120 sectors
sylvain> at 2048 (type fd) /dev/sdc: MBR Magic : aa55 Partition[0] :
sylvain> 1953520002 sectors at 63 (type fd) /dev/sdd: MBR Magic : aa55
sylvain> Partition[0] : 1953520002 sectors at 63 (type fd) /dev/sde:
sylvain> MBR Magic : aa55 Partition[0] : 4294965247 sectors at 2048
sylvain> (type fd) I have 2 way to solve the issue. The first, is to
sylvain> have special command to pass bad sector during re-assemble as
sylvain> "mdadm --assemble --force /dev/md2 /dev/sd[bcde]1" The second
sylvain> is change the disk sde with the old good one, but some datas
sylvain> have been changed on the raid since i have remove it. But
sylvain> these datas are not important. It's only logs and history
sylvain> activity. What can i do to recover a maximum datas without
sylvain> too many risk? Thank's in advance Best Regards
sylvain> ---------------------------------- Sylvain Depuille (in
sylvain> trouble) sylvain.depuille@laposte.net -- To unsubscribe from
sylvain> this list: send the line "unsubscribe linux-raid" in the body
sylvain> of a message to majordomo@vger.kernel.org More majordomo info
sylvain> at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Manibalan P @ 2014-12-30 11:06 UTC (permalink / raw)
To: linux-raid; +Cc: NeilBrown
Dear Neil,
Few this for you kind attention,
1. I tried the same test with FC11 (2.6.32 kernel before MD code change). And the issue is not there
2. But with Centos 6.4 (2.6.32 kernel after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
Also, a bug has been raise with RHEL regarding this issue. Please find the bug link "https://access.redhat.com/support/cases/#/case/01320319"
Thanks,
Manibalan.
-----Original Message-----
From: Manibalan P
Sent: Wednesday, December 24, 2014 12:15 PM
To: neilb@suse.de; 'linux-raid'
Cc: 'NeilBrown'
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear Neil,
Few this for you kind attention,
1. I tried the same tesst with FC11 (2.6 kernel before MD code change). And the issue is not there 2. But with Centos 6.4 (2.6 after MD code change). I am getting this issue.. and also even with the latest kernel, able to reproduce the issue.
Thanks,
Manibalan.
-----Original Message-----
From: Manibalan P
Sent: Thursday, December 18, 2014 11:38 AM
To: 'linux-raid'
Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear neil,
I also compiled the latest 3.18 kernel on CentOS 6.4 with GIT MD pull patches form 3.19, that also ran in to the same issue after removing a drive during resync.
Dec 17 19:07:32 ITX002590129362 kernel: Linux version 3.18.0 (root@mycentos6) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Dec 17 15:59:09 EST 2014 Dec 17 19:07:32 ITX002590129362 kernel: Command line: ro root=/dev/md255 rd_NO_LVM rd_NO_DM rhgb quiet md_mod.start_ro=1 nmi_watchdog=1 md_mod.start_dirty_degraded=1 … Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sda6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdb6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdc6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdh6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdi6> Dec 17 19:10:15 ITX002590129362 kernel: md: bind<sdj6> Dec 17 19:10:15 ITX002590129362 kernel: async_tx: api initialized (async) Dec 17 19:10:15 ITX002590129362 kernel: xor: measuring software checksum speed
Dec 17 19:10:15 ITX002590129362 kernel: prefetch64-sse: 10048.000 MB/sec
Dec 17 19:10:15 ITX002590129362 kernel: generic_sse: 8824.000 MB/sec
Dec 17 19:10:15 ITX002590129362 kernel: xor: using function: prefetch64-sse (10048.000 MB/sec)
Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x1 5921 MB/s
Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x2 6933 MB/s
Dec 17 19:10:15 ITX002590129362 kernel: raid6: sse2x4 7476 MB/s
Dec 17 19:10:15 ITX002590129362 kernel: raid6: using algorithm sse2x4 (7476 MB/s) Dec 17 19:10:15 ITX002590129362 kernel: raid6: using ssse3x2 recovery algorithm Dec 17 19:10:15 ITX002590129362 kernel: md: raid6 personality registered for level 6 Dec 17 19:10:15 ITX002590129362 kernel: md: raid5 personality registered for level 5 Dec 17 19:10:15 ITX002590129362 kernel: md: raid4 personality registered for level 4 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: not clean -- starting background reconstruction Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdj6 operational as raid disk 5 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdi6 operational as raid disk 4 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdh6 operational as raid disk 3 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdc6 operational as raid disk 2 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sdb6 operational as raid disk 1 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: device sda6 operational as raid disk 0 Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: allocated 0kB Dec 17 19:10:15 ITX002590129362 kernel: md/raid:md0: raid level 5 active with 6 out of 6 devices, algorithm 2 Dec 17 19:10:15 ITX002590129362 kernel: md0: detected capacity change from 0 to 2361059573760 Dec 17 19:10:15 ITX002590129362 kernel: md0: unknown partition table Dec 17 19:10:35 ITX002590129362 kernel: md: md0 switched to read-write mode.
Dec 17 19:10:35 ITX002590129362 kernel: md: resync of RAID array md0 Dec 17 19:10:35 ITX002590129362 kernel: md: minimum _guaranteed_ speed: 10000 KB/sec/disk.
Dec 17 19:10:35 ITX002590129362 kernel: md: using maximum available idle IO bandwidth (but not more than 30000 KB/sec) for resync.
Dec 17 19:10:35 ITX002590129362 kernel: md: using 128k window, over a total of 461144448k.
…
Started IOs using fio tool.
./fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
…
Removed a drive form the system..
Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Dec 17 19:13:23 ITX002590129362 kernel: mpt2sas0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) ..
Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 02 69 03 70 00 00 10 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 40436592 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 51 b3 d0 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 206681040 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
Dec 17 19:13:23 ITX002590129362 kernel: Read(10): 28 00 0c 3a f3 40 00 00 18 00 Dec 17 19:13:23 ITX002590129362 kernel: blk_update_request: I/O error, dev sdh, sector 205189952 Dec 17 19:13:23 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:23 ITX002590129362 kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK … Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb 00 00 00 08 00 Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] Dec 17 19:13:25 ITX002590129362 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Dec 17 19:13:25 ITX002590129362 kernel: sd 0:0:7:0: [sdh] CDB:
Dec 17 19:13:25 ITX002590129362 kernel: Read(10): 28 00 26 8d eb f0 00 00 10 00 Dec 17 19:13:25 ITX002590129362 aghswap: devpath [0:0:7:0] action [remove] devtype [scsi_disk] Dec 17 19:13:25 ITX002590129362 aghswap: MHSA: Sent event 0 0 7 0 remove scsi_disk Dec 17 19:13:25 ITX002590129362 kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x500605ba0101e305) Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Disk failure on sdh6, disabling device.
Dec 17 19:13:25 ITX002590129362 kernel: md/raid:md0: Operation continuing on 5 devices.
Dec 17 19:13:25 ITX002590129362 kernel: md: md0: resync interrupted.
Dec 17 19:13:25 ITX002590129362 kernel: md: checkpointing resync of md0.
..
Log messages after enabling debufgs on raid5.c, it is getting repeated continuously.
__get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
__get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
__get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
__get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0
__get_priority_stripe: handle: busy hold: empty full_writes: 0 bypass_count: 0 handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x10 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x11 read (null) write (null) written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328 for sector 273480328, rmw=2 rcw=1 handling stripe 65238568, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x10 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x18 read (null) write ffff88081a956b00 written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=65238568 for sector 65238568, rmw=2 rcw=1 handling stripe 713868672, state=0x2041 cnt=1, pd_idx=4, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x10 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x18 read (null) write ffff88081f020100 written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=713868672 for sector 713868672, rmw=2 rcw=1 handling stripe 729622496, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x10 read (null) write (null) written (null)
check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622496 for sector 729622496, rmw=2 rcw=1 handling stripe 729622504, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x10 read (null) write (null) written (null)
check 1: state 0x18 read (null) write ffff88081b9bae00 written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=729622504 for sector 729622504, rmw=2 rcw=1 handling stripe 245773680, state=0x2041 cnt=1, pd_idx=0, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x11 read (null) write (null) written (null)
check 1: state 0x18 read (null) write ffff88081cab7a00 written (null)
check 0: state 0x10 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=245773680 for sector 245773680, rmw=2 rcw=1 handling stripe 867965560, state=0x2041 cnt=1, pd_idx=1, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x18 read (null) write ffff880802b2bf00 written (null)
check 1: state 0x10 read (null) write (null) written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=867965560 for sector 867965560, rmw=2 rcw=1 handling stripe 550162280, state=0x2041 cnt=1, pd_idx=2, qd_idx=-1 , check:0, reconstruct:0
check 5: state 0x11 read (null) write (null) written (null)
check 4: state 0x18 read (null) write ffff880802b08800 written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x10 read (null) write (null) written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x11 read (null) write (null) written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1 force RCW max_degraded=1, recovery_cp=7036944 sh->sector=550162280 for sector 550162280, rmw=2 rcw=1
Thanks,
Manibalan
-----Original Message-----
From: Manibalan P
Sent: Wednesday, December 17, 2014 12:11 PM
To: 'linux-raid'
Cc: 'NeilBrown'; Vijayarankan Muthirisavengopal; Dinakaran N
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear Neil,
The same Issue is reproducible in the latest upstream kernel also.
Tested in "3.17.6" latest stable upstream kernel and find the same issue.
[root@root ~]# modinfo raid456
filename: /lib/modules/3.17.6/kernel/drivers/md/raid456.ko
alias: raid6
alias: raid5
alias: md-level-6
alias: md-raid6
alias: md-personality-8
alias: md-level-4
alias: md-level-5
alias: md-raid4
alias: md-raid5
alias: md-personality-4
description: RAID4/5/6 (striping with parity) personality for MD
license: GPL
srcversion: 0EEF680023FDC7410F7989A
depends: async_raid6_recov,async_pq,async_tx,async_memcpy,async_xor
intree: Y
vermagic: 3.17.6 SMP mod_unload modversions
parm: devices_handle_discard_safely:Set to Y if all devices in each array reliably return zeroes on reads from discarded regions (bool)
Thanks,
Manibalan.
-----Original Message-----
From: Manibalan P
Sent: Wednesday, December 17, 2014 12:01 PM
To: 'linux-raid'
Cc: 'NeilBrown'
Subject: RE: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
Dear Neil,
We are facing IO struck issue with raid5 in the following scenario. (please see the attachment for the complete information) In RAID5 array, if a drive is removed while initialization and the same time if IO is happening to that md. Then IO is getting struck, and md_raid5 thread is using 100 % of CPU. Also the md state showing as resync=PENDING
Kernel : Issue found in the following kernels RHEL 6.5 (2.6.32-431.el6.x86_64) CentOS 7 (kernel-3.10.0-123.13.1.el7.x86_64)
Steps to Reproduce the issue:
1. Created a raid 5 md with 4 drives using the below mdadm command.
mdadm -C /dev/md0 -c 64 -l 5 -f -n 4 -e 1.2 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6
2. Make the md writable
mdadm –readwrite /dev/md0
3. Now md will start initialization
4. Run FIO Tool, the the below said configuration /usr/bin/fio --name=md0 --filename=/dev/md0 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --ioengine=libaio --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
4. During MD initialzing, remove a drive(either using MDADM set faulty/remove or remove manually)
5. Now the IO will struck, and cat /proc/mdstat shows states with resync=PENDING
---------------------------------------------------------------------------------------------
top - output show, md_raid5 using 100% cpu
top - 17:55:06 up 1:09, 3 users, load average: 11.98, 8.53, 3.99
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2690 root 20 0 0 0 0 R 100.0 0.0 6:44.41 md0_raid5
---------------------------------------------------------------------------------------------
dmesg - show the stack trace
INFO: task fio:2715 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000a 0 2715 2654 0x00000080
ffff88043b623598 0000000000000082 0000000000000000 ffffffff81058d53
ffff88043b623548 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b40b098 ffff88043b623fd8 000000000000fbc8 ffff88043b40b098 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8140fa39>] ? md_wakeup_thread+0x39/0x70 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffffa0308f66>] ? make_request+0x306/0xc6c [raid456] [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81122283>] ? mempool_alloc+0x63/0x140 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c767a>] do_direct_IO+0x7ca/0xfa0 [<ffffffff811c8196>] __blockdev_direct_IO_newtrunc+0x346/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2717 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000004 0 2717 2654 0x00000080
ffff880439e97698 0000000000000082 ffff880439e97628 ffffffff81058d53
ffff880439e97648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0adab8 ffff880439e97fd8 000000000000fbc8 ffff88043b0adab8 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2718 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000005 0 2718 2654 0x00000080
ffff88043bc13698 0000000000000082 ffff88043bc13628 ffffffff81058d53
ffff88043bc13648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ad058 ffff88043bc13fd8 000000000000fbc8 ffff88043b0ad058 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8e50>] __blockdev_direct_IO_newtrunc+0x1000/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2719 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000001 0 2719 2654 0x00000080
ffff880439ebb698 0000000000000082 ffff880439ebb628 ffffffff81058d53
ffff880439ebb648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043b0ac5f8 ffff880439ebbfd8 000000000000fbc8 ffff88043b0ac5f8 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2720 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000008 0 2720 2654 0x00000080
ffff88043b8cf698 0000000000000082 ffff88043b8cf628 ffffffff81058d53
ffff88043b8cf648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89af8 ffff88043b8cffd8 000000000000fbc8 ffff880439e89af8 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2721 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2721 2654 0x00000080
ffff88043b047698 0000000000000082 ffff88043b047628 ffffffff81058d53
ffff88043b047648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e89098 ffff88043b047fd8 000000000000fbc8 ffff880439e89098 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2722 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000000 0 2722 2654 0x00000080
ffff880439ea3698 0000000000000082 ffff880439ea3628 ffffffff81058d53
ffff880439ea3648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff880439e88638 ffff880439ea3fd8 000000000000fbc8 ffff880439e88638 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2723 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000006 0 2723 2654 0x00000080
ffff88043bf5f698 0000000000000082 ffff88043bf5f628 ffffffff81058d53
ffff88043bf5f648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183ab8 ffff88043bf5ffd8 000000000000fbc8 ffff88043a183ab8 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2724 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 000000000000000b 0 2724 2654 0x00000080
ffff88043be05698 0000000000000082 ffff88043be05628 ffffffff81058d53
ffff88043be05648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a183058 ffff88043be05fd8 000000000000fbc8 ffff88043a183058 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task fio:2725 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
fio D 0000000000000003 0 2725 2654 0x00000080
ffff88043be07698 0000000000000082 ffff88043be07628 ffffffff81058d53
ffff88043be07648 ffff880230e49cc0 ffff8802389aa228 ffff88043b2ad1b8
ffff88043a1825f8 ffff88043be07fd8 000000000000fbc8 ffff88043a1825f8 Call Trace:
[<ffffffff81058d53>] ? __wake_up+0x53/0x70 [<ffffffffa030334b>] ? md_raid5_unplug_device+0x7b/0x100 [raid456] [<ffffffffa0304146>] get_active_stripe+0x236/0x830 [raid456] [<ffffffff81065df0>] ? default_wake_function+0x0/0x20 [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa0308e15>] make_request+0x1b5/0xc6c [raid456] [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff811220e5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81415b41>] md_make_request+0xe1/0x230 [<ffffffff811c3fd2>] ? bvec_alloc_bs+0x62/0x110 [<ffffffff811c32f0>] ? __bio_add_page+0x110/0x230 [<ffffffff81266c50>] generic_make_request+0x240/0x5a0 [<ffffffff811c742c>] ? do_direct_IO+0x57c/0xfa0 [<ffffffff81267020>] submit_bio+0x70/0x120 [<ffffffff811c8acd>] __blockdev_direct_IO_newtrunc+0xc7d/0x1270
[<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c9137>] __blockdev_direct_IO+0x77/0xe0 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff811c53b7>] blkdev_direct_IO+0x57/0x60 [<ffffffff811c4330>] ? blkdev_get_block+0x0/0x20 [<ffffffff81120552>] generic_file_direct_write+0xc2/0x190
[<ffffffff81121e71>] __generic_file_aio_write+0x3a1/0x490
[<ffffffff811d64c0>] ? aio_read_evt+0xa0/0x170 [<ffffffff811c490c>] blkdev_aio_write+0x3c/0xa0 [<ffffffff811c48d0>] ? blkdev_aio_write+0x0/0xa0 [<ffffffff811d4f64>] aio_rw_vect_retry+0x84/0x200 [<ffffffff811d6924>] aio_run_iocb+0x64/0x170 [<ffffffff811d7d51>] do_io_submit+0x291/0x920 [<ffffffff811d83f0>] sys_io_submit+0x10/0x20 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[root@root ~]# cat /proc/2690/stack
[<ffffffff810686da>] __cond_resched+0x2a/0x40 [<ffffffffa030361c>] ops_run_io+0x2c/0x920 [raid456] [<ffffffffa03052cc>] handle_stripe+0x9cc/0x2980 [raid456] [<ffffffffa03078a4>] raid5d+0x624/0x850 [raid456] [<ffffffff81416f05>] md_thread+0x115/0x150 [<ffffffff8109aef6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffffffffffff>] 0xffffffffffffffff
[root@root ~]# cat /proc/2690/stat
2690 (md0_raid5) R 2 0 0 0 -1 2149613632 0 0 0 0 0 68495 0 0 20 0 1 0 350990 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483391 256 0 0 0 17 2 0 0 6855 0 0 [root@root ~]# cat /proc/2690/statm
0 0 0 0 0 0 0
[root@root ~]# cat /proc/2690/stat
stat statm status
[root@root ~]# cat /proc/2690/status
Name: md0_raid5
State: R (running)
Tgid: 2690
Pid: 2690
PPid: 2
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
Utrace: 0
FDSize: 64
Groups:
Threads: 1
SigQ: 2/128402
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: fffffffffffffeff
SigCgt: 0000000000000100
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: fffffffffffffeff
CapBnd: ffffffffffffffff
Cpus_allowed: ffffff
Cpus_allowed_list: 0-23
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 5411612
nonvoluntary_ctxt_switches: 257032
Thanks,
Manibalan.
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: sylvain.depuille @ 2014-12-30 9:37 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <21665.47943.879001.195325@quad.stoffel.home>
Hi John,
I'm sorry! I'm French and my English is poor!
Yesterday, i wrote the mail on a tablett, but my mail's provider doesn't offer plain text option on mobile site :-(
I'm back on a PC under Linux :-) (not crash this one, but not on the same site, connected by ssh to the broke system).
For the history, i have installed PC of many friends under Linux (no more Virus problem and Windows Blue Screen).
Yesterday, i have try to you explain what is the history of the issue.
The Raid was fully operational before i change the disk sde by a 3TB disk to grow the Raid, by changing all disks of the raid.
But when i have do the re-assemble the disk sdc goes burny :-(
I have keep the old sde 1TB disk in state.
The result of your asked command :
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : inactive sdc1[2](S) sdd1[3](S) sde1[4](S) sdb1[5](S)
5077760089 blocks super 1.2
major minor #blocks name
8 0 125034840 sda
8 1 131072 sda1
8 2 62914560 sda2
8 3 61865984 sda3
8 32 976762584 sdc
8 33 976760001 sdc1
8 48 976762584 sdd
8 49 976760001 sdd1
8 16 976762584 sdb
8 17 976761560 sdb1
8 64 2930266584 sde
8 65 2147482623 sde1
mdadm --detail /dev/md2
/dev/md2:
Version : 1.2
Raid Level : raid0
Total Devices : 4
Persistence : Superblock is persistent
State : inactive
Name : le-bohec:2 (local to host le-bohec)
UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
Events : 167456
Number Major Minor RaidDevice
- 8 17 - /dev/sdb1
- 8 33 - /dev/sdc1
- 8 49 - /dev/sdd1
- 8 65 - /dev/sde1
mdadm -E /dev/sd[bcde]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
Name : le-bohec:2 (local to host le-bohec)
Creation Time : Tue Apr 9 17:56:19 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953521072 (931.51 GiB 1000.20 GB)
Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=3504 sectors
State : clean
Device UUID : 8506e09c:b87a44ed:7b4ee314:777ce89c
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 27 22:08:34 2014
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bad62d22 - correct
Events : 167456
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
Name : le-bohec:2 (local to host le-bohec)
Creation Time : Tue Apr 9 17:56:19 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=386 sectors
State : clean
Device UUID : 44002aad:d3e17729:a93854eb:4139972e
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 27 22:08:22 2014
Checksum : 6f69285d - correct
Events : 167431
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
Name : le-bohec:2 (local to host le-bohec)
Creation Time : Tue Apr 9 17:56:19 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=386 sectors
State : clean
Device UUID : 5cff6f7f:ea6b89b6:28e4e8b3:7a2b5a7f
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 27 22:08:34 2014
Checksum : e81a4f63 - correct
Events : 167456
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 2a1440cd:762a90fb:e3bd2f4d:617acb0e
Name : le-bohec:2 (local to host le-bohec)
Creation Time : Tue Apr 9 17:56:19 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 4294963199 (2048.00 GiB 2199.02 GB)
Array Size : 2930276352 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1953517568 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=2341445631 sectors
State : clean
Device UUID : 0ebce28d:1a792d55:76a86538:12cc94dd
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 27 22:08:34 2014
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 3801cfa - correct
Events : 167456
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
Now, i have installed ddrescue on the system, and i go to my friends to add the 2TB disk in the tower, and launch the command.
Many thank's for your help and your patience.
Best Regards
Sylvain Depuille
----- Mail original -----
De: "John Stoffel" <john@stoffel.org>
À: "sylvain depuille" <sylvain.depuille@laposte.net>
Cc: "John Stoffel" <john@stoffel.org>, linux-raid@vger.kernel.org
Envoyé: Lundi 29 Décembre 2014 21:36:23
Objet: Re: Re : Re: Big trouble during reassemble a Raid5
sylvain> Hi john, thanks for your answer! I have change a 1TB disk to
sylvain> growing the raid with 3TB disk. if i can re-insert the old
sylvain> 1TB disk in place of 3TB disk, only some log and history are
sylvain> corrupted. i think that is the best way to relaunch the raid
sylvain> without data loss. But i dont known how change the timestamp
sylvain> of the one raid disk. Have you a magic command to change a
sylvain> timestamp of a raid partition, and how known the timestamp of
sylvain> the other disk of the raid? After' raid relaunch, i can
sylvain> change the burn disk by a 3TB new one. To do the ddrescue, i
sylvain> have a 2TB disk spare! Its not the same geometry, is it
sylvain> possible? thanks in advance for your help
Sylvain,
Always glad to help here. I'm going to try and understand what you
wrote and do my best to reply.
Is the 1Tb disk the bad disk? And if you re-insert it and re-start
the RAID5 array, you only have some minor lost files? If so, I would
probably just copy all the data off the RAID5 onto the single 3Tb disk
as a quick and dirty backup, then I'd use 'dd_rescue' to copy the bad
1Tb disk onto the new 2Tb disk.
All you would have to do is make a partition on the 2tb disk which is
the same size (or a little bigger) than the partition on the 1tb disk,
then copy the partition over like this:
ddrescue /dev/sd[BAD DISK LETTER HERE]1 /dev/sd[2TB disk letter]1 \
/tmp/rescue.log
So say the bad disk is sdc, and the good 2tb is sdf, you would do:
ddrescue /dev/sdc1 /dev/sdf1 /tmp/rescue.log
and let it go. Then you would assemble the array using the NEW 2tb
disk. Ideally you would remove the bad 1tb disk from the system when
trying to do this.
But you really do need send us the output of the following commands:
cat /proc/mdstat
cat /proc/partitions
mdadm --detail /dev/md#
do the above for the RADI5 array.
mdadm --examine /dev/sd#1
for each disk in the RAID5 array.
And we can give you better advice.
Good luck!
sylvain> ---------------------------------- Sylvain Depuille
sylvain> sylvain.depuille@laposte.net ----- Mail d'origine ----- De:
sylvain> John Stoffel <john@stoffel.org> À: sylvain depuille
sylvain> <sylvain.depuille@laposte.net> Cc: linux-raid@vger.kernel.org
sylvain> Envoyé: Mon, 29 Dec 2014 19:32:04 +0100 (CET) Objet: Re: Big
sylvain> trouble during reassemble a Raid5
sylvain> Sylvain, I would recommend that you buy a replacement disk
sylvain> for the one throwing errors and then run dd_rescue to copy as
sylvain> much data from the dying disk to the replacement. Then, and
sylvain> only then, do you try to reassemble the array with the
sylvain> --force option. That disk is dying, and dying quickly. Can
sylvain> you also post the output of mdadm -E /dev/sd[bcde]1 for each
sylvain> disk, even the dying one, so we can look at the counts and
sylvain> give you some more advice. Also, the output of the mdadm
sylvain> --assemble --force /dev/md2 /dev/sd[bcde]1 would also be
sylvain> good. The more info the better. Good luck! John
sylvain> i'm sorry to ask this questions but the raid 5 with 4 disk is
sylvain> in big trouble during re-assemble. 2 disks are out of order.
sylvain> I have change a disk of the raid 5 (sde) to growing the raid.
sylvain> But a second disk (sdc) have too many bad sector during the
sylvain> re-assemble, and shutdown the re-assemble. "mdadm --assemble
sylvain> --force /dev/md2 /dev/sd[bcde]1" I have try to correct bad
sylvain> sectors with badblocks, but it's finished by no more spare
sylvain> sectors and the disk still have some bad sector. badblocks -b
sylvain> 512 -o badblocks-sdc.txt -v -n /dev/sdc 1140170000 1140169336
sylvain> 1140169400 1140169401 1140169402 1140169403 1140169404
sylvain> 1140169405 1140169406 1140169407 1140169416 1140169417
sylvain> 1140169418 1140169419 1140169420 1140169421 1140169422
sylvain> 1140169423
sylvain> For information the mdadm examine return : cat mdadm-exam.txt
sylvain> /dev/sdb: MBR Magic : aa55 Partition[0] : 1953523120 sectors
sylvain> at 2048 (type fd) /dev/sdc: MBR Magic : aa55 Partition[0] :
sylvain> 1953520002 sectors at 63 (type fd) /dev/sdd: MBR Magic : aa55
sylvain> Partition[0] : 1953520002 sectors at 63 (type fd) /dev/sde:
sylvain> MBR Magic : aa55 Partition[0] : 4294965247 sectors at 2048
sylvain> (type fd) I have 2 way to solve the issue. The first, is to
sylvain> have special command to pass bad sector during re-assemble as
sylvain> "mdadm --assemble --force /dev/md2 /dev/sd[bcde]1" The second
sylvain> is change the disk sde with the old good one, but some datas
sylvain> have been changed on the raid since i have remove it. But
sylvain> these datas are not important. It's only logs and history
sylvain> activity. What can i do to recover a maximum datas without
sylvain> too many risk? Thank's in advance Best Regards
sylvain> ---------------------------------- Sylvain Depuille (in
sylvain> trouble) sylvain.depuille@laposte.net -- To unsubscribe from
sylvain> this list: send the line "unsubscribe linux-raid" in the body
sylvain> of a message to majordomo@vger.kernel.org More majordomo info
sylvain> at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Can't reshape raid0 to raid10
From: Xiao Ni @ 2014-12-30 3:13 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1718053393.1731733.1419906625563.JavaMail.zimbra@redhat.com>
Hi Neil
When I try to reshape a raid0 to raid10, it'll fail like this:
[root@dhcp-12-133 mdadm-3.3.2]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
├─sda1 8:1 0 1000M 0 part /boot
├─sda2 8:2 0 29.3G 0 part /
├─sda3 8:3 0 512M 0 part [SWAP]
├─sda4 8:4 0 1K 0 part
├─sda5 8:5 0 102M 0 part
└─sda6 8:6 0 10.1G 0 part
└─VolGroup00-LogVol00 254:0 0 9.9G 0 lvm
sdb 8:16 0 111.8G 0 disk
├─sdb1 8:17 0 2G 0 part
└─sdb2 8:18 0 10G 0 part
sdc 8:32 0 186.3G 0 disk
├─sdc1 8:33 0 2G 0 part
└─sdc2 8:34 0 10G 0 part
sdd 8:48 0 111.8G 0 disk
├─sdd1 8:49 0 2G 0 part
└─sdd2 8:50 0 10G 0 part
[root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2 /dev/sdc2 /dev/sdd2
mdadm: level of /dev/md0 changed to raid10
mdadm: add new device failed for /dev/sdb2 as 6: No space left on device
But if I reshape the raid0 to raid5, reshape raid5 to raid0, then reshape raid0 to raid10 use
the same command it'll succeed.
[root@dhcp-12-133 mdadm-3.3.2]# mdadm -CR /dev/md0 -l0 -n3 /dev/sdb1 /dev/sdc1 /dev/sdd1
[root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l5
[root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
md0 : active raid5 sdd1[2] sdc1[1] sdb1[0]
6285312 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
unused devices: <none>
[root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l0
[root@dhcp-12-133 mdadm-3.3.2]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
md0 : active raid0 sdd1[2] sdc1[1] sdb1[0]
6285312 blocks super 1.2 512k chunks
unused devices: <none>
[root@dhcp-12-133 mdadm-3.3.2]# mdadm --grow /dev/md0 -l10 -a /dev/sdb2 /dev/sdc2 /dev/sdd2
mdadm: level of /dev/md0 changed to raid10
mdadm: added /dev/sdb2
mdadm: added /dev/sdc2
mdadm: added /dev/sdd2
So I guess it's the problem add the disk to raid10 after the reshaping. In the function
super_1_validate, it'll set the mddev->dev_sectors using the superblock read from disks.
If it's raid0, the le64_to_cpu(sb-size) is 0. So when add disk to raid10 bind_rdev_to_array
return -ENOSPC.
When create raid0, it doesn't write give the value to s->size. So the sb-size is 0.
I modify the code about Create.c. I'm not sure whether it's right to do so. But it can resolve
the problem.
diff --git a/Create.c b/Create.c
index 330c5b4..f3135c5 100644
--- a/Create.c
+++ b/Create.c
@@ -489,7 +489,7 @@ int Create(struct supertype *st, char *mddev,
pr_err("no size and no drives given - aborting create.\n");
return 1;
}
- if (s->level > 0 || s->level == LEVEL_MULTIPATH
+ if (s->level >= 0 || s->level == LEVEL_MULTIPATH
|| s->level == LEVEL_FAULTY
|| st->ss->external ) {
/* size is meaningful */
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* AW: md_raid5 recovering failed need help
From: Stephan Hafiz @ 2014-12-29 21:18 UTC (permalink / raw)
To: 'Phil Turmel', 'linux-raid'
In-Reply-To: <549ECF26.2040104@turmel.org>
Good day Phil,
David is ok, Stephan too. David is my second firstname. I always think stephan is for english native speaker hard to say (?)
!SMART -x at the end
!Distro
PRETTY_NAME="Ubuntu 14.04.1 LTS"
Ubuntu 3.13.0-43.72-generic 3.13.11.11
Linux pluto 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
!logs
Unfortunately, no syslogmessages, the raid has failed in july.... since then i try to find the cause ...
I have only my puttylogs.
! sysrescuecd.org CD
:-) !!! yes, it works now! But on disk was/is removed. Backup to an other NAS is in progress.
( The md was /dev/md0 )
I can't understand why this distro has no problems to assemble this raid ???
Thx very mutch for your time to help me.... in german we say "i stand on the hose" :-)
Happy new Year for you an your familie, David (Stephan)
-----Ursprüngliche Nachricht-----
Von: Phil Turmel [mailto:philip@turmel.org]
Gesendet: Samstag, 27. Dezember 2014 16:24
An: Stephan Hafiz; linux-raid
Betreff: Re: md_raid5 recovering failed need help
Good morning David,
{ or Stephan ? }
On 12/25/2014 09:24 AM, Stephan Hafiz wrote:
> Hi! I’m from germany and my raid and me needs help.
> My english isn’t very good, but i think it’s sufficient. And i think,
> this mailinglist is my last hope ☺
This is the right place for problems with linux raid arrays.
> So on, …. Here ist my problem.
> The raid5 has lost 2 of 5 disks. First one disk and then the second one.
Ok. Not uncommon.
[trim /]
> !SMART Status
> for i in a b c d e f; do echo Device sd$i; smartctl -H /dev/sd$i |
> egrep overall; echo; done; Device sda SMART overall-health
> self-assessment test result: PASSED
>
> Device sdb
> SMART overall-health self-assessment test result: PASSED
>
> Device sdc
> SMART overall-health self-assessment test result: PASSED
>
> Device sdd
> SMART overall-health self-assessment test result: PASSED
>
> Device sde
> SMART overall-health self-assessment test result: PASSED
>
> Device sdf
> SMART overall-health self-assessment test result: PASSED
It is extremely common to have an overall result of "PASSED" when you aren't safe at all. Please redo this without trimming, like so:
for x in /dev/sd[b-f] ; do echo $x ; smartctl -x $x ; done
Paste the result at the end of you next mail--no need to attach nor need for pastebin services.
Also, if you still have any syslogs from the time of the failure, it would be good to see the kernel messages that triggered the drive ejections from the raid.
> !mdadm version
> mdadm - v3.2.5 - 18th May 2012
> I have read about recent versions 3.3.x @ raid.wiki.kernel.org, i haven’t tested this version.
It may be necessary. You haven't reported your distro nor your kernel version.
> !superblock informations
> Only the Events from sdb1 are off
[trim /]
Very good report! You've saved all the superblocks and you haven't tried to do any --create operations.
[trim /]
> !reassemble force
> mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
> /dev/sdf1 --force
> mdadm: ignoring /dev/sdd1 as it reports /dev/sdc1 as failed
> mdadm: ignoring /dev/sde1 as it reports /dev/sdc1 as failed
> mdadm: ignoring /dev/sdf1 as it reports /dev/sdc1 as failed
> mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.
This should have worked. Hmmm.
> i hope i don’t get the award „paint onself in to the corner“ ……
Probably not. :-)
The simplest way forward would probably be to boot a rescue CD (I generally use the one from sysrescuecd.org) that has a recent kernel and mdadm combination. Such CDs will probably attempt to assemble your array during boot to /dev/md127 instead of /dev/md0, but it will fail.
So, within the rescue environment, do:
mdadm --stop /dev/md127 {or whatever shows in /proc/mdstat}
mdadm --assemble --force --verbose /dev/md0 /dev/sd[b-f]1
If that doesn't work, show us the verbose output, along with the matching part of the dmesg.
If it does work, just do a clean shutdown and reboot back into your regular OS.
> merry christmas … David
And Merry Christmas to you!
When you are done celebrating the revival of your array, you will need to find out why it broke in the first place. The most common cause seen on this list is the use of consumer-grade drives without dealing with the timeout mismatch problem. You might want to review this old thread:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
Phil
#####################################################################################
root@pluto:~# for x in /dev/sd[b-f] ; do echo $x ; smartctl -x $x ; done
/dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-43-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68AX9N0
Serial Number: WD-WMC1T3211821
LU WWN Device Id: 5 0014ee 6034288f1
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 28 21:36:39 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (40680) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 408) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 208 176 021 - 4583
4 Start_Stop_Count -O--CK 100 100 000 - 108
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 094 094 000 - 4704
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 107
192 Power-Off_Retract_Count -O--CK 200 200 000 - 16
193 Load_Cycle_Count -O--CK 200 200 000 - 91
194 Temperature_Celsius -O---K 113 103 000 - 37
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 31646 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 31646 [13] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.154 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.154 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.154 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.153 IDENTIFY DEVICE
Error 31645 [12] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.154 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.153 IDENTIFY DEVICE
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.153 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
Error 31644 [11] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.153 IDENTIFY DEVICE
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.153 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
Error 31643 [10] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 08 00 00 00 00 00 00 e0 00 Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.153 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Enable SATA feature]
Error 31642 [9] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.153 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
Error 31641 [8] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.150 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.150 SET FEATURES [Enable SATA feature]
Error 31640 [7] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.151 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.151 IDENTIFY DEVICE
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.150 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.150 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.150 IDENTIFY DEVICE
Error 31639 [6] occurred at disk power-on lifetime: 4520 hours (188 days + 8 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 08 00 00 00 00 00 00 e0 00 Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
c8 00 00 00 08 00 00 00 00 00 00 e0 08 11d+03:17:23.150 READ DMA
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.150 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 11d+03:17:23.150 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 11d+03:17:23.149 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 11d+03:17:23.149 SET FEATURES [Enable SATA feature]
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 60% 1605 -
# 2 Extended offline Completed without error 00% 1600 -
# 3 Extended offline Completed without error 00% 1579 -
# 4 Extended offline Completed without error 00% 1567 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 21/39 Celsius
Lifetime Min/Max Temperature: 16/47 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (135)
Index Estimated Time Temperature Celsius
136 2014-12-28 13:39 37 ******************
... ..(476 skipped). .. ******************
135 2014-12-28 21:36 37 ******************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 4 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 303936 Vendor specific
/dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-43-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68AX9N0
Serial Number: WD-WMC1T3022609
LU WWN Device Id: 5 0014ee 603429af6
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 28 21:36:40 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (39840) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 399) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 191 174 021 - 5450
4 Start_Stop_Count -O--CK 100 100 000 - 105
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 094 094 000 - 4713
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 104
192 Power-Off_Retract_Count -O--CK 200 200 000 - 12
193 Load_Cycle_Count -O--CK 200 200 000 - 92
194 Temperature_Celsius -O---K 113 102 000 - 37
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 17984 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 17984 [7] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 3f 00 00 0f 00 00 01 af 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
91 00 20 00 3f 00 00 0f 00 00 01 af 00 15:41:39.736 INITIALIZE DEVICE PARAMETERS [OBS-6]
ec 00 00 00 00 00 00 00 00 00 01 a0 00 15:41:39.734 IDENTIFY DEVICE
ec 00 00 00 01 00 00 00 00 00 01 a0 00 15:41:39.438 IDENTIFY DEVICE
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.452 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.451 IDENTIFY DEVICE
Error 17983 [6] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.452 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.451 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 15:41:32.451 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.447 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.446 IDENTIFY DEVICE
Error 17982 [5] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 15:41:32.451 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.447 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.446 IDENTIFY DEVICE
ea 00 00 00 00 00 00 00 00 00 00 e0 08 15:41:32.421 FLUSH CACHE EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.417 SET FEATURES [Enable SATA feature]
Error 17981 [4] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.447 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.446 IDENTIFY DEVICE
ea 00 00 00 00 00 00 00 00 00 00 e0 08 15:41:32.421 FLUSH CACHE EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.417 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.417 IDENTIFY DEVICE
Error 17980 [3] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 00 00 00 00 00 00 00 a0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ea 00 00 00 00 00 00 00 00 00 00 e0 08 15:41:32.421 FLUSH CACHE EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.417 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.417 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 15:41:32.416 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.412 SET FEATURES [Enable SATA feature]
Error 17979 [2] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.417 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.417 IDENTIFY DEVICE
ef 00 03 00 46 00 00 00 00 00 00 a0 08 15:41:32.416 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.412 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.412 IDENTIFY DEVICE
Error 17978 [1] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 46 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 03 00 46 00 00 00 00 00 00 a0 08 15:41:32.416 SET FEATURES [Set transfer mode]
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.412 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.412 IDENTIFY DEVICE
ea 00 00 00 00 00 00 00 00 00 00 e0 08 15:41:32.387 FLUSH CACHE EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.383 SET FEATURES [Enable SATA feature]
Error 17977 [0] occurred at disk power-on lifetime: 4545 hours (189 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 61 00 02 00 00 00 00 00 00 a0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.412 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.412 IDENTIFY DEVICE
ea 00 00 00 00 00 00 00 00 00 00 e0 08 15:41:32.387 FLUSH CACHE EXT
ef 00 10 00 02 00 00 00 00 00 00 a0 08 15:41:32.383 SET FEATURES [Enable SATA feature]
ec 00 00 00 00 00 00 00 00 00 00 a0 08 15:41:32.383 IDENTIFY DEVICE
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 60% 1614 -
# 2 Extended offline Completed without error 00% 1609 -
# 3 Extended offline Completed without error 00% 1588 -
# 4 Extended offline Completed without error 00% 1575 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 22/39 Celsius
Lifetime Min/Max Temperature: 15/48 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (190)
Index Estimated Time Temperature Celsius
191 2014-12-28 13:39 37 ******************
... ..(476 skipped). .. ******************
190 2014-12-28 21:36 37 ******************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 4 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 303934 Vendor specific
/dev/sdd
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-43-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68AX9N0
Serial Number: WD-WMC1T3222539
LU WWN Device Id: 5 0014ee 603429eeb
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 28 21:36:40 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (38400) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 385) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 205 171 021 - 4733
4 Start_Stop_Count -O--CK 100 100 000 - 105
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 094 094 000 - 4696
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 104
192 Power-Off_Retract_Count -O--CK 200 200 000 - 11
193 Load_Cycle_Count -O--CK 200 200 000 - 93
194 Temperature_Celsius -O---K 110 099 000 - 40
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 147 000 - 67
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 60% 1597 -
# 2 Extended offline Completed without error 00% 1591 -
# 3 Extended offline Completed without error 00% 1570 -
# 4 Extended offline Completed without error 00% 1557 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 40 Celsius
Power Cycle Min/Max Temperature: 21/41 Celsius
Lifetime Min/Max Temperature: 15/51 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (70)
Index Estimated Time Temperature Celsius
71 2014-12-28 13:39 39 ********************
... ..(186 skipped). .. ********************
258 2014-12-28 16:46 39 ********************
259 2014-12-28 16:47 40 *********************
... ..( 10 skipped). .. *********************
270 2014-12-28 16:58 40 *********************
271 2014-12-28 16:59 39 ********************
... ..(276 skipped). .. ********************
70 2014-12-28 21:36 39 ********************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 4 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 303940 Vendor specific
/dev/sde
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-43-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68AX9N0
Serial Number: WD-WMC1T3057889
LU WWN Device Id: 5 0014ee 658a09a4d
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 28 21:36:40 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (39720) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 399) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 187 171 021 - 5608
4 Start_Stop_Count -O--CK 100 100 000 - 101
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 094 094 000 - 4606
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 253 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 100
192 Power-Off_Retract_Count -O--CK 200 200 000 - 9
193 Load_Cycle_Count -O--CK 200 200 000 - 91
194 Temperature_Celsius -O---K 113 105 000 - 37
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 60% 1508 -
# 2 Extended offline Completed without error 00% 1502 -
# 3 Extended offline Completed without error 00% 1481 -
# 4 Extended offline Completed without error 00% 1469 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 22/39 Celsius
Lifetime Min/Max Temperature: 16/46 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (9)
Index Estimated Time Temperature Celsius
10 2014-12-28 13:39 37 ******************
... ..(476 skipped). .. ******************
9 2014-12-28 21:36 37 ******************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 4 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 303940 Vendor specific
/dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-43-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (AF)
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WMC4N0368841
LU WWN Device Id: 5 0014ee 6ae752b54
Firmware Version: 80.00A80
User Capacity: 3.000.592.982.016 bytes [3,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sun Dec 28 21:36:40 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (40980) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 411) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 207 173 021 - 4641
4 Start_Stop_Count -O--CK 100 100 000 - 50
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 096 096 000 - 3098
10 Spin_Retry_Count -O--CK 100 253 000 - 0
11 Calibration_Retry_Count -O--CK 100 253 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 50
192 Power-Off_Retract_Count -O--CK 200 200 000 - 6
193 Load_Cycle_Count -O--CK 200 200 000 - 43
194 Temperature_Celsius -O---K 113 102 000 - 37
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 100 253 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 100 253 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb7 GPL,SL VS 1 Device vendor specific log
0xbd GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 93 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 22/39 Celsius
Lifetime Min/Max Temperature: 16/48 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (401)
Index Estimated Time Temperature Celsius
402 2014-12-28 13:39 37 ******************
... ..(476 skipped). .. ******************
401 2014-12-28 21:36 37 ******************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 4 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 303942 Vendor specific
root@pluto:~#
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Re : Re: Big trouble during reassemble a Raid5
From: John Stoffel @ 2014-12-29 20:36 UTC (permalink / raw)
To: sylvain.depuille; +Cc: John Stoffel, linux-raid
In-Reply-To: <164935924.13594562.1419881845418.JavaMail.zimbra@laposte.net>
sylvain> Hi john, thanks for your answer! I have change a 1TB disk to
sylvain> growing the raid with 3TB disk. if i can re-insert the old
sylvain> 1TB disk in place of 3TB disk, only some log and history are
sylvain> corrupted. i think that is the best way to relaunch the raid
sylvain> without data loss. But i dont known how change the timestamp
sylvain> of the one raid disk. Have you a magic command to change a
sylvain> timestamp of a raid partition, and how known the timestamp of
sylvain> the other disk of the raid? After' raid relaunch, i can
sylvain> change the burn disk by a 3TB new one. To do the ddrescue, i
sylvain> have a 2TB disk spare! Its not the same geometry, is it
sylvain> possible? thanks in advance for your help
Sylvain,
Always glad to help here. I'm going to try and understand what you
wrote and do my best to reply.
Is the 1Tb disk the bad disk? And if you re-insert it and re-start
the RAID5 array, you only have some minor lost files? If so, I would
probably just copy all the data off the RAID5 onto the single 3Tb disk
as a quick and dirty backup, then I'd use 'dd_rescue' to copy the bad
1Tb disk onto the new 2Tb disk.
All you would have to do is make a partition on the 2tb disk which is
the same size (or a little bigger) than the partition on the 1tb disk,
then copy the partition over like this:
ddrescue /dev/sd[BAD DISK LETTER HERE]1 /dev/sd[2TB disk letter]1 \
/tmp/rescue.log
So say the bad disk is sdc, and the good 2tb is sdf, you would do:
ddrescue /dev/sdc1 /dev/sdf1 /tmp/rescue.log
and let it go. Then you would assemble the array using the NEW 2tb
disk. Ideally you would remove the bad 1tb disk from the system when
trying to do this.
But you really do need send us the output of the following commands:
cat /proc/mdstat
cat /proc/partitions
mdadm --detail /dev/md#
do the above for the RADI5 array.
mdadm --examine /dev/sd#1
for each disk in the RAID5 array.
And we can give you better advice.
Good luck!
sylvain> ---------------------------------- Sylvain Depuille
sylvain> sylvain.depuille@laposte.net ----- Mail d'origine ----- De:
sylvain> John Stoffel <john@stoffel.org> À: sylvain depuille
sylvain> <sylvain.depuille@laposte.net> Cc: linux-raid@vger.kernel.org
sylvain> Envoyé: Mon, 29 Dec 2014 19:32:04 +0100 (CET) Objet: Re: Big
sylvain> trouble during reassemble a Raid5
sylvain> Sylvain, I would recommend that you buy a replacement disk
sylvain> for the one throwing errors and then run dd_rescue to copy as
sylvain> much data from the dying disk to the replacement. Then, and
sylvain> only then, do you try to reassemble the array with the
sylvain> --force option. That disk is dying, and dying quickly. Can
sylvain> you also post the output of mdadm -E /dev/sd[bcde]1 for each
sylvain> disk, even the dying one, so we can look at the counts and
sylvain> give you some more advice. Also, the output of the mdadm
sylvain> --assemble --force /dev/md2 /dev/sd[bcde]1 would also be
sylvain> good. The more info the better. Good luck! John
sylvain> i'm sorry to ask this questions but the raid 5 with 4 disk is
sylvain> in big trouble during re-assemble. 2 disks are out of order.
sylvain> I have change a disk of the raid 5 (sde) to growing the raid.
sylvain> But a second disk (sdc) have too many bad sector during the
sylvain> re-assemble, and shutdown the re-assemble. "mdadm --assemble
sylvain> --force /dev/md2 /dev/sd[bcde]1" I have try to correct bad
sylvain> sectors with badblocks, but it's finished by no more spare
sylvain> sectors and the disk still have some bad sector. badblocks -b
sylvain> 512 -o badblocks-sdc.txt -v -n /dev/sdc 1140170000 1140169336
sylvain> 1140169400 1140169401 1140169402 1140169403 1140169404
sylvain> 1140169405 1140169406 1140169407 1140169416 1140169417
sylvain> 1140169418 1140169419 1140169420 1140169421 1140169422
sylvain> 1140169423
sylvain> For information the mdadm examine return : cat mdadm-exam.txt
sylvain> /dev/sdb: MBR Magic : aa55 Partition[0] : 1953523120 sectors
sylvain> at 2048 (type fd) /dev/sdc: MBR Magic : aa55 Partition[0] :
sylvain> 1953520002 sectors at 63 (type fd) /dev/sdd: MBR Magic : aa55
sylvain> Partition[0] : 1953520002 sectors at 63 (type fd) /dev/sde:
sylvain> MBR Magic : aa55 Partition[0] : 4294965247 sectors at 2048
sylvain> (type fd) I have 2 way to solve the issue. The first, is to
sylvain> have special command to pass bad sector during re-assemble as
sylvain> "mdadm --assemble --force /dev/md2 /dev/sd[bcde]1" The second
sylvain> is change the disk sde with the old good one, but some datas
sylvain> have been changed on the raid since i have remove it. But
sylvain> these datas are not important. It's only logs and history
sylvain> activity. What can i do to recover a maximum datas without
sylvain> too many risk? Thank's in advance Best Regards
sylvain> ---------------------------------- Sylvain Depuille (in
sylvain> trouble) sylvain.depuille@laposte.net -- To unsubscribe from
sylvain> this list: send the line "unsubscribe linux-raid" in the body
sylvain> of a message to majordomo@vger.kernel.org More majordomo info
sylvain> at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [SOLVED] Re: On RAID5 read error during syncing - array .A.A
From: Emery Guevremont @ 2014-12-29 19:40 UTC (permalink / raw)
To: Emery Guevremont, linux-raid
Hello,
I just wanted to document in this thread the commands and steps I
followed with Robin's help after having a second drive fail during the
synching of a RAID5 array.
- First step I did was take a picture of my error on screen. And save
a copy of the output of
mdadm -E /dev/sd[abcd]3
and
mdadm --detail --scan /dev/md0.
and
smartctl -a /dev/sda #for each drives since it was nice to have the
serial numbers match up with each UUID device mdadm -E gave.
- Second step I did was to boot with a liveusb and use ddrescue to
clone 3 of my 4 drives. The first drive that failed was totally
malfunctioning, but I was able to ddrescue my 3 other drives. I booted
with ubuntu's rescue remix live usb distro and ddrescued each drive
one at a time with a new drives I purchased.
ddrescue -f -n /dev/sda /dev/sdb /mnt/sdc2/ddrescue.sda.log
ddrescue -f -r3 /dev/sda /dev/sdb /mnt/sdc2/ddrescue.sda.log
- After the cloning process was done, I booted again with only the new
cloned drives. I ran
mdadm -E /dev/sd[abc]3 | grep "UUID Dev"
and compared that output to the one I took original when everything
first failed. This was important since the ordering of my drives did
not match anymore. After figuring out the correct order I ran thing
command that Robin helped me sort out.
mdadm -v --create --assume-clean --level=5 --chunk=512
--size=1952795136 --raid-devices=4 /dev/md0 /dev/sdd3 /dev/sdb3
/dev/sdc3 missing
- At this point I was able to run
fsck -n /dev/md0
which told me my ext4 partition was clean.
From this point on you have 2 options. mount in read-only and backup,
or insert a new drive to add to your array. I went the safe way and
mounted in read-only:
mount -o ro,noload /dev/md0 /mnt
rsync -axvSH /mnt/ /mnt2/
- The final step for me was to add a new fourth drive to my degraded
raid5 array.
mdadm --add /dev/md0 /dev/sda3
A cat /proc/mdstat confirmed the resynching process.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox