* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-24 12:59 UTC (permalink / raw)
To: Neil Brown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <87io70wmst.fsf@notabene.neil.brown.name>
Neil Brown <neilb@suse.de> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>
>> Neil Brown <neilb@suse.de> writes:
>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>>
>>>> Hi Neil,
>>>>
>>>> I think we have some bad side effects with this patch:
>>>>
>>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>>> Author: NeilBrown <neilb@suse.com>
>>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>>
>>>> md/raid0: update queue parameter in a safer location.
>>>>
>>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>>> of queue parameters (e.g. max number of sectors per bio) is
>>>> done in the wrong place.
>>>> It should be part of ->run, but it is actually part of ->takeover.
>>>> This means it happens before level_store() calls:
>>>>
>>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>>
>>>> Running the '03r0assem' test suite fills my kernel log with output like
>>>> below. Yi Zhang also had issues where writes failed too.
>>>>
>>>> robably something we need to resolve for 4.2-final or revert the
>>>> offending patch.
>>>>
>>>> Cheers,
>>>> Jes
>>>>
>>>> md: bind<loop0>
>>>> md: bind<loop1>
>>>> md: bind<loop2>
>>>> md/raid0:md2: md_size is 116736 sectors.
>>>> md: RAID0 configuration for md2 - 1 zone
>>>> md: zone0=[loop0/loop1/loop2]
>>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>>
>>>> md2: detected capacity change from 0 to 59768832
>>>> bio too big device loop0 (296 > 255)
>>>> bio too big device loop0 (272 > 255)
>>>
>>> 1/ Why do you blame that particular patch?
>>>
>>> 2/ Where is that error message coming from? I cannot find "bio too big"
>>> in the kernel (except in a comment).
>>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>>> arbitrarily sized bios")
>>> removed the only instance of the error message that I know of.
>>>
>>> Which kernel exactly are you testing?
>>
>> I blame it because of bisect - I revert that patch and the issue goes
>> away.
>>
>> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
>> see the bio too large. I revert it and it goes away.
>
> Well that's pretty convincing - thanks.
> And as you say - it is tagged for -stable so really needs to be fixed.
>
> Stares at the code again. And again.
>
> Ahhh. that patch moved the
> blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
> to after
> disk_stack_limits(...);
>
> That is wrong.
>
> Could you confirm that this fixes your test?
I refuse to do that! .... since Xiao beat me to it! Thanks!
I was half way bisecting my way through it last night. For some reason
the problem was reproducible in 4.2 if I applied the offending patch,
but not in 4.3-rc2.
Any chance you'll push these to your git tree in the near future?
Thanks!
Jes
^ permalink raw reply
* RAID5 Recovery From NAS With 4x3TB SATA Disks
From: mike @ 2015-09-24 12:41 UTC (permalink / raw)
To: linux-raid
I am attempting to recover some data from a RAID-5 NAS enclosure on my PC.
A number of things were already done wrong, but I am hopeful to recover at
least some of the data from the array.
The array was created on a Synology NAS enclosure consisting of 4 x 3TB
drives in a RAID-5 configuration. I gather everything was working well
until a power surge or brownout which caused one drive to die completely
(disk #0) and was likely the cause of some read errors on the other disks.
When the owner of the array noticed the disk failure, the wrong disk was
replaced to cause a rebuild! Disk #2 was replaced instead of disk #0, so
the NAS would never have succeeded at a rebuild operation. At this point
my friend came to me for assistance and I began pulling the drives from
the NAS into my PC running Ubuntu 14.04 (amd64) to attempt to restore at
least some of the files.
I cannot get disk #0 to spin at all. Disk #1 and disk #3 appear to
automatically assemble into the array in the correct slots.
Disk #2 appears to mdadm as "disk #32770".
Here is the output of `mdadm --examine` on each of the RAID partitions,
and the contents of /proc/mdstat when the drives are spun up inside my PC:
/dev/sdb5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d06e7cf6:05704b33:ea53fc30:0c087b55
Name : DiskStation:2
Creation Time : Thu Apr 3 13:41:58 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
Array Size : 8776594944 (8370.01 GiB 8987.23 GB)
Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 46277a0d:f75e3612:911d4e5f:50b46678
Update Time : Fri May 29 08:20:37 2015
Checksum : cd5c1a5f - correct
Events : 1908912
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 32770
Array State : .A.A ('A' == active, '.' == missing)
/dev/sdc5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d06e7cf6:05704b33:ea53fc30:0c087b55
Name : DiskStation:2
Creation Time : Thu Apr 3 13:41:58 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
Array Size : 8776594944 (8370.01 GiB 8987.23 GB)
Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa5bf9cd:629d2a75:7e3d9ab3:6189e4f9
Update Time : Sun May 31 11:05:20 2015
Checksum : c69bcc20 - correct
Events : 1908922
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AA.A ('A' == active, '.' == missing)
/dev/sdd5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : d06e7cf6:05704b33:ea53fc30:0c087b55
Name : DiskStation:2
Creation Time : Thu Apr 3 13:41:58 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
Array Size : 8776594944 (8370.01 GiB 8987.23 GB)
Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 0da09e5b:b3557017:dec76908:02543203
Update Time : Sun May 31 11:05:20 2015
Checksum : 54a41d85 - correct
Events : 1908922
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : AA.A ('A' == active, '.' == missing)
/proc/mdstat:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md2 : inactive sdb5[2](S) sdc5[1](S) sdd5[3](S)
8776595520 blocks super 1.2
I've read a bunch of forum posts and articles detailing a number of
techniques to try and recover some of the data. I gather from these
Synology instructions that once the array is assembled I must use
`vgchange` to register a Volume Group:
https://www.synology.com/en-us/knowledgebase/faq/579
This wiki page gave me the idea of using `mdadm --create --assume-clean
...` to bring the array to a state where I might be able to see some of
its contents: https://raid.wiki.kernel.org/index.php/RAID_Recovery
I'm also familiar with concerns over using large disks in a RAID-5
configuration and would like to be clear that this was not my idea and I
am simply trying to help a friend retrieve at least some of the files from
the array - I do not expect complete recovery will be possible:
http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/
I attempted to bring this array up with this command:
mdadm --create --assume-clean --level=5 --raid-devices=4 --chunk=64
/dev/md2 missing /dev/sdc5 /dev/sdb5 /dev/sdd5
This appeared to bring up a new array with 0 Events. I've restored my
backups and can try again from the best state I have to work with.
All disks experienced some read errors while creating the backups, and I
realize complete recovery will be impossible, but if I could retrieve any
data we would be pleased with that.
Any help greatly appreciated.
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Alexander Afonyashin @ 2015-09-24 9:17 UTC (permalink / raw)
To: Mathias Burén; +Cc: Linux-RAID
In-Reply-To: <CADNH=7E796sv+A4PDb2n=EJomAR8KPRoGt-=+52BYfrELBFZDg@mail.gmail.com>
Hi Matias,
I wonder why Ubuntu detects raid6 parts as /dev/sdX1 (and counts only
5 disks) while Arch detects /dev/sdX (and counts 6 disks)? Try to
assemble raid6 on Arch manually again by issuing:
mdadm -S /dev/md0
mdadm -A /dev/md0 /dev/sdb /dev/sda /dev/sde /dev/sdd /dev/sdc -
exactly as shown in mdadm's output on Ubuntu
P.S. Still confused why Arch doesn't recognize /dev/sda1 etc.
Regards,
Alexander
On Wed, Sep 23, 2015 at 9:35 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
> Hi Alexander,
>
> I didn't try to grow the array or anything, just swapped out the OS.
> When I boot back into Ubuntu it assembles fine.
>
> This is from within Ubuntu (I did a mdadm --assemble --scan using
> mdadm - v3.3.2-7-g21dc471 - 03th November 2014)
>
> mdadm -D on RAID6
> /dev/md0:
> Version : 1.2
> Creation Time : Thu Nov 20 23:52:58 2014
> Raid Level : raid6
> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
> Used Dev Size : 1952015360 (1861.59 GiB 1998.86 GB)
> Raid Devices : 5
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Sun Sep 20 19:47:50 2015
> State : clean
> Active Devices : 5
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Name : ion:0 (local to host ion)
> UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
> Events : 59736
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 1 8 33 1 active sync /dev/sdc1
> 2 8 49 2 active sync /dev/sdd1
> 3 8 65 3 active sync /dev/sde1
> 4 8 1 4 active sync /dev/sda1
>
>
> mdadm -D on RAID0
> /dev/md1:
> Version : 1.2
> Creation Time : Thu Nov 20 23:53:48 2014
> Raid Level : raid0
> Array Size : 4098048 (3.91 GiB 4.20 GB)
> Raid Devices : 3
> Total Devices : 3
> Persistence : Superblock is persistent
>
> Update Time : Thu Nov 20 23:53:48 2014
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 0
> Spare Devices : 0
>
> Chunk Size : 512K
>
> Name : ion:1 (local to host ion)
> UUID : a20b70d4:7ee17e3f:abab74f8:dadb8cd8
> Events : 0
>
> Number Major Minor RaidDevice State
> 0 8 34 0 active sync /dev/sdc2
> 1 8 50 1 active sync /dev/sdd2
> 2 8 66 2 active sync /dev/sde2
>
>
>
> lsdrv from within Ubuntu
> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
> Logic MegaRAID SAS 1068
> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> │ └sda1 1.82t [8:1] MD raid6 (4/5) (w/ sdb1,sdc1,sdd1,sde1) in_sync
> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> │ └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
> {4cae433f:a40afcf5:f9aba91d:d8217b69}
> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> └sdb1 1.82t [8:17] MD raid6 (0/5) (w/ sda1,sdc1,sdd1,sde1) in_sync
> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
> {4cae433f:a40afcf5:f9aba91d:d8217b69}
> ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
> ├scsi 1:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
> │└sdf 55.90g [8:80] Partitioned (dos)
> │ ├sdf1 243.00m [8:81] ext2 {b45c13c8-4246-43ee-ac2c-f9bb5de099f8}
> │ │└Mounted as /dev/sdf1 @ /boot
> │ ├sdf2 1.00k [8:82] Partitioned (dos)
> │ └sdf5 55.66g [8:85] PV LVM2_member 55.66g used, 0 free
> {zZ5Dgy-E7v8-Y1Mi-EuqG-uXUb-DseW-s7goxG}
> │ └VG ion 55.66g 0 free {XSaCyP-phLN-m4IH-2cJ7-1XXw-b3uh-qWBl8N}
> │ ├dm-0 52.41g [252:0] LV root ext4 {63342e0d-1b4f-4475-9835-d3a2a6610e8f}
> │ │└Mounted as /dev/mapper/ion-root @ /
> │ └dm-1 3.25g [252:1] LV swap_1 Empty/Unknown
> │ └dm-2 3.25g [252:2] swap {0c81625d-ee9e-4a7a-9613-410c1cf53e72}
> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
> │└sdc 1.82t [8:32] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> │ ├sdc1 1.82t [8:33] MD raid6 (1/5) (w/ sda1,sdb1,sdd1,sde1) in_sync
> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
> {4cae433f:a40afcf5:f9aba91d:d8217b69}
> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
> │ └sdc2 1.30g [8:34] MD raid0 (0/3) (w/ sdd2,sde2) in_sync 'ion:1'
> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
> ├scsi 3:x:x:x [Empty]
> ├scsi 4:x:x:x [Empty]
> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> │ ├sdd1 1.82t [8:49] MD raid6 (2/5) (w/ sda1,sdb1,sdc1,sde1) in_sync
> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
> {4cae433f:a40afcf5:f9aba91d:d8217b69}
> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
> │ └sdd2 1.30g [8:50] MD raid0 (1/3) (w/ sdc2,sde2) in_sync 'ion:1'
> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
> └sde 2.73t [8:64] Partitioned (dos)
> ├sde1 1.82t [8:65] MD raid6 (3/5) (w/ sda1,sdb1,sdc1,sdd1) in_sync
> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
> {4cae433f:a40afcf5:f9aba91d:d8217b69}
> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
> ├sde2 1.30g [8:66] MD raid0 (2/3) (w/ sdc2,sdd2) in_sync 'ion:1'
> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
> │└md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
> └sde3 184.98g [8:67] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
> └Mounted as /dev/sde3 @ /media/198GB
> Other Block Devices
> ├loop0 0.00k [7:0] Empty/Unknown
> ├loop1 0.00k [7:1] Empty/Unknown
> ├loop2 0.00k [7:2] Empty/Unknown
> ├loop3 0.00k [7:3] Empty/Unknown
> ├loop4 0.00k [7:4] Empty/Unknown
> ├loop5 0.00k [7:5] Empty/Unknown
> ├loop6 0.00k [7:6] Empty/Unknown
> ├loop7 0.00k [7:7] Empty/Unknown
> ├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
> │ Empty/Unknown
> ├ram0 64.00m [1:0] Empty/Unknown
> ├ram1 64.00m [1:1] Empty/Unknown
> ├ram2 64.00m [1:2] Empty/Unknown
> ├ram3 64.00m [1:3] Empty/Unknown
> ├ram4 64.00m [1:4] Empty/Unknown
> ├ram5 64.00m [1:5] Empty/Unknown
> ├ram6 64.00m [1:6] Empty/Unknown
> ├ram7 64.00m [1:7] Empty/Unknown
> ├ram8 64.00m [1:8] Empty/Unknown
> ├ram9 64.00m [1:9] Empty/Unknown
> ├ram10 64.00m [1:10] Empty/Unknown
> ├ram11 64.00m [1:11] Empty/Unknown
> ├ram12 64.00m [1:12] Empty/Unknown
> ├ram13 64.00m [1:13] Empty/Unknown
> ├ram14 64.00m [1:14] Empty/Unknown
> └ram15 64.00m [1:15] Empty/Unknown
>
>
> /proc/mdstat on Ubuntu
> Personalities : [raid6] [raid5] [raid4] [raid0]
> md0 : active raid6 sdb1[0] sda1[4] sde1[3] sdd1[2] sdc1[1]
> 5856046080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
> bitmap: 0/15 pages [0KB], 65536KB chunk
>
> md1 : active raid0 sdc2[0] sde2[2] sdd2[1]
> 4098048 blocks super 1.2 512k chunks
>
> Regards,
> Mathias
>
> On 23 September 2015 at 07:24, Alexander Afonyashin
> <a.afonyashin@madnet-team.ru> wrote:
>> Hi,
>>
>> I wonder why metainfo from /dev/sdf1 thinks that there're only 5
>> devices in raid6. Did you try to grow your array recently?
>> Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
>> Can you assemble the array with 4 disks: /dev/sd[a-d]?
>>
>> P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
>> from another md-array (check Array UUID field). You may need to zero
>> metadata on /dev/sdf1 and re-add in to existing raid6 array.
>>
>> Regards,
>> Alexander
>>
>> On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>> Hi (please reply-all)
>>>
>>> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
>>> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
>>> 2015, kernel 4.1.6
>>>
>>> Here is the mdadm --examine for each drive:
>>>
>>>
>>>
>>> [root@ion ~]# mdadm --examine /dev/sda
>>> /dev/sda:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>> Name : ion:md0 (local to host ion)
>>> Creation Time : Tue Feb 5 17:33:27 2013
>>> Raid Level : raid6
>>> Raid Devices : 6
>>>
>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>> State : clean
>>> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>>>
>>> Update Time : Tue Nov 4 21:43:49 2014
>>> Checksum : 528563ee - correct
>>> Events : 97557
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 3
>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>> [root@ion ~]# mdadm --examine /dev/sdb
>>> /dev/sdb:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>> Name : ion:md0 (local to host ion)
>>> Creation Time : Tue Feb 5 17:33:27 2013
>>> Raid Level : raid6
>>> Raid Devices : 6
>>>
>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>> State : clean
>>> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>>>
>>> Update Time : Tue Nov 4 21:43:49 2014
>>> Checksum : 49d756ca - correct
>>> Events : 97557
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 0
>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>>
>>> [root@ion ~]# mdadm --examine /dev/sdd
>>> /dev/sdd:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>> Name : ion:md0 (local to host ion)
>>> Creation Time : Tue Feb 5 17:33:27 2013
>>> Raid Level : raid6
>>> Raid Devices : 6
>>>
>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=262064 sectors, after=1200 sectors
>>> State : clean
>>> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>>>
>>> Update Time : Tue Nov 4 21:43:49 2014
>>> Checksum : 50c95b7c - correct
>>> Events : 97557
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 5
>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>> [root@ion ~]# mdadm --examine /dev/sde
>>> /dev/sde:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>> Name : ion:md0 (local to host ion)
>>> Creation Time : Tue Feb 5 17:33:27 2013
>>> Raid Level : raid6
>>> Raid Devices : 6
>>>
>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=262064 sectors, after=1200 sectors
>>> State : clean
>>> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>>>
>>> Update Time : Tue Nov 4 21:43:49 2014
>>> Checksum : 71b191d6 - correct
>>> Events : 97557
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 1
>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>> [root@ion ~]# mdadm --examine /dev/sdf1
>>> /dev/sdf1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x1
>>> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>> Name : ion:0 (local to host ion)
>>> Creation Time : Thu Nov 20 23:52:58 2014
>>> Raid Level : raid6
>>> Raid Devices : 5
>>>
>>> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>> Data Offset : 262144 sectors
>>> Super Offset : 8 sectors
>>> Unused Space : before=262056 sectors, after=0 sectors
>>> State : clean
>>> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>>>
>>> Internal Bitmap : 8 sectors from superblock
>>> Update Time : Sun Sep 20 19:47:50 2015
>>> Bad Block Log : 512 entries available at offset 72 sectors
>>> Checksum : b1f6f725 - correct
>>> Events : 59736
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 3
>>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>
>>>
>>>
>>>
>>>
>>> Here is ldrv:
>>>
>>> [root@ion ~]# python2 lsdrv
>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>> Logic MegaRAID SAS 1068
>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
>>> │└sdc 149.05g [8:32] Partitioned (dos)
>>> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
>>> │ │└Mounted as /dev/sdc1 @ /boot
>>> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
>>> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
>>> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
>>> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
>>> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
>>> │ │└Mounted as /dev/mapper/ArchVG-root @ /
>>> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> ├scsi 3:x:x:x [Empty]
>>> ├scsi 4:x:x:x [Empty]
>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>> └sdf 2.73t [8:80] Partitioned (dos)
>>> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
>>> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>>
>>>
>>> If I try:
>>>
>>> [root@ion ~]# mdadm --assemble --scan
>>> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
>>> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>>>
>>> From dmesg:
>>>
>>> [ 1227.671344] sda: sda1
>>> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
>>> [ 1227.680414] md: md_import_device returned -22
>>> [ 1227.680462] md: md0 stopped.
>>> [ 1227.707969] sdb: sdb1
>>> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
>>> [ 1227.718611] md: md_import_device returned -22
>>> [ 1227.718631] md: md0 stopped.
>>> [ 1286.334542] md: md0 stopped.
>>> [ 1286.338250] md: bind<sdf1>
>>> [ 1286.338390] md: md0 stopped.
>>> [ 1286.338400] md: unbind<sdf1>
>>> [ 1286.348350] md: export_rdev(sdf1)
>>> [ 1286.372268] md: bind<sdf1>
>>> [ 1286.373390] md: md1 stopped.
>>> [ 1286.373936] md: bind<sdf2>
>>> [ 1286.373977] md: md1 stopped.
>>> [ 1286.373983] md: unbind<sdf2>
>>> [ 1286.388061] md: export_rdev(sdf2)
>>> [ 1286.405140] md: bind<sdf2>
>>>
>>> [root@ion ~]# cat /proc/mdstat
>>> Personalities :
>>> md1 : inactive sdf2[2](S)
>>> 1366104 blocks super 1.2
>>>
>>> md0 : inactive sdf1[3](S)
>>> 1952015360 blocks super 1.2
>>>
>>> unused devices: <none>
>>>
>>>
>>> If I try manually (I'm not sure of the order though):
>>>
>>> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
>>> /dev/sdb /dev/sdd /dev/sde
>>> mdadm: looking for devices for /dev/md0
>>> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
>>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
>>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
>>> mdadm: added /dev/sde to /dev/md0 as 1
>>> mdadm: no uptodate device for slot 2 of /dev/md0
>>> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
>>> mdadm: no uptodate device for slot 4 of /dev/md0
>>> mdadm: added /dev/sdd to /dev/md0 as 5
>>> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
>>> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>>>
>>>
>>> Any idea where I should start?
>>>
>>> Thanks
>>> Mathias
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Xiao Ni @ 2015-09-24 8:48 UTC (permalink / raw)
To: Neil Brown; +Cc: Jes Sorensen, linux-raid, yizhan
In-Reply-To: <87io70wmst.fsf@notabene.neil.brown.name>
----- Original Message -----
> From: "Neil Brown" <neilb@suse.de>
> To: "Jes Sorensen" <Jes.Sorensen@redhat.com>
> Cc: "Xiao Ni" <xni@redhat.com>, "linux-raid" <linux-raid@vger.kernel.org>, yizhan@redhat.com
> Sent: Thursday, September 24, 2015 10:53:06 AM
> Subject: Re: Bad raid0 bio too large problem
>
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>
> > Neil Brown <neilb@suse.de> writes:
> >> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >>
> >>> Hi Neil,
> >>>
> >>> I think we have some bad side effects with this patch:
> >>>
> >>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
> >>> Author: NeilBrown <neilb@suse.com>
> >>> Date: Mon Aug 3 13:11:47 2015 +1000
> >>>
> >>> md/raid0: update queue parameter in a safer location.
> >>>
> >>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
> >>> of queue parameters (e.g. max number of sectors per bio) is
> >>> done in the wrong place.
> >>> It should be part of ->run, but it is actually part of ->takeover.
> >>> This means it happens before level_store() calls:
> >>>
> >>> blk_set_stacking_limits(&mddev->queue->limits);
> >>>
> >>> Running the '03r0assem' test suite fills my kernel log with output like
> >>> below. Yi Zhang also had issues where writes failed too.
> >>>
> >>> robably something we need to resolve for 4.2-final or revert the
> >>> offending patch.
> >>>
> >>> Cheers,
> >>> Jes
> >>>
> >>> md: bind<loop0>
> >>> md: bind<loop1>
> >>> md: bind<loop2>
> >>> md/raid0:md2: md_size is 116736 sectors.
> >>> md: RAID0 configuration for md2 - 1 zone
> >>> md: zone0=[loop0/loop1/loop2]
> >>> zone-offset= 0KB, device-offset= 0KB, size=
> >>> 58368KB
> >>>
> >>> md2: detected capacity change from 0 to 59768832
> >>> bio too big device loop0 (296 > 255)
> >>> bio too big device loop0 (272 > 255)
> >>
> >> 1/ Why do you blame that particular patch?
> >>
> >> 2/ Where is that error message coming from? I cannot find "bio too big"
> >> in the kernel (except in a comment).
> >> Commit: 54efd50bfd87 ("block: make generic_make_request handle
> >> arbitrarily sized bios")
> >> removed the only instance of the error message that I know of.
> >>
> >> Which kernel exactly are you testing?
> >
> > I blame it because of bisect - I revert that patch and the issue goes
> > away.
> >
> > I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
> > see the bio too large. I revert it and it goes away.
>
> Well that's pretty convincing - thanks.
> And as you say - it is tagged for -stable so really needs to be fixed.
>
> Stares at the code again. And again.
>
> Ahhh. that patch moved the
> blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
> to after
> disk_stack_limits(...);
>
> That is wrong.
>
> Could you confirm that this fixes your test?
>
> Thanks,
> NeilBrown
>
> diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
> index 4a13c3cb940b..0875e5e7e09a 100644
> --- a/drivers/md/raid0.c
> +++ b/drivers/md/raid0.c
> @@ -431,12 +431,6 @@ static int raid0_run(struct mddev *mddev)
> struct md_rdev *rdev;
> bool discard_supported = false;
>
> - rdev_for_each(rdev, mddev) {
> - disk_stack_limits(mddev->gendisk, rdev->bdev,
> - rdev->data_offset << 9);
> - if (blk_queue_discard(bdev_get_queue(rdev->bdev)))
> - discard_supported = true;
> - }
> blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
> blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
> blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
> @@ -445,6 +439,12 @@ static int raid0_run(struct mddev *mddev)
> blk_queue_io_opt(mddev->queue,
> (mddev->chunk_sectors << 9) * mddev->raid_disks);
>
> + rdev_for_each(rdev, mddev) {
> + disk_stack_limits(mddev->gendisk, rdev->bdev,
> + rdev->data_offset << 9);
> + if (blk_queue_discard(bdev_get_queue(rdev->bdev)))
> + discard_supported = true;
> + }
> if (!discard_supported)
> queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
> else
>
Hi Neil, Jes
The problem is fixed with this patch.
Best Regards
Xiao
^ permalink raw reply
* Re: [PATCH 2/2] raid5: update analysis state for failed stripe
From: Neil Brown @ 2015-09-24 5:26 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <20150923063425.GA2813864@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 3758 bytes --]
Shaohua Li <shli@fb.com> writes:
> On Wed, Sep 23, 2015 at 04:21:58PM +1000, Neil Brown wrote:
>> Shaohua Li <shli@fb.com> writes:
>>
>> > handle_failed_stripe() makes the stripe fail, eg, all IO will return
>> > with a failure, but it doesn't update stripe_head_state. Later
>> > handle_stripe() has special handling for raid6 for handle_stripe_fill().
>> > That check before handle_stripe_fill() doesn't skip the failed stripe
>> > and we get a kernel crash in need_this_block. This patch clear the
>> > analysis state to make sure no functions wrongly called after
>> > handle_failed_stripe()
>> >
>> > Signed-off-by: Shaohua Li <shli@fb.com>
>> > ---
>> > drivers/md/raid5.c | 4 ++++
>> > 1 file changed, 4 insertions(+)
>> >
>> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> > index 394cdf8..8e4fb89a 100644
>> > --- a/drivers/md/raid5.c
>> > +++ b/drivers/md/raid5.c
>> > @@ -3155,6 +3155,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
>> > spin_unlock_irq(&sh->stripe_lock);
>> > if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
>> > wake_up(&conf->wait_for_overlap);
>> > + if (bi)
>> > + s->to_read--;
>> > while (bi && bi->bi_iter.bi_sector <
>> > sh->dev[i].sector + STRIPE_SECTORS) {
>> > struct bio *nextbi =
>> > @@ -3173,6 +3175,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
>> > */
>> > clear_bit(R5_LOCKED, &sh->dev[i].flags);
>> > }
>> > + s->to_write = 0;
>> > + s->written = 0;
>> >
>> > if (test_and_clear_bit(STRIPE_FULL_WRITE, &sh->state))
>> > if (atomic_dec_and_test(&conf->pending_full_writes))
>> > --
>> > 1.8.1
>>
>> Again, this probably is a sensible fix, but I would like to be certain.
>> Where exactly in need_this_block does the kernel crash? I cannot see
>> anything that could cause an invalid address....
>
>
>>>for (i = 0; i < s->failed; i++) {
>>> if (fdev[i]->towrite &&
> the fdev[i]->towrite. because s->failed >=2 (it's 3 in my case), while
> the array size is 2.
>
> Thanks,
> Shaohua
Ahh, of course.
In that case I think I'd like to limit the for loop as well.
So I've applied your patch and this one as well.
Thanks,
NeilBrown
From 76e308d70b204ff0af0028458caabfeacac4541a Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.com>
Date: Thu, 24 Sep 2015 15:25:36 +1000
Subject: [PATCH] md/raid5: don't index beyond end of array in
need_this_block().
When need_this_block probably shouldn't be called when there
are more than 2 failed devices, we really don't want it to try
indexing beyond the end of the failed_num[] of fdev[] arrays.
So limit the loops to at most 2 iterations.
Reported-by: Shaohua Li <shli@fb.com>
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 903d8a2b7b07..0f49ce411c9a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3304,7 +3304,7 @@ static int need_this_block(struct stripe_head *sh, struct stripe_head_state *s,
*/
return 0;
- for (i = 0; i < s->failed; i++) {
+ for (i = 0; i < s->failed && i < 2; i++) {
if (fdev[i]->towrite &&
!test_bit(R5_UPTODATE, &fdev[i]->flags) &&
!test_bit(R5_OVERWRITE, &fdev[i]->flags))
@@ -3328,7 +3328,7 @@ static int need_this_block(struct stripe_head *sh, struct stripe_head_state *s,
sh->sector < sh->raid_conf->mddev->recovery_cp)
/* reconstruct-write isn't being forced */
return 0;
- for (i = 0; i < s->failed; i++) {
+ for (i = 0; i < s->failed && i < 2; i++) {
if (s->failed_num[i] != sh->pd_idx &&
s->failed_num[i] != sh->qd_idx &&
!test_bit(R5_UPTODATE, &fdev[i]->flags) &&
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: [PATCH 1/2] md: clear CHANGE_PENDING in readonly array
From: Neil Brown @ 2015-09-24 4:03 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <20150923062302.GA2780132@devbig084.prn1.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 3855 bytes --]
Shaohua Li <shli@fb.com> writes:
> On Wed, Sep 23, 2015 at 04:05:33PM +1000, Neil Brown wrote:
>> Shaohua Li <shli@fb.com> writes:
>>
>> > If faulty disks of an array are more than allowed degraded number, the
>> > array enters error handling. It will be marked as read-only with
>> > MD_CHANGE_PENDING/RECOVERY_NEEDED set. But currently recovery doesn't
>> > clear CHANGE_PENDING bit for read-only array. If MD_CHANGE_PENDING is
>> > set for a raid5 array, all returned IO will be hold on a list till the
>> > bit is clear. But recovery nevery clears this bit, the IO is always in
>> > pending state and nevery finish. This has bad effects like upper layer
>> > can't get an IO error and the array can't be stopped.
>> >
>> > Signed-off-by: Shaohua Li <shli@fb.com>
>> > ---
>> > drivers/md/md.c | 1 +
>> > 1 file changed, 1 insertion(+)
>> >
>> > diff --git a/drivers/md/md.c b/drivers/md/md.c
>> > index 95824fb..c596b73 100644
>> > --- a/drivers/md/md.c
>> > +++ b/drivers/md/md.c
>> > @@ -8209,6 +8209,7 @@ void md_check_recovery(struct mddev *mddev)
>> > md_reap_sync_thread(mddev);
>> > clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
>> > clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
>> > + clear_bit(MD_CHANGE_PENDING, &mddev->flags);
>> > goto unlock;
>> > }
>> >
>> > --
>> > 1.8.1
>>
>> Hi,
>> I can see that clearing MD_CHANGE_PENDING there is probably correct -
>> bug introduced by
>> Commit: c3cce6cda162 ("md/raid5: ensure device failure recorded before write request returns.")
>>
>> However I don't understand your reasoning. You say that the array is
>> marked as read-only, but I don't see how that would happen. What
>> causes the array to be marked "read-only"?
>
> It's set read-only by mdadm. I didn't look carefully, but looks there is
> disk failure event, mdadm is invoked automatically by some background
> daemon. It's a ubuntu distribution.
Thanks.
This raises a couple of questions.
1/ What should md_set_readonly do if it finds that MD_CHANGE_PENDING is
set?
Maybe it should wait for md_check_recovery to get run which should
clear the bit, after probably writing out the metadata.
2/ Why didn't md_check_recovery already do that before mdadm had a
chance to set the array read-only?
I guess that is just a timing thing. md_check_recovery could be
delayed, and mdadm could get called by udev rather quickly.
I think I'll get md_set_readonly to
wait_event(mddev->sb_wait,
!test_bit(MD_CHANGE_PENDING, &mddev->flags));
because I think that is the right thing to do. But if the array is
already read-only that won't help, so I'll still need you patch.
Would you be able to test that the following patch (without your patch)
also fixes the symptom?
Thanks.
NeilBrown
From a0a21b57523c41e87e0a50333b15f6c7ca02d714 Mon Sep 17 00:00:00 2001
From: NeilBrown <neilb@suse.com>
Date: Thu, 24 Sep 2015 14:00:51 +1000
Subject: [PATCH] md: wait for pending superblock updates before switching to
read-only
If a superblock update is pending, wait for it to complete before
letting md_set_readonly() switch to readonly.
Otherwise we might lose important information about a device having
failed.
Reported-by: Shaohua Li <shli@fb.com>
Signed-off-by: NeilBrown <neilb@suse.com>
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 23651bf8dd80..9882f70f45e1 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -5432,6 +5432,8 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
mddev_unlock(mddev);
wait_event(resync_wait, !test_bit(MD_RECOVERY_RUNNING,
&mddev->recovery));
+ wait_event(mddev->sb_wait,
+ !test_bit(MD_CHANGE_PENDING, &mddev->flags));
mddev_lock_nointr(mddev);
mutex_lock(&mddev->open_mutex);
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: Bad raid0 bio too large problem
From: Neil Brown @ 2015-09-24 2:53 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjvbb12xl8.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3671 bytes --]
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Neil Brown <neilb@suse.de> writes:
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>
>>> Hi Neil,
>>>
>>> I think we have some bad side effects with this patch:
>>>
>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>> Author: NeilBrown <neilb@suse.com>
>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>
>>> md/raid0: update queue parameter in a safer location.
>>>
>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>> of queue parameters (e.g. max number of sectors per bio) is
>>> done in the wrong place.
>>> It should be part of ->run, but it is actually part of ->takeover.
>>> This means it happens before level_store() calls:
>>>
>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>
>>> Running the '03r0assem' test suite fills my kernel log with output like
>>> below. Yi Zhang also had issues where writes failed too.
>>>
>>> robably something we need to resolve for 4.2-final or revert the
>>> offending patch.
>>>
>>> Cheers,
>>> Jes
>>>
>>> md: bind<loop0>
>>> md: bind<loop1>
>>> md: bind<loop2>
>>> md/raid0:md2: md_size is 116736 sectors.
>>> md: RAID0 configuration for md2 - 1 zone
>>> md: zone0=[loop0/loop1/loop2]
>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>
>>> md2: detected capacity change from 0 to 59768832
>>> bio too big device loop0 (296 > 255)
>>> bio too big device loop0 (272 > 255)
>>
>> 1/ Why do you blame that particular patch?
>>
>> 2/ Where is that error message coming from? I cannot find "bio too big"
>> in the kernel (except in a comment).
>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>> arbitrarily sized bios")
>> removed the only instance of the error message that I know of.
>>
>> Which kernel exactly are you testing?
>
> I blame it because of bisect - I revert that patch and the issue goes
> away.
>
> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
> see the bio too large. I revert it and it goes away.
Well that's pretty convincing - thanks.
And as you say - it is tagged for -stable so really needs to be fixed.
Stares at the code again. And again.
Ahhh. that patch moved the
blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
to after
disk_stack_limits(...);
That is wrong.
Could you confirm that this fixes your test?
Thanks,
NeilBrown
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 4a13c3cb940b..0875e5e7e09a 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -431,12 +431,6 @@ static int raid0_run(struct mddev *mddev)
struct md_rdev *rdev;
bool discard_supported = false;
- rdev_for_each(rdev, mddev) {
- disk_stack_limits(mddev->gendisk, rdev->bdev,
- rdev->data_offset << 9);
- if (blk_queue_discard(bdev_get_queue(rdev->bdev)))
- discard_supported = true;
- }
blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
@@ -445,6 +439,12 @@ static int raid0_run(struct mddev *mddev)
blk_queue_io_opt(mddev->queue,
(mddev->chunk_sectors << 9) * mddev->raid_disks);
+ rdev_for_each(rdev, mddev) {
+ disk_stack_limits(mddev->gendisk, rdev->bdev,
+ rdev->data_offset << 9);
+ if (blk_queue_discard(bdev_get_queue(rdev->bdev)))
+ discard_supported = true;
+ }
if (!discard_supported)
queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
else
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Mathias Burén @ 2015-09-23 18:35 UTC (permalink / raw)
To: Alexander Afonyashin; +Cc: Linux-RAID
In-Reply-To: <CAETWcfsw2M3JkNmE3M3nsr0_M4N86ta+kaRo62OgK+Ft0J5iiw@mail.gmail.com>
Hi Alexander,
I didn't try to grow the array or anything, just swapped out the OS.
When I boot back into Ubuntu it assembles fine.
This is from within Ubuntu (I did a mdadm --assemble --scan using
mdadm - v3.3.2-7-g21dc471 - 03th November 2014)
mdadm -D on RAID6
/dev/md0:
Version : 1.2
Creation Time : Thu Nov 20 23:52:58 2014
Raid Level : raid6
Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
Used Dev Size : 1952015360 (1861.59 GiB 1998.86 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Sep 20 19:47:50 2015
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ion:0 (local to host ion)
UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
Events : 59736
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
4 8 1 4 active sync /dev/sda1
mdadm -D on RAID0
/dev/md1:
Version : 1.2
Creation Time : Thu Nov 20 23:53:48 2014
Raid Level : raid0
Array Size : 4098048 (3.91 GiB 4.20 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Nov 20 23:53:48 2014
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Name : ion:1 (local to host ion)
UUID : a20b70d4:7ee17e3f:abab74f8:dadb8cd8
Events : 0
Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 66 2 active sync /dev/sde2
lsdrv from within Ubuntu
PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
Logic MegaRAID SAS 1068
├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
│└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
│ └sda1 1.82t [8:1] MD raid6 (4/5) (w/ sdb1,sdc1,sdd1,sde1) in_sync
'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
│ └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
{4cae433f:a40afcf5:f9aba91d:d8217b69}
│ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
└scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
└sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
└sdb1 1.82t [8:17] MD raid6 (0/5) (w/ sda1,sdc1,sdd1,sde1) in_sync
'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
{4cae433f:a40afcf5:f9aba91d:d8217b69}
ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
├scsi 1:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
│└sdf 55.90g [8:80] Partitioned (dos)
│ ├sdf1 243.00m [8:81] ext2 {b45c13c8-4246-43ee-ac2c-f9bb5de099f8}
│ │└Mounted as /dev/sdf1 @ /boot
│ ├sdf2 1.00k [8:82] Partitioned (dos)
│ └sdf5 55.66g [8:85] PV LVM2_member 55.66g used, 0 free
{zZ5Dgy-E7v8-Y1Mi-EuqG-uXUb-DseW-s7goxG}
│ └VG ion 55.66g 0 free {XSaCyP-phLN-m4IH-2cJ7-1XXw-b3uh-qWBl8N}
│ ├dm-0 52.41g [252:0] LV root ext4 {63342e0d-1b4f-4475-9835-d3a2a6610e8f}
│ │└Mounted as /dev/mapper/ion-root @ /
│ └dm-1 3.25g [252:1] LV swap_1 Empty/Unknown
│ └dm-2 3.25g [252:2] swap {0c81625d-ee9e-4a7a-9613-410c1cf53e72}
├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
│└sdc 1.82t [8:32] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
│ ├sdc1 1.82t [8:33] MD raid6 (1/5) (w/ sda1,sdb1,sdd1,sde1) in_sync
'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
│ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
{4cae433f:a40afcf5:f9aba91d:d8217b69}
│ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
│ └sdc2 1.30g [8:34] MD raid0 (0/3) (w/ sdd2,sde2) in_sync 'ion:1'
{a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
│ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
│ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
├scsi 3:x:x:x [Empty]
├scsi 4:x:x:x [Empty]
├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
│└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
│ ├sdd1 1.82t [8:49] MD raid6 (2/5) (w/ sda1,sdb1,sdc1,sde1) in_sync
'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
│ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
{4cae433f:a40afcf5:f9aba91d:d8217b69}
│ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
│ └sdd2 1.30g [8:50] MD raid0 (1/3) (w/ sdc2,sde2) in_sync 'ion:1'
{a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
│ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
│ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
└scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
└sde 2.73t [8:64] Partitioned (dos)
├sde1 1.82t [8:65] MD raid6 (3/5) (w/ sda1,sdb1,sdc1,sdd1) in_sync
'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
│└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
{4cae433f:a40afcf5:f9aba91d:d8217b69}
│ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
├sde2 1.30g [8:66] MD raid0 (2/3) (w/ sdc2,sdd2) in_sync 'ion:1'
{a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
│└md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
│ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
└sde3 184.98g [8:67] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
└Mounted as /dev/sde3 @ /media/198GB
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
├loop7 0.00k [7:7] Empty/Unknown
├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
│ Empty/Unknown
├ram0 64.00m [1:0] Empty/Unknown
├ram1 64.00m [1:1] Empty/Unknown
├ram2 64.00m [1:2] Empty/Unknown
├ram3 64.00m [1:3] Empty/Unknown
├ram4 64.00m [1:4] Empty/Unknown
├ram5 64.00m [1:5] Empty/Unknown
├ram6 64.00m [1:6] Empty/Unknown
├ram7 64.00m [1:7] Empty/Unknown
├ram8 64.00m [1:8] Empty/Unknown
├ram9 64.00m [1:9] Empty/Unknown
├ram10 64.00m [1:10] Empty/Unknown
├ram11 64.00m [1:11] Empty/Unknown
├ram12 64.00m [1:12] Empty/Unknown
├ram13 64.00m [1:13] Empty/Unknown
├ram14 64.00m [1:14] Empty/Unknown
└ram15 64.00m [1:15] Empty/Unknown
/proc/mdstat on Ubuntu
Personalities : [raid6] [raid5] [raid4] [raid0]
md0 : active raid6 sdb1[0] sda1[4] sde1[3] sdd1[2] sdc1[1]
5856046080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md1 : active raid0 sdc2[0] sde2[2] sdd2[1]
4098048 blocks super 1.2 512k chunks
Regards,
Mathias
On 23 September 2015 at 07:24, Alexander Afonyashin
<a.afonyashin@madnet-team.ru> wrote:
> Hi,
>
> I wonder why metainfo from /dev/sdf1 thinks that there're only 5
> devices in raid6. Did you try to grow your array recently?
> Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
> Can you assemble the array with 4 disks: /dev/sd[a-d]?
>
> P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
> from another md-array (check Array UUID field). You may need to zero
> metadata on /dev/sdf1 and re-add in to existing raid6 array.
>
> Regards,
> Alexander
>
> On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Hi (please reply-all)
>>
>> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
>> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
>> 2015, kernel 4.1.6
>>
>> Here is the mdadm --examine for each drive:
>>
>>
>>
>> [root@ion ~]# mdadm --examine /dev/sda
>> /dev/sda:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>> Name : ion:md0 (local to host ion)
>> Creation Time : Tue Feb 5 17:33:27 2013
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>> State : clean
>> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>>
>> Update Time : Tue Nov 4 21:43:49 2014
>> Checksum : 528563ee - correct
>> Events : 97557
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@ion ~]# mdadm --examine /dev/sdb
>> /dev/sdb:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>> Name : ion:md0 (local to host ion)
>> Creation Time : Tue Feb 5 17:33:27 2013
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>> State : clean
>> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>>
>> Update Time : Tue Nov 4 21:43:49 2014
>> Checksum : 49d756ca - correct
>> Events : 97557
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>>
>> [root@ion ~]# mdadm --examine /dev/sdd
>> /dev/sdd:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>> Name : ion:md0 (local to host ion)
>> Creation Time : Tue Feb 5 17:33:27 2013
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1200 sectors
>> State : clean
>> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>>
>> Update Time : Tue Nov 4 21:43:49 2014
>> Checksum : 50c95b7c - correct
>> Events : 97557
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 5
>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@ion ~]# mdadm --examine /dev/sde
>> /dev/sde:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>> Name : ion:md0 (local to host ion)
>> Creation Time : Tue Feb 5 17:33:27 2013
>> Raid Level : raid6
>> Raid Devices : 6
>>
>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=1200 sectors
>> State : clean
>> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>>
>> Update Time : Tue Nov 4 21:43:49 2014
>> Checksum : 71b191d6 - correct
>> Events : 97557
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> [root@ion ~]# mdadm --examine /dev/sdf1
>> /dev/sdf1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>> Name : ion:0 (local to host ion)
>> Creation Time : Thu Nov 20 23:52:58 2014
>> Raid Level : raid6
>> Raid Devices : 5
>>
>> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262056 sectors, after=0 sectors
>> State : clean
>> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Sun Sep 20 19:47:50 2015
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : b1f6f725 - correct
>> Events : 59736
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>>
>>
>>
>>
>> Here is ldrv:
>>
>> [root@ion ~]# python2 lsdrv
>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>> Logic MegaRAID SAS 1068
>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
>> │└sdc 149.05g [8:32] Partitioned (dos)
>> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
>> │ │└Mounted as /dev/sdc1 @ /boot
>> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
>> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
>> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
>> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
>> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
>> │ │└Mounted as /dev/mapper/ArchVG-root @ /
>> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> ├scsi 3:x:x:x [Empty]
>> ├scsi 4:x:x:x [Empty]
>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>> └sdf 2.73t [8:80] Partitioned (dos)
>> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
>> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>
>>
>> If I try:
>>
>> [root@ion ~]# mdadm --assemble --scan
>> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
>> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>>
>> From dmesg:
>>
>> [ 1227.671344] sda: sda1
>> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
>> [ 1227.680414] md: md_import_device returned -22
>> [ 1227.680462] md: md0 stopped.
>> [ 1227.707969] sdb: sdb1
>> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
>> [ 1227.718611] md: md_import_device returned -22
>> [ 1227.718631] md: md0 stopped.
>> [ 1286.334542] md: md0 stopped.
>> [ 1286.338250] md: bind<sdf1>
>> [ 1286.338390] md: md0 stopped.
>> [ 1286.338400] md: unbind<sdf1>
>> [ 1286.348350] md: export_rdev(sdf1)
>> [ 1286.372268] md: bind<sdf1>
>> [ 1286.373390] md: md1 stopped.
>> [ 1286.373936] md: bind<sdf2>
>> [ 1286.373977] md: md1 stopped.
>> [ 1286.373983] md: unbind<sdf2>
>> [ 1286.388061] md: export_rdev(sdf2)
>> [ 1286.405140] md: bind<sdf2>
>>
>> [root@ion ~]# cat /proc/mdstat
>> Personalities :
>> md1 : inactive sdf2[2](S)
>> 1366104 blocks super 1.2
>>
>> md0 : inactive sdf1[3](S)
>> 1952015360 blocks super 1.2
>>
>> unused devices: <none>
>>
>>
>> If I try manually (I'm not sure of the order though):
>>
>> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
>> /dev/sdb /dev/sdd /dev/sde
>> mdadm: looking for devices for /dev/md0
>> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
>> mdadm: added /dev/sde to /dev/md0 as 1
>> mdadm: no uptodate device for slot 2 of /dev/md0
>> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
>> mdadm: no uptodate device for slot 4 of /dev/md0
>> mdadm: added /dev/sdd to /dev/md0 as 5
>> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
>> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>>
>>
>> Any idea where I should start?
>>
>> Thanks
>> Mathias
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Wols Lists @ 2015-09-23 18:11 UTC (permalink / raw)
To: Guillaume Paumier, Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <CAKWBTuG2AngM6joiODt15Qe+HAXfF6OnJVD-HNuNKikfyv-6MQ@mail.gmail.com>
On 23/09/15 04:20, Guillaume Paumier wrote:
> I figure I have a 50% chance of restoring the array, and 50% chance of
> making it completely unusable. I'm not very comfortable with those
> odds, so I would appreciate if anyone had any advice regarding what to
> do now, and alternatives to manually re-creating the array, if there
> are any.
Read up on loopback devices. Sorry, never done it, can't help
personally, but it sticks a ram-disk layer between mdadm and the actual
disk.
So the disks themselves are read-only, and you can try things out to see
what works. If it fails, you throw the loopback layer away, if it works
you can run it on the disk itself.
Cheers,
Wol
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-23 11:42 UTC (permalink / raw)
To: Neil Brown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjmvwd2xgo.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> Neil Brown <neilb@suse.de> writes:
>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>>
>>>> Hi Neil,
>>>>
>>>> I think we have some bad side effects with this patch:
>>>>
>>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>>> Author: NeilBrown <neilb@suse.com>
>>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>>
>>>> md/raid0: update queue parameter in a safer location.
>>>>
>>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>>> of queue parameters (e.g. max number of sectors per bio) is
>>>> done in the wrong place.
>>>> It should be part of ->run, but it is actually part of ->takeover.
>>>> This means it happens before level_store() calls:
>>>>
>>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>>
>>>> Running the '03r0assem' test suite fills my kernel log with output like
>>>> below. Yi Zhang also had issues where writes failed too.
>>>>
>>>> robably something we need to resolve for 4.2-final or revert the
>>>> offending patch.
>>>>
>>>> Cheers,
>>>> Jes
>>>>
>>>> md: bind<loop0>
>>>> md: bind<loop1>
>>>> md: bind<loop2>
>>>> md/raid0:md2: md_size is 116736 sectors.
>>>> md: RAID0 configuration for md2 - 1 zone
>>>> md: zone0=[loop0/loop1/loop2]
>>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>>
>>>> md2: detected capacity change from 0 to 59768832
>>>> bio too big device loop0 (296 > 255)
>>>> bio too big device loop0 (272 > 255)
>>>
>>> 1/ Why do you blame that particular patch?
>>>
>>> 2/ Where is that error message coming from? I cannot find "bio too big"
>>> in the kernel (except in a comment).
>>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>>> arbitrarily sized bios")
>>> removed the only instance of the error message that I know of.
>>>
>>> Which kernel exactly are you testing?
>>
>> I blame it because of bisect - I revert that patch and the issue goes
>> away.
>>
>> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
>> see the bio too large. I revert it and it goes away.
>
> Hmmm Xiao tells me that the warning message is not in upstream, but I
> was sure I reproduced it in the upstream kernel as well. If I screwed
> up, my apologies, I am going back to my cave and will investigate
> further.
Digging a bit further, I was testing on a 4.2.0-rc5+ where I saw the
issue. That does raise the question of what this patch does to older
kernels, since it was submitted it for 2.6.35+.
"bio too big" message went away with this commit:
commit 54efd50bfd873e2dbf784e0b21a8027ba4299a3e
Author: Kent Overstreet <kent.overstreet@gmail.com>
Date: Thu Apr 23 22:37:18 2015 -0700
block: make generic_make_request handle arbitrarily sized bios
Cheers,
Jes
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-23 11:20 UTC (permalink / raw)
To: Neil Brown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjvbb12xl8.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Neil Brown <neilb@suse.de> writes:
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>
>>> Hi Neil,
>>>
>>> I think we have some bad side effects with this patch:
>>>
>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>> Author: NeilBrown <neilb@suse.com>
>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>
>>> md/raid0: update queue parameter in a safer location.
>>>
>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>> of queue parameters (e.g. max number of sectors per bio) is
>>> done in the wrong place.
>>> It should be part of ->run, but it is actually part of ->takeover.
>>> This means it happens before level_store() calls:
>>>
>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>
>>> Running the '03r0assem' test suite fills my kernel log with output like
>>> below. Yi Zhang also had issues where writes failed too.
>>>
>>> robably something we need to resolve for 4.2-final or revert the
>>> offending patch.
>>>
>>> Cheers,
>>> Jes
>>>
>>> md: bind<loop0>
>>> md: bind<loop1>
>>> md: bind<loop2>
>>> md/raid0:md2: md_size is 116736 sectors.
>>> md: RAID0 configuration for md2 - 1 zone
>>> md: zone0=[loop0/loop1/loop2]
>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>
>>> md2: detected capacity change from 0 to 59768832
>>> bio too big device loop0 (296 > 255)
>>> bio too big device loop0 (272 > 255)
>>
>> 1/ Why do you blame that particular patch?
>>
>> 2/ Where is that error message coming from? I cannot find "bio too big"
>> in the kernel (except in a comment).
>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>> arbitrarily sized bios")
>> removed the only instance of the error message that I know of.
>>
>> Which kernel exactly are you testing?
>
> I blame it because of bisect - I revert that patch and the issue goes
> away.
>
> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
> see the bio too large. I revert it and it goes away.
Hmmm Xiao tells me that the warning message is not in upstream, but I
was sure I reproduced it in the upstream kernel as well. If I screwed
up, my apologies, I am going back to my cave and will investigate
further.
Jes
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-23 11:18 UTC (permalink / raw)
To: Neil Brown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <87k2rhyiqe.fsf@notabene.neil.brown.name>
Neil Brown <neilb@suse.de> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>
>> Hi Neil,
>>
>> I think we have some bad side effects with this patch:
>>
>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>> Author: NeilBrown <neilb@suse.com>
>> Date: Mon Aug 3 13:11:47 2015 +1000
>>
>> md/raid0: update queue parameter in a safer location.
>>
>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>> of queue parameters (e.g. max number of sectors per bio) is
>> done in the wrong place.
>> It should be part of ->run, but it is actually part of ->takeover.
>> This means it happens before level_store() calls:
>>
>> blk_set_stacking_limits(&mddev->queue->limits);
>>
>> Running the '03r0assem' test suite fills my kernel log with output like
>> below. Yi Zhang also had issues where writes failed too.
>>
>> robably something we need to resolve for 4.2-final or revert the
>> offending patch.
>>
>> Cheers,
>> Jes
>>
>> md: bind<loop0>
>> md: bind<loop1>
>> md: bind<loop2>
>> md/raid0:md2: md_size is 116736 sectors.
>> md: RAID0 configuration for md2 - 1 zone
>> md: zone0=[loop0/loop1/loop2]
>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>
>> md2: detected capacity change from 0 to 59768832
>> bio too big device loop0 (296 > 255)
>> bio too big device loop0 (272 > 255)
>
> 1/ Why do you blame that particular patch?
>
> 2/ Where is that error message coming from? I cannot find "bio too big"
> in the kernel (except in a comment).
> Commit: 54efd50bfd87 ("block: make generic_make_request handle
> arbitrarily sized bios")
> removed the only instance of the error message that I know of.
>
> Which kernel exactly are you testing?
I blame it because of bisect - I revert that patch and the issue goes
away.
I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
see the bio too large. I revert it and it goes away.
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH 1/1] md/raid1: Avoid raid1 resync getting stuck
From: Neil Brown @ 2015-09-23 6:49 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, nate.dailey
In-Reply-To: <1442413205-9593-2-git-send-email-Jes.Sorensen@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> close_sync() needs to set conf->next_resync to a large, but safe value
> below MaxSector and use it to determine whether or not to set
> start_next_window in wait_barrier()
>
> Solution suggested by Neil Brown.
>
> Reported-by: Nate Dailey <nate.dailey@stratus.com>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> drivers/md/raid1.c | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 4517f06..763a0a8 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -881,8 +881,7 @@ static sector_t wait_barrier(struct r1conf *conf, struct bio *bio)
> }
>
> if (bio && bio_data_dir(bio) == WRITE) {
> - if (bio->bi_iter.bi_sector >=
> - conf->mddev->curr_resync_completed) {
> + if (bio->bi_iter.bi_sector >= conf->next_resync) {
> if (conf->start_next_window == MaxSector)
> conf->start_next_window =
> conf->next_resync +
> @@ -1516,7 +1515,7 @@ static void close_sync(struct r1conf *conf)
> conf->r1buf_pool = NULL;
>
> spin_lock_irq(&conf->resync_lock);
> - conf->next_resync = 0;
> + conf->next_resync = MaxSector - 2 * NEXT_NORMALIO_DISTANCE;
> conf->start_next_window = MaxSector;
> conf->current_window_requests +=
> conf->next_window_requests;
> --
> 2.4.3
Applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 2/2] raid5: update analysis state for failed stripe
From: Shaohua Li @ 2015-09-23 6:34 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <87bncty7sp.fsf@notabene.neil.brown.name>
On Wed, Sep 23, 2015 at 04:21:58PM +1000, Neil Brown wrote:
> Shaohua Li <shli@fb.com> writes:
>
> > handle_failed_stripe() makes the stripe fail, eg, all IO will return
> > with a failure, but it doesn't update stripe_head_state. Later
> > handle_stripe() has special handling for raid6 for handle_stripe_fill().
> > That check before handle_stripe_fill() doesn't skip the failed stripe
> > and we get a kernel crash in need_this_block. This patch clear the
> > analysis state to make sure no functions wrongly called after
> > handle_failed_stripe()
> >
> > Signed-off-by: Shaohua Li <shli@fb.com>
> > ---
> > drivers/md/raid5.c | 4 ++++
> > 1 file changed, 4 insertions(+)
> >
> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> > index 394cdf8..8e4fb89a 100644
> > --- a/drivers/md/raid5.c
> > +++ b/drivers/md/raid5.c
> > @@ -3155,6 +3155,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
> > spin_unlock_irq(&sh->stripe_lock);
> > if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
> > wake_up(&conf->wait_for_overlap);
> > + if (bi)
> > + s->to_read--;
> > while (bi && bi->bi_iter.bi_sector <
> > sh->dev[i].sector + STRIPE_SECTORS) {
> > struct bio *nextbi =
> > @@ -3173,6 +3175,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
> > */
> > clear_bit(R5_LOCKED, &sh->dev[i].flags);
> > }
> > + s->to_write = 0;
> > + s->written = 0;
> >
> > if (test_and_clear_bit(STRIPE_FULL_WRITE, &sh->state))
> > if (atomic_dec_and_test(&conf->pending_full_writes))
> > --
> > 1.8.1
>
> Again, this probably is a sensible fix, but I would like to be certain.
> Where exactly in need_this_block does the kernel crash? I cannot see
> anything that could cause an invalid address....
>>for (i = 0; i < s->failed; i++) {
>> if (fdev[i]->towrite &&
the fdev[i]->towrite. because s->failed >=2 (it's 3 in my case), while
the array size is 2.
Thanks,
Shaohua
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Alexander Afonyashin @ 2015-09-23 6:24 UTC (permalink / raw)
To: Mathias Burén; +Cc: Linux-RAID
In-Reply-To: <CADNH=7Hr6HXVPQP6-V8Jkr9aSwqHojoWApjA7ZKnQOPL0inUYw@mail.gmail.com>
Hi,
I wonder why metainfo from /dev/sdf1 thinks that there're only 5
devices in raid6. Did you try to grow your array recently?
Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
Can you assemble the array with 4 disks: /dev/sd[a-d]?
P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
from another md-array (check Array UUID field). You may need to zero
metadata on /dev/sdf1 and re-add in to existing raid6 array.
Regards,
Alexander
On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
> Hi (please reply-all)
>
> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
> 2015, kernel 4.1.6
>
> Here is the mdadm --examine for each drive:
>
>
>
> [root@ion ~]# mdadm --examine /dev/sda
> /dev/sda:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
> Name : ion:md0 (local to host ion)
> Creation Time : Tue Feb 5 17:33:27 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
> State : clean
> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>
> Update Time : Tue Nov 4 21:43:49 2014
> Checksum : 528563ee - correct
> Events : 97557
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@ion ~]# mdadm --examine /dev/sdb
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
> Name : ion:md0 (local to host ion)
> Creation Time : Tue Feb 5 17:33:27 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
> State : clean
> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>
> Update Time : Tue Nov 4 21:43:49 2014
> Checksum : 49d756ca - correct
> Events : 97557
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> [root@ion ~]# mdadm --examine /dev/sdd
> /dev/sdd:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
> Name : ion:md0 (local to host ion)
> Creation Time : Tue Feb 5 17:33:27 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1200 sectors
> State : clean
> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>
> Update Time : Tue Nov 4 21:43:49 2014
> Checksum : 50c95b7c - correct
> Events : 97557
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 5
> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@ion ~]# mdadm --examine /dev/sde
> /dev/sde:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
> Name : ion:md0 (local to host ion)
> Creation Time : Tue Feb 5 17:33:27 2013
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=1200 sectors
> State : clean
> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>
> Update Time : Tue Nov 4 21:43:49 2014
> Checksum : 71b191d6 - correct
> Events : 97557
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>
> [root@ion ~]# mdadm --examine /dev/sdf1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
> Name : ion:0 (local to host ion)
> Creation Time : Thu Nov 20 23:52:58 2014
> Raid Level : raid6
> Raid Devices : 5
>
> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=0 sectors
> State : clean
> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Sun Sep 20 19:47:50 2015
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : b1f6f725 - correct
> Events : 59736
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
>
>
>
> Here is ldrv:
>
> [root@ion ~]# python2 lsdrv
> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
> Logic MegaRAID SAS 1068
> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
> │└sdc 149.05g [8:32] Partitioned (dos)
> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
> │ │└Mounted as /dev/sdc1 @ /boot
> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
> │ │└Mounted as /dev/mapper/ArchVG-root @ /
> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> ├scsi 3:x:x:x [Empty]
> ├scsi 4:x:x:x [Empty]
> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
> {0ad2603e-e432-83ee-0218-077398e716ef}
> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
> └sdf 2.73t [8:80] Partitioned (dos)
> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>
>
> If I try:
>
> [root@ion ~]# mdadm --assemble --scan
> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>
> From dmesg:
>
> [ 1227.671344] sda: sda1
> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
> [ 1227.680414] md: md_import_device returned -22
> [ 1227.680462] md: md0 stopped.
> [ 1227.707969] sdb: sdb1
> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
> [ 1227.718611] md: md_import_device returned -22
> [ 1227.718631] md: md0 stopped.
> [ 1286.334542] md: md0 stopped.
> [ 1286.338250] md: bind<sdf1>
> [ 1286.338390] md: md0 stopped.
> [ 1286.338400] md: unbind<sdf1>
> [ 1286.348350] md: export_rdev(sdf1)
> [ 1286.372268] md: bind<sdf1>
> [ 1286.373390] md: md1 stopped.
> [ 1286.373936] md: bind<sdf2>
> [ 1286.373977] md: md1 stopped.
> [ 1286.373983] md: unbind<sdf2>
> [ 1286.388061] md: export_rdev(sdf2)
> [ 1286.405140] md: bind<sdf2>
>
> [root@ion ~]# cat /proc/mdstat
> Personalities :
> md1 : inactive sdf2[2](S)
> 1366104 blocks super 1.2
>
> md0 : inactive sdf1[3](S)
> 1952015360 blocks super 1.2
>
> unused devices: <none>
>
>
> If I try manually (I'm not sure of the order though):
>
> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
> /dev/sdb /dev/sdd /dev/sde
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
> mdadm: added /dev/sde to /dev/md0 as 1
> mdadm: no uptodate device for slot 2 of /dev/md0
> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
> mdadm: no uptodate device for slot 4 of /dev/md0
> mdadm: added /dev/sdd to /dev/md0 as 5
> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>
>
> Any idea where I should start?
>
> Thanks
> Mathias
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/2] md: clear CHANGE_PENDING in readonly array
From: Shaohua Li @ 2015-09-23 6:23 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <87eghpy8k2.fsf@notabene.neil.brown.name>
On Wed, Sep 23, 2015 at 04:05:33PM +1000, Neil Brown wrote:
> Shaohua Li <shli@fb.com> writes:
>
> > If faulty disks of an array are more than allowed degraded number, the
> > array enters error handling. It will be marked as read-only with
> > MD_CHANGE_PENDING/RECOVERY_NEEDED set. But currently recovery doesn't
> > clear CHANGE_PENDING bit for read-only array. If MD_CHANGE_PENDING is
> > set for a raid5 array, all returned IO will be hold on a list till the
> > bit is clear. But recovery nevery clears this bit, the IO is always in
> > pending state and nevery finish. This has bad effects like upper layer
> > can't get an IO error and the array can't be stopped.
> >
> > Signed-off-by: Shaohua Li <shli@fb.com>
> > ---
> > drivers/md/md.c | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > index 95824fb..c596b73 100644
> > --- a/drivers/md/md.c
> > +++ b/drivers/md/md.c
> > @@ -8209,6 +8209,7 @@ void md_check_recovery(struct mddev *mddev)
> > md_reap_sync_thread(mddev);
> > clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> > clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> > + clear_bit(MD_CHANGE_PENDING, &mddev->flags);
> > goto unlock;
> > }
> >
> > --
> > 1.8.1
>
> Hi,
> I can see that clearing MD_CHANGE_PENDING there is probably correct -
> bug introduced by
> Commit: c3cce6cda162 ("md/raid5: ensure device failure recorded before write request returns.")
>
> However I don't understand your reasoning. You say that the array is
> marked as read-only, but I don't see how that would happen. What
> causes the array to be marked "read-only"?
It's set read-only by mdadm. I didn't look carefully, but looks there is
disk failure event, mdadm is invoked automatically by some background
daemon. It's a ubuntu distribution.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 2/2] raid5: update analysis state for failed stripe
From: Neil Brown @ 2015-09-23 6:21 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <0fc9dd19baf6f214a112b040401a4cf3d6313942.1442596586.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 1710 bytes --]
Shaohua Li <shli@fb.com> writes:
> handle_failed_stripe() makes the stripe fail, eg, all IO will return
> with a failure, but it doesn't update stripe_head_state. Later
> handle_stripe() has special handling for raid6 for handle_stripe_fill().
> That check before handle_stripe_fill() doesn't skip the failed stripe
> and we get a kernel crash in need_this_block. This patch clear the
> analysis state to make sure no functions wrongly called after
> handle_failed_stripe()
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5.c | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 394cdf8..8e4fb89a 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3155,6 +3155,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
> spin_unlock_irq(&sh->stripe_lock);
> if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
> wake_up(&conf->wait_for_overlap);
> + if (bi)
> + s->to_read--;
> while (bi && bi->bi_iter.bi_sector <
> sh->dev[i].sector + STRIPE_SECTORS) {
> struct bio *nextbi =
> @@ -3173,6 +3175,8 @@ handle_failed_stripe(struct r5conf *conf, struct stripe_head *sh,
> */
> clear_bit(R5_LOCKED, &sh->dev[i].flags);
> }
> + s->to_write = 0;
> + s->written = 0;
>
> if (test_and_clear_bit(STRIPE_FULL_WRITE, &sh->state))
> if (atomic_dec_and_test(&conf->pending_full_writes))
> --
> 1.8.1
Again, this probably is a sensible fix, but I would like to be certain.
Where exactly in need_this_block does the kernel crash? I cannot see
anything that could cause an invalid address....
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 1/2] md: clear CHANGE_PENDING in readonly array
From: Neil Brown @ 2015-09-23 6:05 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <bda150068cfdb69bf8b632c32dc2a7db17c919f4.1442596586.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]
Shaohua Li <shli@fb.com> writes:
> If faulty disks of an array are more than allowed degraded number, the
> array enters error handling. It will be marked as read-only with
> MD_CHANGE_PENDING/RECOVERY_NEEDED set. But currently recovery doesn't
> clear CHANGE_PENDING bit for read-only array. If MD_CHANGE_PENDING is
> set for a raid5 array, all returned IO will be hold on a list till the
> bit is clear. But recovery nevery clears this bit, the IO is always in
> pending state and nevery finish. This has bad effects like upper layer
> can't get an IO error and the array can't be stopped.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 95824fb..c596b73 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -8209,6 +8209,7 @@ void md_check_recovery(struct mddev *mddev)
> md_reap_sync_thread(mddev);
> clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + clear_bit(MD_CHANGE_PENDING, &mddev->flags);
> goto unlock;
> }
>
> --
> 1.8.1
Hi,
I can see that clearing MD_CHANGE_PENDING there is probably correct -
bug introduced by
Commit: c3cce6cda162 ("md/raid5: ensure device failure recorded before write request returns.")
However I don't understand your reasoning. You say that the array is
marked as read-only, but I don't see how that would happen. What
causes the array to be marked "read-only"?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Guillaume Paumier @ 2015-09-23 3:20 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1509221224470.8750@uplift.swm.pp.se>
Hello,
On 22 September 2015 at 03:29, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> You're not the first one here with this problem.
>
> I can't find it right now, but I believe someone issued --grow --continue on
> the array to get it out of the state that you have right now.
>
> I don't know if it's this:
>
> http://www.spinics.net/lists/raid/msg49077.html
>
> I do not recommend a reboot or anything else, others have had problems with
> this. Also make sure you collect as much information as possible, mdadm
> --examine, copy the superblocks in binary form using dd to somewhere etc, so
> you can get back to the state where you are now.
>
> Look through the mailing list archives from the past 6 months, you're not
> alone in having this problem.
Thank you for your advice and pointers. I looked further into that
thread and others, and they helped to investigate the issue.
Unfortunately, it seems (of course) that I have only succeeded in
making the situation worse.
The thread you linked to was indeed similar to my issue. I first
looked into mdadm-grow-continue@md0.service, as suggested in
http://www.spinics.net/lists/raid/msg49042.html :
______________________________________________
# systemctl status mdadm-grow-continue@md0.service
-------------------------------------------------------------------------------
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
Active: failed (Result: exit-code) since dim. 2015-09-20 16:48:15
PDT; 1 day 20h ago
Process: 10989 ExecStart=/sbin/mdadm --grow --continue /dev/%I
(code=exited, status=1/FAILURE)
Main PID: 10989 (code=exited, status=1/FAILURE)
______________________________________________
The service was inactive, which was consistent with the fact that I
didn't see any activity in top. I tried to re-start it, but the result
was the same.
Per http://www.spinics.net/lists/raid/msg49098.html , I commented out
the StandardInput, StandardOutput and StandardError entries in the
service file, and reloaded the daemon. This gave me these additional
lines (bottom two lines of the output):
______________________________________________
# nano /usr/lib/systemd/system/mdadm-grow-continue@.service
# systemctl daemon-reload
# systemctl start mdadm-grow-continue@md0.service
# systemctl status mdadm-grow-continue@md0.service
-------------------------------------------------------------------------------
mdadm-grow-continue@md0.service - Manage MD Reshape on /dev/md0
Loaded: loaded (/usr/lib/systemd/system/mdadm-grow-continue@.service; static)
Active: failed (Result: exit-code) since mar. 2015-09-22 13:03:26 PDT; 2s ago
Process: 14688 ExecStart=/sbin/mdadm --grow --continue /dev/%I
(code=exited, status=1/FAILURE)
Main PID: 14688 (code=exited, status=1/FAILURE)
sept. 22 13:03:26 mdadm[14688]: mdadm: Need to backup 5376K of
critical section..
sept. 22 13:03:26 mdadm[14688]: mdadm: array: Cannot grow - need a
spare or backup-file to backup critical section
______________________________________________
So apparently, the array couldn't be grown because there was no backup
file originally specified.
At that point, I figured I would stop the array and try to re-assemble
it in its prior state. In retrospect, it was probably a bad idea, but
I didn't realize at the time that stopping the array would be so
problematic.
I attempted to re-assemble using the "--update=revert-reshape" option
as suggested in http://www.spinics.net/lists/raid/msg49596.html , but
that didn't work:
______________________________________________
# mdadm --assemble --force --update=revert-reshape /dev/md0 /dev/sdc1
/dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1
--backup-file=/root/backup.bak --invalid-backup --verbose
-------------------------------------------------------------------------------
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdj1 is identified as a member of /dev/md0, slot 5.
mdadm: :/dev/md0 has an active reshape - checking if critical section
needs to be restored
mdadm: Cannot read from /root/backup.bak
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/sdd1 to /dev/md0 as 1
mdadm: added /dev/sdg1 to /dev/md0 as 2
mdadm: added /dev/sdh1 to /dev/md0 as 3
mdadm: added /dev/sdi1 to /dev/md0 as 4
mdadm: added /dev/sdj1 to /dev/md0 as 5
mdadm: added /dev/sdf1 to /dev/md0 as 6
mdadm: added /dev/sde1 to /dev/md0 as 7
mdadm: no uptodate device for slot 16 of /dev/md0
mdadm: added /dev/sdc1 to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
______________________________________________
This was the output of mdadm -D afterwards:
______________________________________________
# mdadm -D /dev/md0
-------------------------------------------------------------------------------
/dev/md0:
Version : 1.0
Raid Level : raid0
Total Devices : 8
Persistence : Superblock is persistent
State : inactive
Delta Devices : -1, (1->0)
New Level : raid6
New Layout : left-symmetric
New Chunksize : 128K
Name : (redacted):0
UUID : eea59047:120a0365:353da182:6787e030
Events : 35500
Number Major Minor RaidDevice
- 8 33 - /dev/sdc1
- 8 49 - /dev/sdd1
- 8 65 - /dev/sde1
- 8 81 - /dev/sdf1
- 8 97 - /dev/sdg1
- 8 113 - /dev/sdh1
- 8 129 - /dev/sdi1
- 8 145 - /dev/sdj1
______________________________________________
dmesg provided some additional information, saying "reshape_position
too early for auto-recovery":
______________________________________________
# dmesg | grep md
-------------------------------------------------------------------------------
[164758.944492] md: bind<sdd1>
[164758.944819] md: bind<sdg1>
[164758.944977] md: bind<sdh1>
[164758.945152] md: bind<sdi1>
[164758.945339] md: bind<sdj1>
[164758.945472] md: bind<sdf1>
[164758.945610] md: bind<sde1>
[164758.945785] md: bind<sdc1>
[164758.952353] md/raid:md0: reshape_position too early for
auto-recovery - aborting.
[164758.952357] md: pers->run() failed ...
______________________________________________
I think this is the same situation as the one described by Daniel Koch
in the thread "Failed to grow" from August 2015:
http://www.spinics.net/lists/raid/msg49643.html . The signs look very
similar.
I have a version of mdadm --examine before stopping the array, and one
of it now (both are posted at the end of this message). I've also
posted a diff at
https://gist.github.com/gpaumier/da3200b357f35288021d/revisions to
make it easier to see the differences.
As far as I can tell, the only difference is the additional disk I
initially tried to add to the array. When trying to re-assemble, mdadm
complained that that disk (/dev/sdb1) had a different superblock
("mdadm: superblock on /dev/sdb1 doesn't match others - assembly
aborted") so I left it out when trying to re-assemble.
I now realize (per http://www.spinics.net/lists/raid/msg49090.html )
that I should have tried to manually do "mdadm --grow --continue" with
an empty backup file before stopping the array, unfortunately that
wasn't clear to me at the time.
According to http://www.spinics.net/lists/raid/msg49669.html , my last
option seems to be to re-create the array, which worked for the
poster. However, that operation (described at
https://raid.wiki.kernel.org/index.php/RAID_Recovery#Restore_array_by_recreating_.28after_multiple_device_failure.29
) seems very dangerous so I wanted to ask again here before doing
anything else.
This is how I see the current situation:
* The good news is that all the data and metadata still appear clean,
so if I tread carefully I should be able to recover my array with no
data loss (and then re-attempt to grow)
* The bad news is that my only option at this point seems to be to try
to re-create the array manually, which would basically cause all data
to be lost if if doesn't work.
I figure I have a 50% chance of restoring the array, and 50% chance of
making it completely unusable. I'm not very comfortable with those
odds, so I would appreciate if anyone had any advice regarding what to
do now, and alternatives to manually re-creating the array, if there
are any.
Many thanks for your help.
______________________________________________
Appendix 1: Output of mdadm --examine /dev/sd[b-j]1 before stopping the array
-------------------------------------------------------------------------------
# mdadm --examine /dev/sd[b-j]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : bde6e5ab:5f4a56e9:10ad82fc:301d477b
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 28369459 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 8
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : e1b689b5:b4a2c5a7:56057b69:a9101af0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : d20d0897 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 1d8e74d3:9abd37f8:f2cf0ab8:02fdcfd6
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : 75afb1b0 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : ddf17d3d:ea944bfb:6886cc91:3366f55f
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : 45b40c6b - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 7
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 38675f59:ea412b1f:67d6ed9a:a33fc5dd
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : c665836 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 6
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : b24758e6:042412c5:9b5a3c06:f167aedf
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : ac7dc747 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 2
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 00e47d82:b49c3905:3ed961fe:40a5f259
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : fb349837 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : a7e34040:fa12382f:c2ef3d85:9c95b1d0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : e0911505 - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 27349115136 (26082.15 GiB 28005.49 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 9d89c55d:9f4a2181:6b87922f:0681d580
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (8->9)
Update Time : Sun Sep 20 16:48:15 2015
Checksum : aa777dfc - correct
Events : 35499
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
______________________________________________
______________________________________________
Appendix 2: Output of mdadm --examine /dev/sd[b-j]1 currently.
-------------------------------------------------------------------------------
# mdadm --examine /dev/sd[b-j]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : bde6e5ab:5f4a56e9:10ad82fc:301d477b
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 283907e0 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 8
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : e1b689b5:b4a2c5a7:56057b69:a9101af0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : d20f7c1e - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 1d8e74d3:9abd37f8:f2cf0ab8:02fdcfd6
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : 75b22537 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : ddf17d3d:ea944bfb:6886cc91:3366f55f
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : 45b67ff2 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 7
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 38675f59:ea412b1f:67d6ed9a:a33fc5dd
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : c68cbbd - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 6
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : b24758e6:042412c5:9b5a3c06:f167aedf
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : ac803ace - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 2
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 00e47d82:b49c3905:3ed961fe:40a5f259
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : fb370bbe - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : a7e34040:fa12382f:c2ef3d85:9c95b1d0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : e093888c - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 9d89c55d:9f4a2181:6b87922f:0681d580
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : aa79f183 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
______________________________________________
--
Guillaume Paumier
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Neil Brown @ 2015-09-23 2:25 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjoagucvyq.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Hi Neil,
>
> I think we have some bad side effects with this patch:
>
> commit 199dc6ed5179251fa6158a461499c24bdd99c836
> Author: NeilBrown <neilb@suse.com>
> Date: Mon Aug 3 13:11:47 2015 +1000
>
> md/raid0: update queue parameter in a safer location.
>
> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
> of queue parameters (e.g. max number of sectors per bio) is
> done in the wrong place.
> It should be part of ->run, but it is actually part of ->takeover.
> This means it happens before level_store() calls:
>
> blk_set_stacking_limits(&mddev->queue->limits);
>
> Running the '03r0assem' test suite fills my kernel log with output like
> below. Yi Zhang also had issues where writes failed too.
>
> robably something we need to resolve for 4.2-final or revert the
> offending patch.
>
> Cheers,
> Jes
>
> md: bind<loop0>
> md: bind<loop1>
> md: bind<loop2>
> md/raid0:md2: md_size is 116736 sectors.
> md: RAID0 configuration for md2 - 1 zone
> md: zone0=[loop0/loop1/loop2]
> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>
> md2: detected capacity change from 0 to 59768832
> bio too big device loop0 (296 > 255)
> bio too big device loop0 (272 > 255)
1/ Why do you blame that particular patch?
2/ Where is that error message coming from? I cannot find "bio too big"
in the kernel (except in a comment).
Commit: 54efd50bfd87 ("block: make generic_make_request handle arbitrarily sized bios")
removed the only instance of the error message that I know of.
Which kernel exactly are you testing?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Unable to assemble RAID6 after Ubuntu>Arch switch
From: Mathias Burén @ 2015-09-22 21:30 UTC (permalink / raw)
To: Linux-RAID
Hi (please reply-all)
I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
2015, kernel 4.1.6
Here is the mdadm --examine for each drive:
[root@ion ~]# mdadm --examine /dev/sda
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
Name : ion:md0 (local to host ion)
Creation Time : Tue Feb 5 17:33:27 2013
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=18446744073706818560 sectors
State : clean
Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
Update Time : Tue Nov 4 21:43:49 2014
Checksum : 528563ee - correct
Events : 97557
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
[root@ion ~]# mdadm --examine /dev/sdb
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
Name : ion:md0 (local to host ion)
Creation Time : Tue Feb 5 17:33:27 2013
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=18446744073706818560 sectors
State : clean
Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
Update Time : Tue Nov 4 21:43:49 2014
Checksum : 49d756ca - correct
Events : 97557
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
[root@ion ~]# mdadm --examine /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
Name : ion:md0 (local to host ion)
Creation Time : Tue Feb 5 17:33:27 2013
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1200 sectors
State : clean
Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
Update Time : Tue Nov 4 21:43:49 2014
Checksum : 50c95b7c - correct
Events : 97557
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
[root@ion ~]# mdadm --examine /dev/sde
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
Name : ion:md0 (local to host ion)
Creation Time : Tue Feb 5 17:33:27 2013
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=1200 sectors
State : clean
Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
Update Time : Tue Nov 4 21:43:49 2014
Checksum : 71b191d6 - correct
Events : 97557
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
[root@ion ~]# mdadm --examine /dev/sdf1
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
Name : ion:0 (local to host ion)
Creation Time : Thu Nov 20 23:52:58 2014
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Sep 20 19:47:50 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b1f6f725 - correct
Events : 59736
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
Here is ldrv:
[root@ion ~]# python2 lsdrv
PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
Logic MegaRAID SAS 1068
├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
│└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
└scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
└sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
│└sdc 149.05g [8:32] Partitioned (dos)
│ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
│ │└Mounted as /dev/sdc1 @ /boot
│ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
{KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
│ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
│ ├dm-1 32.00g [254:1] LV root ext4 'root'
{82fa2951-25c8-4e36-bf36-9f4f7747ac46}
│ │└Mounted as /dev/mapper/ArchVG-root @ /
│ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
│└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
├scsi 3:x:x:x [Empty]
├scsi 4:x:x:x [Empty]
├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
│└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
{0ad2603e-e432-83ee-0218-077398e716ef}
└scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
└sdf 2.73t [8:80] Partitioned (dos)
├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
{4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
{a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
└sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
If I try:
[root@ion ~]# mdadm --assemble --scan
mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
From dmesg:
[ 1227.671344] sda: sda1
[ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
[ 1227.680414] md: md_import_device returned -22
[ 1227.680462] md: md0 stopped.
[ 1227.707969] sdb: sdb1
[ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
[ 1227.718611] md: md_import_device returned -22
[ 1227.718631] md: md0 stopped.
[ 1286.334542] md: md0 stopped.
[ 1286.338250] md: bind<sdf1>
[ 1286.338390] md: md0 stopped.
[ 1286.338400] md: unbind<sdf1>
[ 1286.348350] md: export_rdev(sdf1)
[ 1286.372268] md: bind<sdf1>
[ 1286.373390] md: md1 stopped.
[ 1286.373936] md: bind<sdf2>
[ 1286.373977] md: md1 stopped.
[ 1286.373983] md: unbind<sdf2>
[ 1286.388061] md: export_rdev(sdf2)
[ 1286.405140] md: bind<sdf2>
[root@ion ~]# cat /proc/mdstat
Personalities :
md1 : inactive sdf2[2](S)
1366104 blocks super 1.2
md0 : inactive sdf1[3](S)
1952015360 blocks super 1.2
unused devices: <none>
If I try manually (I'm not sure of the order though):
[root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
/dev/sdb /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
mdadm: added /dev/sde to /dev/md0 as 1
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sdd to /dev/md0 as 5
mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
Any idea where I should start?
Thanks
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-22 16:07 UTC (permalink / raw)
To: NeilBrown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjoagucvyq.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Hi Neil,
>
> I think we have some bad side effects with this patch:
>
> commit 199dc6ed5179251fa6158a461499c24bdd99c836
> Author: NeilBrown <neilb@suse.com>
> Date: Mon Aug 3 13:11:47 2015 +1000
>
> md/raid0: update queue parameter in a safer location.
>
> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
> of queue parameters (e.g. max number of sectors per bio) is
> done in the wrong place.
> It should be part of ->run, but it is actually part of ->takeover.
> This means it happens before level_store() calls:
>
> blk_set_stacking_limits(&mddev->queue->limits);
>
> Running the '03r0assem' test suite fills my kernel log with output like
> below. Yi Zhang also had issues where writes failed too.
>
> robably something we need to resolve for 4.2-final or revert the
> offending patch.
Obviously I meant 4.3 here :)
Jes
^ permalink raw reply
* Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-22 15:30 UTC (permalink / raw)
To: NeilBrown; +Cc: Xiao Ni, linux-raid, yizhan
Hi Neil,
I think we have some bad side effects with this patch:
commit 199dc6ed5179251fa6158a461499c24bdd99c836
Author: NeilBrown <neilb@suse.com>
Date: Mon Aug 3 13:11:47 2015 +1000
md/raid0: update queue parameter in a safer location.
When a (e.g.) RAID5 array is reshaped to RAID0, the updating
of queue parameters (e.g. max number of sectors per bio) is
done in the wrong place.
It should be part of ->run, but it is actually part of ->takeover.
This means it happens before level_store() calls:
blk_set_stacking_limits(&mddev->queue->limits);
Running the '03r0assem' test suite fills my kernel log with output like
below. Yi Zhang also had issues where writes failed too.
robably something we need to resolve for 4.2-final or revert the
offending patch.
Cheers,
Jes
md: bind<loop0>
md: bind<loop1>
md: bind<loop2>
md/raid0:md2: md_size is 116736 sectors.
md: RAID0 configuration for md2 - 1 zone
md: zone0=[loop0/loop1/loop2]
zone-offset= 0KB, device-offset= 0KB, size= 58368KB
md2: detected capacity change from 0 to 59768832
bio too big device loop0 (296 > 255)
bio too big device loop0 (272 > 255)
bio too big device loop1 (672 > 255)
bio too big device loop1 (352 > 255)
bio too big device loop2 (1024 > 255)
bio too big device loop0 (672 > 255)
bio too big device loop1 (256 > 255)
bio too big device loop2 (288 > 255)
bio too big device loop2 (736 > 255)
bio too big device loop0 (288 > 255)
bio too big device loop2 (256 > 255)
bio too big device loop2 (368 > 255)
bio too big device loop0 (488 > 255)
bio too big device loop0 (360 > 255)
bio too big device loop0 (256 > 255)
bio too big device loop1 (288 > 255)
bio too big device loop1 (736 > 255)
bio too big device loop2 (288 > 255)
bio too big device loop1 (256 > 255)
bio too big device loop1 (512 > 255)
bio too big device loop2 (856 > 255)
bio too big device loop2 (256 > 255)
bio too big device loop0 (288 > 255)
bio too big device loop0 (736 > 255)
bio too big device loop1 (288 > 255)
bio too big device loop0 (256 > 255)
bio too big device loop0 (512 > 255)
bio too big device loop1 (856 > 255)
bio too big device loop1 (256 > 255)
bio too big device loop2 (288 > 255)
bio too big device loop2 (736 > 255)
md2: detected capacity change from 59768832 to 0
md: md2 stopped.
^ permalink raw reply
* Apply for loan at 3.9% interest rate
From: Loan @ 2015-09-22 13:27 UTC (permalink / raw)
To: Loan
Greetings to You
Are you a business man or woman? Are you in any financial mess or do you
need loan to start up your own business? Do you need loan to
settle your debt, pay off your bills or start a nice business?
Do you have a low credit score and you are finding it hard to obtain
capital loan from local banks/other financial institutes
We are offer all types of loan at a low interest rate of 3.9% percent
without any collateral.
If you are interested please contact us today for
the loan application form and more details
Regards
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Mikael Abrahamsson @ 2015-09-22 10:29 UTC (permalink / raw)
To: Guillaume Paumier; +Cc: linux-raid
In-Reply-To: <CAKWBTuFk4Tz20B4aZHgPF3FoEYXDqwQeKeAwN3Nt-cSAopUQ0Q@mail.gmail.com>
On Mon, 21 Sep 2015, Guillaume Paumier wrote:
> I'm looking for guidance regarding what to do next. I'm not sure
> whether I should wait longer (how long?), or try to re-assemble the
> seemingly-intact original array (how do I stop/restart the reshape
> cleanly?), or something else. Any advice would be greatly appreciated.
You're not the first one here with this problem.
I can't find it right now, but I believe someone issued --grow --continue
on the array to get it out of the state that you have right now.
I don't know if it's this:
http://www.spinics.net/lists/raid/msg49077.html
I do not recommend a reboot or anything else, others have had problems
with this. Also make sure you collect as much information as possible,
mdadm --examine, copy the superblocks in binary form using dd to somewhere
etc, so you can get back to the state where you are now.
Look through the mailing list archives from the past 6 months, you're not
alone in having this problem.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox