Re: Linux RAID autodetect partitions go missing from /dev, but fdisk can see them

["Tor Arne Vestbø" <torarnv@gmail.com>]

Hi Neil!

Neil Brown wrote:
> On Wednesday December 10, torarnv@gmail.com wrote:
>> I have a very strange problem that I've been trying to debug for
>> days now. I had a RAID5 with four drives and one spare,
>> /dev/sd[bcde]1 + /dev/sdf1, and everything was working fine, until
>> one day one of the drives in the array (sdb) no longer had a
>> partition (sdb1). Letting the spare take over I ignored this for a
>> few days, but then it happened again, this time with sdc1.

>> I'm hoping someone on this list may have ran into this before, or
>> have any tips on how I can continue debugging this, because I have to
>> admit I'm a little lost...
> 
> Yes, it does sound rather weird.

First of all, thank you so much for helping me out with this, as I'm
still very lost :)

In addition to the things listed in the first e-mail, I've also tried
installing the latest kernel from kernel.org, but that did not solve
anything. Also, in case it's relevant, I'm running openSUSE 10.3.

> Can you:
> 
>   mdadm -Esv

http://pastebin.com/d7b14d14e

For some reason it seems to think that /dev/sdc and /dev/sdb are part of
the array, while it really is /dev/sdc1 and /dev/sdb1. I'm guessing
since they are missing somehow from the device nodes in /dev mdadm
assumes the disk itself is the member?

> and
>   mdadm --stop /dev/md0
>   strace -o /tmp/str -s 200 mdadm --assemble --scan --verbose /dev/md0

http://pastebin.com/f2c1db2e4

The original array had sd[bcde]1 + sdf1 as spare. Then sdb1 went missing
and the spare kicked in, and then sdc1 went missing, leaving me with a
degraded array.

> Also the contents of /etc/mdadm.conf might help.

http://pastebin.com/f573346ef

Is there anything else I can run, cat, and/or paste that would shed
light over what's going on?

> Thanks,

Thank _you_ :)

Tor Arne



>> raid support in. The symptoms are:
>>
>>   - The kernel seems to detect the partitions (lines 396 and 407 in the
>> dmesg [1])
>>
>>   - But once the boot process finishes and the RAID is started, there is
>> no longer any sdc1 or sdb1, so the RAID fails to start (lines 550-576 in
>> dmesg [1])
>>
>>   - Running fdisk -l shows that the drives in question (sdb and sdc) do
>> have similar partitions as the other working drives, namely one Linux
>> RAID autodetect partition each (see command output [2])
>>
>>   - But, the partitions are missing from /proc/partitions (see [3])
>>
>>   - Manually adding device nodes using mknod works, but doing file -sL
>> on the device gives "writable, no read permission", even though
>> permissions are the same as the other sd* nodes in /dev
>>
>>   - Running 'partprobe -s' successfully finds the two missing partitions
>> and adds device nodes, and the nodes can be 'file -sL'ed, but when
>> trying to assemble the array again with these new nodes in the system,
>> I'm told that sdc1 is not found, and after the --assemble is done, the
>> device nodes are once again missing (!) see [4]
>>
>>   - I've tried using the 'dmraid' command to look for fakeraid
>> partitions or meta data on the drives, which I was told could mess up
>> the auto-detection of Linux software ride partitions, but could not find
>> any issues.
>>
>>
>> As you can tell I've exhausted all my current options, so any help on
>> what I could try next would be very much appreciated. I am especially
>> curious as to why I lose the partitions when mdadm tries to assemble the
>> array?
>>
>> Thanks!
>>
>> Tor Arne Vestbø
>>
>> [1] http://pastebin.com/m15b9c275   dmesg
>> [2] http://pastebin.com/f50fb323a   fdisk -l
>> [3] http://pastebin.com/f4547c2ca   cat /proc/partitions
>> [4] http://pastebin.com/m4475c9ae   partprobe + mdadm --assemble
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html