Re: failed drive in raid 1 array

[Iordan Iordanov <iordan@cdf.toronto.edu>]

[-- Attachment #1: Type: text/plain, Size: 8741 bytes --]

Hi Roberto (Spadim),

I am attaching the two files necessary for this functionality. The first 
one (gpt_id) is the script which given this example input:

   gpt_id sdb 1

should give this example output:

   PARTITION_LABEL=itest00-drive00-part00

The second file is a udev configuration file which needs to be dropped 
into /etc/udev/rules.d/. When a new device is attached, it runs gpt_id 
on its partitions, and if a GPT label is found, a link in 
/dev/disk/by-label magically appears to the partition in question.

To create a GPT label and name a 100GB partition on /dev/sdb, one would 
do something like this (WARNING, WARNING, WARNING THIS IS A 
DATA-DESTRUCTIVE PROCESS):

parted /dev/sdb
mklabel y gpt
mkpart primary ext3 0 100GB
name 1 itest00-drive00-part00
print
quit

To trigger udevadm to rescan all the devices and remake all the 
symlinks, you can run:

udevadm trigger

The gpt_id and the udev rules file are home-brewed at our department. Enjoy!

Cheers,
Iordan

On 02/24/11 15:08, Roberto Spadim wrote:
> do you have the udev configuration for this (static)?
>
> 2011/2/24 Iordan Iordanov<iordan@cdf.toronto.edu>:
>> Hi guys,
>>
>> I saw a bunch of discussion of devices changing names when hot-plugged. If
>> you get the device name right when you add it to the array first, all is
>> good since the superblock is used to "discover" the device later.
>>
>> However, to make things easier/clearer, and to avoid errors, one can take a
>> look at the set of directories:
>>
>> /dev/disk/by-id
>> /dev/disk/by-path
>> /dev/disk/by-uuid
>> /dev/disk/by-label
>>
>> for a predictable, more static view of the drives. The symlinks in these
>> directories are created by udev, and are simply links to the "real" device
>> nodes /dev/sd{a-z}*. You can either just use these symlinks as a way of
>> verifying that you are adding the right device, or add the device using the
>> symlink.
>>
>> At our location, we even augmented udev to add links to labeled GPT
>> partitions in /dev/disk/by-label, and now our drives/partitions look like
>> this:
>>
>> iscsi00-drive00-part00 ->  ../../sda1
>> iscsi00-drive01-part00 ->  ../../sdb1
>> iscsi00-drive02-part00 ->  ../../sdc1
>> iscsi00-drive03-part00 ->  ../../sdd1
>> iscsi00-drive04-part00 ->  ../../sde1
>>
>> This way, we know exactly which bay contains exactly which drive, and it
>> stays this way. If you guys want, I can share with you the changes to udev
>> necessary and the script which extracts the GPT label and reports it to udev
>> for this magic to happen :). Please reply to this thread with a request if
>> you think it may be useful to you.
>>
>> Cheers,
>> Iordan
>>
>>
>> On 02/23/11 17:13, Roberto Nunnari wrote:
>>>
>>> Roberto Spadim wrote:
>>>>
>>>> hum, maybe you are using mdadm.conf or autodetect, non autodetect
>>>> should be something like this:
>>>> i don´t know the best solution, but it works ehhehe
>>>>
>>>> kernel /vmlinuz-2.6.9-89.31.1.ELsmp ro root=/dev/md0 rhgb
>>>> quiet md=0,/dev/sda,/dev/sdb md=1,xxxx,yyyy.....
>>>>
>>>> or another md array...
>>>>
>>>> humm i readed the sata specification and removing isn´t a problem, at
>>>> eletronic level the sata channel is only data, no power source, all
>>>> channels are diferencial (like rs422 or rs485), i don´t see anyproblem
>>>> removing it. i tryed hot plug a revodrive (pciexpress ssd) and it
>>>> don´t work (reboot) hehehe, pci-express isn´t hot plug =P, sata2 don´t
>>>> have problems, the main problem is a short circuit at power source, if
>>>> you remove with caution no problems =)
>>>>
>>>> i tried in some others distros and udev created a new device when add
>>>> a diferent disk for example, remove sdb, and add another disk create
>>>> sdc (not sdb), maybe with another udev configuration should work
>>>
>>> Ok. I'll keep all that in mind tomorrow.
>>> Best regards.
>>> Robi
>>>
>>>
>>>>
>>>>
>>>> 2011/2/23 Roberto Nunnari<roberto.nunnari@supsi.ch>:
>>>>>
>>>>> Roberto Spadim wrote:
>>>>>>
>>>>>> i don´t know how you setup your kernel (with or without raid
>>>>>
>>>>> I use the official CentOS kernel with no modification and don't
>>>>> know about raid autodetect, but:
>>>>> # cat /boot/config-2.6.24-28-server |grep -i raid
>>>>> CONFIG_BLK_DEV_3W_XXXX_RAID=m
>>>>> CONFIG_MD_RAID0=m
>>>>> CONFIG_MD_RAID1=m
>>>>> CONFIG_MD_RAID10=m
>>>>> CONFIG_MD_RAID456=m
>>>>> CONFIG_MD_RAID5_RESHAPE=y
>>>>> CONFIG_MEGARAID_LEGACY=m
>>>>> CONFIG_MEGARAID_MAILBOX=m
>>>>> CONFIG_MEGARAID_MM=m
>>>>> CONFIG_MEGARAID_NEWGEN=y
>>>>> CONFIG_MEGARAID_SAS=m
>>>>> CONFIG_RAID_ATTRS=m
>>>>> CONFIG_SCSI_AACRAID=m
>>>>>
>>>>>
>>>>>> autodetect?) do you use kernel command line to setup raid? autodetect?
>>>>>
>>>>> /dev/md0 in grub
>>>>> I don't know if that means autodetect, but I guess so..
>>>>>
>>>>>
>>>>>> here in my test machine i´m using kernel command line (grub), i don´t
>>>>>> have a server with hotplug bay, i open the case and remove the wire
>>>>>> with my hands =) after reconecting it with another device kerenel
>>>>>
>>>>> Is it safe? Isn't it a blind bet to fry up the controller and/or disk?
>>>>>
>>>>>
>>>>>> recognize the new device reread the parititions etc etc and i can add
>>>>>> it to array again
>>>>>> my grub is something like:
>>>>>>
>>>>>> md=0,/dev/sda,/dev/sdb .....
>>>>>>
>>>>>> internal meta data, raid1, i didn´t like the autodetect (it´s good)
>>>>>> but i prefer hardcoded kernel command line (it´s not good with usb
>>>>>> devices)
>>>>>
>>>>> the relevant part of my grub is:
>>>>>
>>>>> default=0
>>>>> timeout=5
>>>>> splashimage=(hd0,0)/grub/splash.xpm.gz
>>>>> hiddenmenu
>>>>> title CentOS (2.6.9-89.31.1.ELsmp)
>>>>> root (hd0,0)
>>>>> kernel /vmlinuz-2.6.9-89.31.1.ELsmp ro root=/dev/md0 rhgb quiet
>>>>> initrd /initrd-2.6.9-89.31.1.ELsmp.img
>>>>>
>>>>> Best regards.
>>>>> Robi
>>>>>
>>>>>
>>>>>> 2011/2/23 Roberto Nunnari<roberto.nunnari@supsi.ch>:
>>>>>>>
>>>>>>> Roberto Spadim wrote:
>>>>>>>>
>>>>>>>> sata2 without hot plug?
>>>>>>>
>>>>>>> Hi Roberto.
>>>>>>>
>>>>>>> I mean that there is no hot-plug bay, with sliding rails etc..
>>>>>>> The drives are connected to the mb using standard sata cables.
>>>>>>>
>>>>>>>
>>>>>>>> check if your sda sdb sdc will change after removing it, it愀 depends
>>>>>>>> on your udev or another /dev filesystem
>>>>>>>
>>>>>>> Ok, thank you.
>>>>>>> That means that if I take care to check the above, and
>>>>>>> the new drive will be sdb, then taking the steps indicated
>>>>>>> in my original post will do the job?
>>>>>>>
>>>>>>> Best regards.
>>>>>>> Robi
>>>>>>>
>>>>>>>
>>>>>>>> 2011/2/23 Roberto Nunnari<roberto.nunnari@supsi.ch>:
>>>>>>>>>
>>>>>>>>> Hello.
>>>>>>>>>
>>>>>>>>> I have a linux box, with two 2TB sata HD in raid 1.
>>>>>>>>>
>>>>>>>>> Now, one disk is in failed state and it has no spares:
>>>>>>>>> # cat /proc/mdstat
>>>>>>>>> Personalities : [raid1]
>>>>>>>>> md1 : active raid1 sdb4[2](F) sda4[0]
>>>>>>>>> 1910200704 blocks [2/1] [U_]
>>>>>>>>>
>>>>>>>>> md0 : active raid1 sdb1[1] sda2[0]
>>>>>>>>> 40957568 blocks [2/2] [UU]
>>>>>>>>>
>>>>>>>>> unused devices:<none>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The drives are not hot-plug, so I need to shutdown the box.
>>>>>>>>>
>>>>>>>>> My plan is to:
>>>>>>>>> # sfdisk -d /dev/sdb>  sdb.sfdisk
>>>>>>>>> # mdadm /dev/md1 -r /dev/sdb4
>>>>>>>>> # mdadm /dev/md0 -r /dev/sdb1
>>>>>>>>> # shutdown -h now
>>>>>>>>>
>>>>>>>>> replace the disk and boot (it should come back up, even without one
>>>>>>>>> drive,
>>>>>>>>> right?)
>>>>>>>>>
>>>>>>>>> # sfdisk /dev/sdb<  sdb.sfdisk
>>>>>>>>> # mdadm /dev/md1 -a /dev/sdb4
>>>>>>>>> # mdadm /dev/md0 -a /dev/sdb1
>>>>>>>>>
>>>>>>>>> and the drives should start to resync, right?
>>>>>>>>>
>>>>>>>>> This is my first time I do such a thing, so please, correct me
>>>>>>>>> if the above is not correct, or is not a best practice for
>>>>>>>>> my configuration.
>>>>>>>>>
>>>>>>>>> My last backup of md1 is of mid november, so I need to be
>>>>>>>>> pretty sure I will not lose my data (over 1TB).
>>>>>>>>>
>>>>>>>>> A bit abount my environment:
>>>>>>>>> # mdadm --version
>>>>>>>>> mdadm - v1.12.0 - 14 June 2005
>>>>>>>>> # cat /etc/redhat-release
>>>>>>>>> CentOS release 4.8 (Final)
>>>>>>>>> # uname -rms
>>>>>>>>> Linux 2.6.9-89.31.1.ELsmp i686
>>>>>>>>>
>>>>>>>>> Thank you very much and best regards.
>>>>>>>>> Robi
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>

[-- Attachment #2: gpt_id --]
[-- Type: text/plain, Size: 238 bytes --]

#!/bin/sh

PARENT_DEVICE="$1"
PARTITION="$2"

# Get the label of the partition using parted.
PARTITION_LABEL="`parted -sm /dev/"$PARENT_DEVICE" print | grep "^$PARTITION:" | awk -F: '{print $6}'`"

echo "PARTITION_LABEL=$PARTITION_LABEL"

[-- Attachment #3: 10-gpt-label.rules --]
[-- Type: text/plain, Size: 1285 bytes --]

# This file contains the rules to create by-GPT-label symlinks for devices

# forward scsi device events to the corresponding block device
ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", \
	TEST=="block",			ATTR{block/*/uevent}="change"

# we are only interested in add and change actions for block devices
ACTION!="add|change",			GOTO="gpt_label_end"
SUBSYSTEM!="block",			GOTO="gpt_label_end"

# and we can safely ignore these kinds of devices
KERNEL=="mtd[0-9]*|mtdblock[0-9]*|ram*|loop*|fd*|nbd*|gnbd*|dm-*|md*|btibm*", GOTO="gpt_label_end"

# skip removable ide devices, because open(2) on them causes an events loop
KERNEL=="hd*[!0-9]", ATTR{removable}=="1", DRIVERS=="ide-cs|ide-floppy", \
					GOTO="gpt_label_end"
KERNEL=="hd*[0-9]", ATTRS{removable}=="1", \
					GOTO="gpt_label_end"

# skip xen virtual hard disks
DRIVERS=="vbd",				GOTO="no_hardware_id"

# check these attributes of /sys/class/block nodes
ENV{DEVTYPE}!="?*", ATTR{range}=="?*",	ENV{DEVTYPE}="disk"
ENV{DEVTYPE}!="?*", ATTR{start}=="?*",	ENV{DEVTYPE}="partition"

# probe GPT partition label of disks
KERNEL!="sr*", ENV{DEVTYPE}=="partition", IMPORT{program}="/sbin/gpt_id $parent $number"

ENV{PARTITION_LABEL}=="?*", SYMLINK+="disk/by-label/$env{PARTITION_LABEL}"

LABEL="gpt_label_end"