Strange RAID behaviour when faced with user error

Strange RAID behaviour when faced with user error

[Jody McIntyre]

A user actually ran into this in the field (RHEL 5.3) and I'm able to reproduce
with:
Linux vm1 2.6.30-rc8 #1 SMP Mon Jun 8 11:32:59 EDT 2009 x86_64 GNU/Linux
mdadm - v2.6.7 - 6th June 2008

I'll investigate (i.e. read/debug code) when I have time but any insights would
be apprecated.

1. Assemble a RAID array incorrectly (forgetting the array name):

# mdadm --assemble /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 --run
mdadm: /dev/sdb1 has been started with 3 drives (out of 4).
(The user did not require --run; I'm not sure why.)

2. An array is actually started.  That's not so weird...

# cat /proc/mdstat
Personalities : [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid10 sdb2[1] sdb4[3] sdb3[2]
      513792 blocks 64K chunks 2 near-copies [4/3] [_UUU]
      
unused devices: <none>

3. But:

# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --stop /dev/sdb1
mdadm: stopped /dev/sdb1
# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: /dev/sdb1 has no superblock - assembly aborted


The problem goes away after a reboot.

Cheers,
Jody

Re: Strange RAID behaviour when faced with user error

[John Robinson]

On 10/06/2009 22:58, Jody McIntyre wrote:
> A user actually ran into this in the field (RHEL 5.3) and I'm able to reproduce
> with:
> Linux vm1 2.6.30-rc8 #1 SMP Mon Jun 8 11:32:59 EDT 2009 x86_64 GNU/Linux
> mdadm - v2.6.7 - 6th June 2008
> 
> I'll investigate (i.e. read/debug code) when I have time but any insights would
> be apprecated.
> 
> 1. Assemble a RAID array incorrectly (forgetting the array name):
> 
> # mdadm --assemble /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 --run
> mdadm: /dev/sdb1 has been started with 3 drives (out of 4).
> (The user did not require --run; I'm not sure why.)
> 
> 2. An array is actually started.  That's not so weird...
> 
> # cat /proc/mdstat
> Personalities : [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
> md127 : active raid10 sdb2[1] sdb4[3] sdb3[2]
>       513792 blocks 64K chunks 2 near-copies [4/3] [_UUU]
>       
> unused devices: <none>
> 
> 3. But:
> 
> # mdadm --examine /dev/sdb1
> mdadm: No md superblock detected on /dev/sdb1.
> # mdadm --stop /dev/sdb1
> mdadm: stopped /dev/sdb1
> # mdadm --examine /dev/sdb1
> mdadm: No md superblock detected on /dev/sdb1.
> # mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
> mdadm: no recogniseable superblock on /dev/sdb1
> mdadm: /dev/sdb1 has no superblock - assembly aborted
> 
> 
> The problem goes away after a reboot.

The device node /dev/sdb1 was turned into a md one - probably 9,127 - 
when you started the array, and stopping the array hasn't reverted it. 
mknod /dev/sdb1 8 17 would probably have fixed it. Assuming you're using 
udev, the /dev/sdb1 device is recreated the next time you reboot, so the 
problem goes away.

I think mdadm 3.0 would have deleted the device node when the array was 
stopped, but mdadm 2.6.x doesn't.

Cheers,

John.

Re: Strange RAID behaviour when faced with user error

[John Robinson]

On 10/06/2009 23:19, I wrote:
> The device node /dev/sdb1 was turned into a md one - probably 9,127 - 
> when you started the array, and stopping the array hasn't reverted it. 

Or possibly /dev/sdb1 was turned into a symlink to /dev/md127.

> mknod /dev/sdb1 8 17 would probably have fixed it.

Sorry, mknod /dev/sdb1 b 8 17, after removing the symlink if that's what 
it is.

> Assuming you're using 
> udev, the /dev/sdb1 device is recreated the next time you reboot, so the 
> problem goes away.

Cheers,

John.

Re: Strange RAID behaviour when faced with user error

[Jody McIntyre]

On Wed, Jun 10, 2009 at 11:19:36PM +0100, John Robinson wrote:

> The device node /dev/sdb1 was turned into a md one - probably 9,127 -  
> when you started the array, and stopping the array hasn't reverted it.  
> mknod /dev/sdb1 8 17 would probably have fixed it. Assuming you're using  
> udev, the /dev/sdb1 device is recreated the next time you reboot, so the  
> problem goes away.

That's exactly right.  Thanks for the insight!

brw-rw---- 1 root disk 9, 127 2009-06-10 17:54 /dev/sdb1

> I think mdadm 3.0 would have deleted the device node when the array was  
> stopped, but mdadm 2.6.x doesn't.

Actually, mdadm 3.0 is even smarter than that:

# ls -l /dev/sdb1
brw-rw---- 1 root disk 8, 17 2009-06-12 13:40 /dev/sdb1
# ./mdadm --version
mdadm - v3.0 - 2nd June 2009
# ./mdadm --assemble --verbose /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: device /dev/sdb1 exists but is not an md array.

Cheers,
Jody

how to resize volume group lvm2?

[Michael Ole Olsen]

How do I resize my volume group with lvm2?

vgextend needs new devices but I only have one device md0 for all my physical
volumes.

I know how to resize my logical volumes under the volume group, but somehow
the volume group st1500 did not extend after raid5 reshape with mdadm?

Is my only option backing up 8TB of data and creating a new volume group?

I dont want to add sd[ai] here with vgextend as lvm2 is on top of 
mdadm so I guess I should have only one device for my VG?

/Michael Ole Olsen

mfs:~# fdisk -l /dev/md0

Disk /dev/md0: 12002.4 GB, 12002414559232 bytes
2 heads, 4 sectors/track, -1364690304 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

mfs:~# vgdisplay
  --- Volume group ---
  VG Name               st1500
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               8,19 TB
  PE Size               4,00 MB
  Total PE              2146198
  Alloc PE / Size       2104832 / 8,03 TB
  Free  PE / Size       41366 / 161,59 GB
  VG UUID               1Rl7ly-OguV-fEbS-TU8F-7tdM-9YH3-wKc9F6

lvm> help vgextend
  vgextend: Add physical volumes to a volume group

vgextend
        [-A|--autobackup y|n]
        [-d|--debug]
        [-h|--help]
        [-t|--test]
        [-v|--verbose]
        [--version]
        VolumeGroupName PhysicalDevicePath [PhysicalDevicePath...]

  Command failed with status code 0.
lvm> vgextend st1500 /dev/md0
  Physical volume '/dev/md0' is already in volume group 'st1500'
  Unable to add physical volume '/dev/md0' to volume group 'st1500'.

Re: how to resize volume group lvm2?

[Michael Ole Olsen]

Guess I was too quick with that one..

pvresize /dev/md0 was all that was needed, no need for restart :-)


On Fri, 12 Jun 2009, Michael Ole Olsen wrote:

> How do I resize my volume group with lvm2?
> 
> vgextend needs new devices but I only have one device md0 for all my physical
> volumes.
> 
> I know how to resize my logical volumes under the volume group, but somehow
> the volume group st1500 did not extend after raid5 reshape with mdadm?
> 
> Is my only option backing up 8TB of data and creating a new volume group?
> 
> I dont want to add sd[ai] here with vgextend as lvm2 is on top of 
> mdadm so I guess I should have only one device for my VG?
> 
> /Michael Ole Olsen
> 
> mfs:~# fdisk -l /dev/md0
> 
> Disk /dev/md0: 12002.4 GB, 12002414559232 bytes
> 2 heads, 4 sectors/track, -1364690304 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
> 
> Disk /dev/md0 doesn't contain a valid partition table
> 
> mfs:~# vgdisplay
>   --- Volume group ---
>   VG Name               st1500
>   System ID             
>   Format                lvm2
>   Metadata Areas        1
>   Metadata Sequence No  4
>   VG Access             read/write
>   VG Status             resizable
>   MAX LV                0
>   Cur LV                2
>   Open LV               1
>   Max PV                0
>   Cur PV                1
>   Act PV                1
>   VG Size               8,19 TB
>   PE Size               4,00 MB
>   Total PE              2146198
>   Alloc PE / Size       2104832 / 8,03 TB
>   Free  PE / Size       41366 / 161,59 GB
>   VG UUID               1Rl7ly-OguV-fEbS-TU8F-7tdM-9YH3-wKc9F6
> 
> lvm> help vgextend
>   vgextend: Add physical volumes to a volume group
> 
> vgextend
>         [-A|--autobackup y|n]
>         [-d|--debug]
>         [-h|--help]
>         [-t|--test]
>         [-v|--verbose]
>         [--version]
>         VolumeGroupName PhysicalDevicePath [PhysicalDevicePath...]
> 
>   Command failed with status code 0.
> lvm> vgextend st1500 /dev/md0
>   Physical volume '/dev/md0' is already in volume group 'st1500'
>   Unable to add physical volume '/dev/md0' to volume group 'st1500'.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to resize volume group lvm2?

[Tapani Tarvainen]

On Fri, Jun 12, 2009 at 08:58:41PM +0200, Michael Ole Olsen (gnu@gmx.net) wrote:

> How do I resize my volume group with lvm2?
> 
> vgextend needs new devices but I only have one device md0 for all my physical
> volumes.
> 
> I know how to resize my logical volumes under the volume group, but somehow
> the volume group st1500 did not extend after raid5 reshape with mdadm?

pvresize is the command you're looking for, I think.

-- 
Tapani Tarvainen