Dumb questions about mdadm #1 - replacing broken disks - "slot" reuse?

Dumb questions about mdadm #1 - replacing broken disks - "slot" reuse?

[John Hughes]

mdadm version 2.6.7.2-3 on Debian Lenny, kernel  2.6.26-2-xen-amd64

I'm new to mdadm, all my experience with software raid/volume management 
type systems has been with Vertas VxVM on UnixWare.

I'm replacing an existing UnixWare system by Linux and I'm trying to get 
a feel for how to perform some simple operations.

As I understand it to replace a failed disk (assuming no hot spares for 
the moment) I just do:

    mdadm --manage /dev/md0 --remove /dev/failed-disk
    mdadm --manage /dev/md0 --add /dev/new-disk

This works, but when I look at the results it looks rather ugly, the new 
disk goes in a new "slot" in the raid superblock.  Is it the case that 
every time I replace a disk I'm going to get a new slot?  Doesn't that 
make the raid "superblock" grow without limit?

Before:

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid10 sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdc[2] sdr[1] sdb[0]
      426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
      bitmap: 1/204 pages [4KB], 1024KB chunk

unused devices: <none>

# mdadm --examine /dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9477b121:204de4c4:a96d58e9:85746699
           Name : caronia:testarray2  (local to host caronia)
  Creation Time : Fri Sep 18 12:59:24 2009
     Raid Level : raid10
   Raid Devices : 12

 Avail Dev Size : 142323568 (67.87 GiB 72.87 GB)
     Array Size : 853940736 (407.19 GiB 437.22 GB)
  Used Dev Size : 142323456 (67.87 GiB 72.87 GB)
    Data Offset : 144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 64949ade:b6d618a8:f45a3b07:29ddc35e

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Sep 18 13:38:41 2009
       Checksum : 256ea9d4 - correct
         Events : 4

         Layout : near=2, far=1
     Chunk Size : 64K

    Array Slot : 0 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
   Array State : Uuuuuuuuuuuu


After:

# mdadm --manage /dev/md0 --fail /dev/sdc
mdadm: set /dev/sdc faulty in /dev/md0
# mdadm --manage /dev/md0 --remove /dev/sdc
mdadm: hot removed /dev/sdc
# mdadm --manage /dev/md0 --add /dev/sdh
mdadm: added /dev/sdh

[...]
# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid10 sdh[12] sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdr[1] sdb[0]
      426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
      bitmap: 0/204 pages [0KB], 1024KB chunk

unused devices: <none>

Eeew, /dev/sdh is in slot 12, where is slot 2?

And:

# mdadm --examine /dev/sdh
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9477b121:204de4c4:a96d58e9:85746699
           Name : caronia:testarray2  (local to host caronia)
  Creation Time : Fri Sep 18 12:59:24 2009
     Raid Level : raid10
   Raid Devices : 12

 Avail Dev Size : 142323568 (67.87 GiB 72.87 GB)
     Array Size : 853940736 (407.19 GiB 437.22 GB)
  Used Dev Size : 142323456 (67.87 GiB 72.87 GB)
    Data Offset : 144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 31c4a4f5:7aef046d:8981c552:b63165c2

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Sep 18 13:56:55 2009
       Checksum : b09071e2 - correct
         Events : 14

         Layout : near=2, far=1
     Chunk Size : 64K

    Array Slot : 12 (0, 1, failed, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2)
   Array State : uuUuuuuuuuuu 1 failed


Is it the case that every time I replace a disk I'm going to get a new 
slot?  Doesn't that make the raid "superblock" grow without limit?


 

 

Re: Dumb questions about mdadm #1 - replacing broken disks - "slot" reuse?

[Mario 'BitKoenig' Holbe]

John Hughes <john@Calva.COM> wrote:
> This works, but when I look at the results it looks rather ugly, the new 
> disk goes in a new "slot" in the raid superblock.  Is it the case that 

This is not a dumb question. This is just unexpected behaviour :)
I have the same here on a 1.1 superblock RAID1

> Before:
> md0 : active raid10 sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdc[2] sdr[1] sdb[0]
>       426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
>       bitmap: 1/204 pages [4KB], 1024KB chunk
> After:
> md0 : active raid10 sdh[12] sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdr[1] sdb[0]
>       426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
>       bitmap: 0/204 pages [0KB], 1024KB chunk

In my case I created the array as:

# uname -a
Linux darkside 2.6.30-1-686 #1 SMP Sat Aug 15 19:11:58 UTC 2009 i686 GNU/Linux
# mdadm --create -e 1.1 -l raid1 -n 2 -b internal -a md /dev/md9 /dev/hdh missing

and subsequently added /dev/sdd.

/proc/mdstat:
md9 : active raid1 sdd[2] hdh[0]
      312571092 blocks super 1.1 [2/2] [UU]
      bitmap: 1/299 pages [4KB], 512KB chunk

# mdadm -E /dev/hdh /dev/sdd
/dev/hdh: 
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
...
     Raid Level : raid1
   Raid Devices : 2
...
    Data Offset : 264 sectors
   Super Offset : 0 sectors
          State : clean
...
Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Sep 18 20:14:24 2009
       Checksum : 754e56a3 - correct
         Events : 12564
...
   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
/dev/sdd: 
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
...
     Raid Level : raid1
   Raid Devices : 2
...
    Data Offset : 264 sectors
   Super Offset : 0 sectors
          State : clean
...
Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Sep 18 20:14:24 2009
       Checksum : 8747d7b3 - correct
         Events : 12564
...
   Device Role : spare
   Array State : AA ('A' == active, '.' == missing)

`mdadm -A --update=summaries' doesn't help (I didn't expect it to, tho)
`mdadm -A --update=resync' doesn't help
`--fail --remove --add' doesn't help
`--fail --remove --grow -n 1 --grow -n 2 --add' doesn't help
`--fail --remove --grow -n 1 --add --grow -n 2' doesn't help

The array resyncs the few bitmap-chunks and the component remains in
slot 2, the array has no device missing, the bitmap stays at 1 page
(1620 bits set). Writing to the array increases the bits set for a short
time and then they go back to 1 page (1620 bits) and remain there.
The components are in sync and remain in sync, I cmp'ed them.


regards
   Mario
-- 
Damit das Mögliche entsteht, muß immer wieder das Unmögliche versucht werden.
                                                  -- Hermann Hesse

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Dumb questions about mdadm #1 - replacing broken disks - "slot" reuse?

[Mario 'BitKoenig' Holbe]

Hello again,

the issue is btw. easily reproducible:

# dd if=/dev/zero of=/tmp/foo1 bs=1k count=100k
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0,18648 s, 562 MB/s
# losetup /dev/loop0 /tmp/foo1 
# mdadm --create -e 1.1 -l raid1 -n 2 -a md /dev/md8 /dev/loop0 missing
mdadm: array /dev/md8 started.
# dd if=/dev/zero of=/tmp/foo2 bs=1k count=100k
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0,196626 s, 533 MB/s
# losetup /dev/loop1 /tmp/foo2 
# mdadm --add /dev/md8 /dev/loop1
mdadm: added /dev/loop1
# grep -A1 ^md8 /proc/mdstat 
md8 : active raid1 loop1[2] loop0[0]
      102392 blocks super 1.1 [2/2] [UU]
# mdadm -E /dev/loop0 | tail -2
   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
You have new mail in /var/mail/root
# mdadm -E /dev/loop1 | tail -2
   Device Role : spare
   Array State : AA ('A' == active, '.' == missing)
# 


regards
   Mario
-- 
<jv> Oh well, config
<jv> one actually wonders what force in the universe is holding it
<jv> and makes it working
<Beeth> chances and accidents :)

How to reclaim device slots on v1 superblock?

[Mario 'BitKoenig' Holbe]

Hello,

md with v1 superblocks seem to somehow associate device slots with
specific devices and doesn't re-use them with others (this was already
reported by John Hughes a while ago, btw.).

So, my question is: how can I force md to finally give up a specific
slot or re-use it with another device?

For example, creating a degraded array reserves slot 1 for the "missing"
device and never re-uses it, but uses slot 2 instead:
	# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 missing
	# mdadm -a /dev/md9 /dev/loop1
	md9 : active raid1 loop1[2] loop0[0]

Creating the array fully developed uses slot 0 and 1 as expected:
	# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 /dev/loop1  
	md9 : active raid1 loop1[1] loop0[0]

The only way I found to enforce using slot 0 and 1 while creating a
degraded array is to grow the number of active devices later:
	# mdadm -C -n 1 --force -l raid1 -a md /dev/md9 /dev/loop0  
	# mdadm -a /dev/md9 /dev/loop1
	# mdadm -G -n 2 /dev/md9
	md9 : active raid1 loop1[1] loop0[0]

Unfortunately, the reverse assumption: reduce the number of active
devices to re-claim unused slots doesn't appear to work:
	# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 missing
	# mdadm -a /dev/md9 /dev/loop1
	md9 : active raid1 loop1[2] loop0[0]
	# mdadm --fail /dev/md9 /dev/loop1
	# mdadm --remove /dev/md9 /dev/loop1
	# mdadm -G -n 1 --force /dev/md9 
	# mdadm -a /dev/md9 /dev/loop1
	# mdadm -G -n 2 /dev/md9
	md9 : active raid1 loop1[2] loop0[0]

I'm using Debians mdadm 3.1.2-2 which defaults to v1.2 superblocks.



regards
   Mario
-- 
Tower: "Say fuelstate." Pilot: "Fuelstate."
Tower: "Say again." Pilot: "Again."
Tower: "Arghl, give me your fuel!" Pilot: "Sorry, need it by myself..."