Growing raid 5 to 6; /proc/mdstat reports a strange value?

Growing raid 5 to 6; /proc/mdstat reports a strange value?

[Michael Evans]

mdX : active raid5 sdd1[8](S) sdb1[7](S) sdf8[0] sdl8[4] sdk2[5]
sdc1[6] sdj6[3] sdi8[1]
     Y blocks super 1.1 level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]

# mdadm --grow /dev/mdX --level=6 --raid-devices=8
--backup-file=/root/mdX.backupfile

mdX : active raid6 sdd1[8] sdb1[7] sdf8[0] sdl8[4] sdk2[5] sdc1[6]
sdj6[3] sdi8[1]
     Y blocks super 1.1 level 6, 128k chunk, algorithm 18 [8/9] [UUUUUU_U]
     [>....................]  reshape =  0.0% (33920/484971520)
finish=952.6min speed=8480K/sec

!!! mdadm 3.1.1 I wanted an 8 device raid-6;  Why do you show 9?

What is it showing me now???

blocks super 1.1 level 6, 128k chunk, algorithm 2 [8/10] [UUUUUUUU]

mdadm --detail /dev/mdX
/dev/mdX:
       Version : 1.01
    Raid Level : raid6
    Array Size : 6z ()
 Used Dev Size : z ()
  Raid Devices : 8
 Total Devices : 8
   Persistence : Superblock is persistent

   Update Time : Sun Jan 24 14:40:32 2010
         State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
 Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 128K

   Number   Major   Minor   RaidDevice State
      0       8        .        0      active sync   /dev/sdf
      1       8        .        1      active sync   /dev/sdi
      3       8        .        2      active sync   /dev/sdj
      6       8        .        3      active sync   /dev/sdc
      5       8        .        4      active sync   /dev/sdk
      4       8        .        5      active sync   /dev/sdl
      8       8        .        6      active sync   /dev/sdd
      7       8        .        7      active sync   /dev/sdb

... Did it actually do what I want but just show me the wrong result
with kernel 2.6.32-gentoo-r2

Re: Growing raid 5 to 6; /proc/mdstat reports a strange value?

[Neil Brown]

On Sun, 24 Jan 2010 19:49:31 -0800
Michael Evans <mjevans1983@gmail.com> wrote:

> mdX : active raid5 sdd1[8](S) sdb1[7](S) sdf8[0] sdl8[4] sdk2[5]
> sdc1[6] sdj6[3] sdi8[1]
>      Y blocks super 1.1 level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]
> 
> # mdadm --grow /dev/mdX --level=6 --raid-devices=8
> --backup-file=/root/mdX.backupfile
> 
> mdX : active raid6 sdd1[8] sdb1[7] sdf8[0] sdl8[4] sdk2[5] sdc1[6]
> sdj6[3] sdi8[1]
>      Y blocks super 1.1 level 6, 128k chunk, algorithm 18 [8/9] [UUUUUU_U]
>      [>....................]  reshape =  0.0% (33920/484971520)
> finish=952.6min speed=8480K/sec
> 
> !!! mdadm 3.1.1 I wanted an 8 device raid-6;  Why do you show 9?

That is weird isn't it.  It is showing that 8 devices are in the array, of
which 9 are working.  That cannot be right.
More worrying is that the second last device claim to not be present, which
doesn't seem right.


> 
> What is it showing me now???
> 
> blocks super 1.1 level 6, 128k chunk, algorithm 2 [8/10] [UUUUUUUU]
> 

So now of the 8 devices, 10 are working!!  At least all are present.


> ... Did it actually do what I want but just show me the wrong result
> with kernel 2.6.32-gentoo-r2

Some simple testing suggests it did do what you want, but reported something
silly in /proc/mdstat.

I'll see if I can work out what is happening.

Thanks for the report.

NeilBrown

Re: Growing raid 5 to 6; /proc/mdstat reports a strange value?

[Michael Evans]

On Fri, Jan 29, 2010 at 4:23 AM, Neil Brown <neilb@suse.de> wrote:
> On Sun, 24 Jan 2010 19:49:31 -0800
> Michael Evans <mjevans1983@gmail.com> wrote:
>
>> mdX : active raid5 sdd1[8](S) sdb1[7](S) sdf8[0] sdl8[4] sdk2[5]
>> sdc1[6] sdj6[3] sdi8[1]
>>      Y blocks super 1.1 level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]
>>
>> # mdadm --grow /dev/mdX --level=6 --raid-devices=8
>> --backup-file=/root/mdX.backupfile
>>
>> mdX : active raid6 sdd1[8] sdb1[7] sdf8[0] sdl8[4] sdk2[5] sdc1[6]
>> sdj6[3] sdi8[1]
>>      Y blocks super 1.1 level 6, 128k chunk, algorithm 18 [8/9] [UUUUUU_U]
>>      [>....................]  reshape =  0.0% (33920/484971520)
>> finish=952.6min speed=8480K/sec
>>
>> !!! mdadm 3.1.1 I wanted an 8 device raid-6;  Why do you show 9?
>
> That is weird isn't it.  It is showing that 8 devices are in the array, of
> which 9 are working.  That cannot be right.
> More worrying is that the second last device claim to not be present, which
> doesn't seem right.
>
>
>>
>> What is it showing me now???
>>
>> blocks super 1.1 level 6, 128k chunk, algorithm 2 [8/10] [UUUUUUUU]
>>
>
> So now of the 8 devices, 10 are working!!  At least all are present.
>
>
>> ... Did it actually do what I want but just show me the wrong result
>> with kernel 2.6.32-gentoo-r2
>
> Some simple testing suggests it did do what you want, but reported something
> silly in /proc/mdstat.
>
> I'll see if I can work out what is happening.
>
> Thanks for the report.
>
> NeilBrown
>
>

As a follow-up:
* Looking at the device via mdadm --detail /dev/mdX (while still
invalid in /proc/mdstat) showed the correct information.
* When I stopped and re-started the array the information in
/proc/mdstat was valid.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Growing raid 5 to 6; /proc/mdstat reports a strange value?

[Neil Brown]

On Fri, 29 Jan 2010 23:23:34 +1100
Neil Brown <neilb@suse.de> wrote:

> On Sun, 24 Jan 2010 19:49:31 -0800
> Michael Evans <mjevans1983@gmail.com> wrote:
> 
> > mdX : active raid5 sdd1[8](S) sdb1[7](S) sdf8[0] sdl8[4] sdk2[5]
> > sdc1[6] sdj6[3] sdi8[1]
> >      Y blocks super 1.1 level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]
> > 
> > # mdadm --grow /dev/mdX --level=6 --raid-devices=8
> > --backup-file=/root/mdX.backupfile
> > 
> > mdX : active raid6 sdd1[8] sdb1[7] sdf8[0] sdl8[4] sdk2[5] sdc1[6]
> > sdj6[3] sdi8[1]
> >      Y blocks super 1.1 level 6, 128k chunk, algorithm 18 [8/9] [UUUUUU_U]
> >      [>....................]  reshape =  0.0% (33920/484971520)
> > finish=952.6min speed=8480K/sec
> > 
> > !!! mdadm 3.1.1 I wanted an 8 device raid-6;  Why do you show 9?
> 
> That is weird isn't it.  It is showing that 8 devices are in the array, of
> which 9 are working.  That cannot be right.
> More worrying is that the second last device claim to not be present, which
> doesn't seem right.

The second last device being missing is actually correct.  The '_' doesn't
actually mean "missing" but just "not completely in-sync".
When you converted from RAID5 to RAID6 it added the 7th device which clearly
was not in-sync.
Then converting to an 8-device array added the 8th device, but as the array
was not expecting any data to be on this device it is by definition
in-sync and so represented by "U".

The only problem is that it says "9" are in-sync where it should say "7"
are.
The following patch, which I have submitted upstream, fixes this.

Thanks again for the report.

NeilBrown


Author: NeilBrown <neilb@suse.de>
Date:   Tue Feb 9 12:31:47 2010 +1100

    md: fix 'degraded' calculation when starting a reshape.
    
    This code was written long ago when it was not possible to
    reshape a degraded array.  Now it is so the current level of
    degraded-ness needs to be taken in to account.  Also newly addded
    devices should only reduce degradedness if they are deemed to be
    in-sync.
    
    In particular, if you convert a RAID5 to a RAID6, and increase the
    number of devices at the same time, then the 5->6 conversion will
    make the array degraded so the current code will produce a wrong
    value for 'degraded' - "-1" to be precise.
    
    If the reshape runs to completion end_reshape will calculate a correct
    new value for 'degraded', but if a device fails during the reshape an
    incorrect decision might be made based on the incorrect value of
    "degraded".
    
    This patch is suitable for 2.6.32-stable and if they are still open,
    2.6.31-stable and 2.6.30-stable as well.
    
    Cc: stable@kernel.org
    Reported-by: Michael Evans <mjevans1983@gmail.com>
    Signed-off-by: NeilBrown <neilb@suse.de>

diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index e84204e..b5629c3 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5464,11 +5464,11 @@ static int raid5_start_reshape(mddev_t *mddev)
 		    !test_bit(Faulty, &rdev->flags)) {
 			if (raid5_add_disk(mddev, rdev) == 0) {
 				char nm[20];
-				if (rdev->raid_disk >= conf->previous_raid_disks)
+				if (rdev->raid_disk >= conf->previous_raid_disks) {
 					set_bit(In_sync, &rdev->flags);
-				else
+					added_devices++;
+				} else
 					rdev->recovery_offset = 0;
-				added_devices++;
 				sprintf(nm, "rd%d", rdev->raid_disk);
 				if (sysfs_create_link(&mddev->kobj,
 						      &rdev->kobj, nm))
@@ -5480,9 +5480,12 @@ static int raid5_start_reshape(mddev_t *mddev)
 				break;
 		}
 
+	/* When a reshape changes the number of devices, ->degraded
+	 * is measured against the large of the pre and post number of
+	 * devices.*/
 	if (mddev->delta_disks > 0) {
 		spin_lock_irqsave(&conf->device_lock, flags);
-		mddev->degraded = (conf->raid_disks - conf->previous_raid_disks)
+		mddev->degraded += (conf->raid_disks - conf->previous_raid_disks)
 			- added_devices;
 		spin_unlock_irqrestore(&conf->device_lock, flags);
 	}