* RAID5 assemble fails after reboot while reshaping
@ 2015-05-17 16:44 Marco Fuckner
2015-05-17 18:51 ` Phil Turmel
2015-05-18 0:03 ` NeilBrown
0 siblings, 2 replies; 4+ messages in thread
From: Marco Fuckner @ 2015-05-17 16:44 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 4085 bytes --]
Hi everybody,
first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
partitioned with the same geometry.
I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
disks and initiating the grow, the reshape didn't seem to start:
md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
sdc1[0]
11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
[7/7] [UUUUUUU]
[>....................] reshape = 0.0% (0/3906681600)
finish=166847860.0min speed=0K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
I waited about three hours and checked again:
md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
sdc1[0]
11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
[7/7] [UUUUUUU]
[>....................] reshape = 0.0% (0/3906681600)
finish=9599856140.0min speed=0K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
Unfortunately, I forgot to save the output of the grow command, but it
exited with 0.
/mdadm --misc --detail /dev/md0/ didn't show anything suspicious to me:
/dev/md0:
Version : 1.2
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Array Size : 11720044800 (11177.11 GiB 12001.33 GB)
Used Dev Size : 3906681600 (3725.70 GiB 4000.44 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon May 11 11:55:07 2015
State : clean, reshaping
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Reshape Status : 0% complete
Delta Devices : 3, (4->7)
Name : anaNAS:0 (local to host anaNAS)
UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Events : 51839
Number Major Minor RaidDevice State
0 8 33 0 active sync
/dev/sdc1
1 8 113 1 active sync
/dev/sdh1
3 8 97 2 active sync
/dev/sdg1
4 8 17 3 active sync
/dev/sdb1
7 8 81 4 active sync
/dev/sdf1
6 8 65 5 active sync
/dev/sde1
5 8 49 6 active sync
/dev/sdd1
As it looked like it wouldn't be ready until long after my death and I
also wrote a backup file, somehow restarting and continuing afterwards
seemed reasonable to me.
The source I was reading suggested running /mdadm /dev/md0 --continue
--backup-file=$FILE/. Apparently this command was wrong, and I couldn't
reassamble the array:
# mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
--backup-file=/root/grow7backup.bak
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: :/dev/md0 has an active reshape - checking if critical
section needs to be restored
mdadm: No backup metadata on /root/grow7backup.bak
mdadm: No backup metadata on device-4
mdadm: No backup metadata on device-5
mdadm: No backup metadata on device-6
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
I started searching for answers but didn't find anything helpful except
the hint on the raid.wiki.kernel.org page to send an email here. The
last sentence from mdadm sounds a bit pessimistic, but I hope someone in
here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
attachment.
Thanks in advance,
Marco
[-- Attachment #2: raid.status --]
[-- Type: text/plain, Size: 7434 bytes --]
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 721f47b6:4374452b:8cf4d3c6:672caeb3
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c6c20176 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 3
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 528ea570:3578ca82:4e86d330:bf30634c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : db5a0a5d - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 0
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : b6dfe4a2:6db19975:36b5c400:f470ac28
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : aca3041b - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : e233293b:919abecc:d5d7f0d4:7c1b748b
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d3000e94 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 5
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7a51397d:9785430a:bb4e502d:c07c6357
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 76e3ef5c - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 4
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7b0832a7:da8108cd:0bd25ee8:c6bc0ba6
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6d5865f4 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 2
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : d15d44f2:1c4ed6dc:0726a874:b0422ef3
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a1a46170 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 1
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: RAID5 assemble fails after reboot while reshaping
2015-05-17 16:44 RAID5 assemble fails after reboot while reshaping Marco Fuckner
@ 2015-05-17 18:51 ` Phil Turmel
2015-05-17 23:14 ` Marco Fuckner
2015-05-18 0:03 ` NeilBrown
1 sibling, 1 reply; 4+ messages in thread
From: Phil Turmel @ 2015-05-17 18:51 UTC (permalink / raw)
To: Marco Fuckner, linux-raid
Hi Marco,
On 05/17/2015 12:44 PM, Marco Fuckner wrote:
> Hi everybody,
>
> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
> partitioned with the same geometry.
>
> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
> disks and initiating the grow, the reshape didn't seem to start:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=166847860.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
Do you have the exact command you used to start the grow available? Did
you include a backup file? Was it on a device outside the raid?
> I waited about three hours and checked again:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=9599856140.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
That's not good. Looks like it is choking on the very first critical
section backup.
> Unfortunately, I forgot to save the output of the grow command, but it
> exited with 0.
[trim /]
> As it looked like it wouldn't be ready until long after my death and I
> also wrote a backup file, somehow restarting and continuing afterwards
> seemed reasonable to me.
> The source I was reading suggested running /mdadm /dev/md0 --continue
> --backup-file=$FILE/. Apparently this command was wrong, and I couldn't
> reassamble the array:
>
> # mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
> --backup-file=/root/grow7backup.bak
Ah. That looks like a backup file in an appropriate location. :-)
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
> mdadm: :/dev/md0 has an active reshape - checking if critical
> section needs to be restored
> mdadm: No backup metadata on /root/grow7backup.bak
> mdadm: No backup metadata on device-4
> mdadm: No backup metadata on device-5
> mdadm: No backup metadata on device-6
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
So nothing (or garbage) was written to your backup in the first place.
Try again with the "--invalid-backup" option to skip trying to read the
supposedly backed up critical section. You may have corruption to fix
for that small section.
> I started searching for answers but didn't find anything helpful except
> the hint on the raid.wiki.kernel.org page to send an email here. The
> last sentence from mdadm sounds a bit pessimistic, but I hope someone in
> here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
> attachment.
Good report. Unfortunately, it sounds like a bug.
If the --invalid-backup option doesn't help, my next suggestion would be
to temporarily boot with a system rescue CD and continuing the --grow
operation with a more stable kernel. If your backup file isn't empty,
put it on a thumb drive or somewhere accessible to a rescue boot.
If it works with a slightly older kernel, we'll need Neil.
Phil
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RAID5 assemble fails after reboot while reshaping
2015-05-17 18:51 ` Phil Turmel
@ 2015-05-17 23:14 ` Marco Fuckner
0 siblings, 0 replies; 4+ messages in thread
From: Marco Fuckner @ 2015-05-17 23:14 UTC (permalink / raw)
To: Phil Turmel, linux-raid
Hey Phil,
Am 17.05.2015 um 20:51 schrieb Phil Turmel:
> Hi Marco,
>
> On 05/17/2015 12:44 PM, Marco Fuckner wrote:
>> Hi everybody,
>>
>> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
>> partitioned with the same geometry.
>>
>> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
>> disks and initiating the grow, the reshape didn't seem to start:
>>
>> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
>> sdc1[0]
>> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
>> [7/7] [UUUUUUU]
>> [>....................] reshape = 0.0% (0/3906681600)
>> finish=166847860.0min speed=0K/sec
>> bitmap: 0/30 pages [0KB], 65536KB chunk
> Do you have the exact command you used to start the grow available? Did
> you include a backup file? Was it on a device outside the raid?
I used mdadm --grow --raid-devices=7 /dev/md0
--backup-file=/root/grow7backup.bak, the system is on a single disk
outside of the RAID.
> So nothing (or garbage) was written to your backup in the first place.
> Try again with the "--invalid-backup" option to skip trying to read the
> supposedly backed up critical section. You may have corruption to fix
> for that small section.
With the --invalid-backup switch it looks like this:
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
mdadm: :/dev/md0 has an active reshape - checking if critical
section needs to be restored
mdadm: No backup metadata on device-4
mdadm: No backup metadata on device-5
mdadm: No backup metadata on device-6
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/sdh1 to /dev/md0 as 1
mdadm: added /dev/sdg1 to /dev/md0 as 2
mdadm: added /dev/sdb1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as 4
mdadm: added /dev/sde1 to /dev/md0 as 5
mdadm: added /dev/sdd1 to /dev/md0 as 6
mdadm: added /dev/sdc1 to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
This was the case with either my normal system or using a rescue CD
(mdadm 3.3.1-r2 and linux 3.14.35).
> Good report. Unfortunately, it sounds like a bug.
>
> If the --invalid-backup option doesn't help, my next suggestion would be
> to temporarily boot with a system rescue CD and continuing the --grow
> operation with a more stable kernel. If your backup file isn't empty,
> put it on a thumb drive or somewhere accessible to a rescue boot.
>
> If it works with a slightly older kernel, we'll need Neil.
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
The backup file itself is about 1,6M in size. Commands with --grow fail
as the array is marked as inactive:
mdadm: /dev/md0 is not an active md array - aborting
I forgot to include the current status in the last mail:
md0 : inactive sdf1[7](S) sdg1[3](S) sdc1[0](S) sde1[6](S)
sdh1[1](S) sdd1[5](S) sdb1[4](S)
27346771700 blocks super 1.2
It seems like every disk is falsely recognized as a spare?
Regards,
Marco
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: RAID5 assemble fails after reboot while reshaping
2015-05-17 16:44 RAID5 assemble fails after reboot while reshaping Marco Fuckner
2015-05-17 18:51 ` Phil Turmel
@ 2015-05-18 0:03 ` NeilBrown
1 sibling, 0 replies; 4+ messages in thread
From: NeilBrown @ 2015-05-18 0:03 UTC (permalink / raw)
To: Marco Fuckner; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 5998 bytes --]
On Sun, 17 May 2015 18:44:01 +0200 Marco Fuckner <marco@die-fuckners.de>
wrote:
> Hi everybody,
>
> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
> partitioned with the same geometry.
>
> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
> disks and initiating the grow, the reshape didn't seem to start:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=166847860.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
>
> I waited about three hours and checked again:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=9599856140.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
This looks very much like the reshape has not done anything at all.
i.e. your data is still exactly where you left it, it is just a case of
getting hold of it.
It's not impossible that running the --assemble with --update=revert-reshape
would work, but I'm far from certain. If you backed up the first gigabyte of
each device (sd?1) first then it would probably be safe enough to try.
Another option is to add --invalid-backup to the --assemble command.
This has a reasonable chance of allowing the reshape to continue, but also
has a reasonable chance of corrupting the first few megabytes of your array
(the part that it things should be backed up).
If you "make test_stripe" in the mdadm source code, you can use that to extra
the first few megabytes of array data so you could restore it if it gets
corrupted.
Something like
test_stripe save /root/thing 4 262144 5 2 0 \
$BIGNUM /dev/sdc1:262144 /dev/sdh1:262144 ......
Check the source to make sure you get the args right.
Make sure the order of the devices and their data_offsets are correct. check
the "Device Role:" for each and order them by that number.
Another option is to recreate the array as 4-drive RAID5. Again you need to
make sure the device order and data offsets are correct, along with all the
other data.
I might be able to dig into the code and find out what happened and maybe
offer an "easier" solution, but that won't be for a day or two at least.
NeilBrown
>
> Unfortunately, I forgot to save the output of the grow command, but it
> exited with 0.
> /mdadm --misc --detail /dev/md0/ didn't show anything suspicious to me:
>
> /dev/md0:
> Version : 1.2
> Creation Time : Sun Nov 9 02:38:25 2014
> Raid Level : raid5
> Array Size : 11720044800 (11177.11 GiB 12001.33 GB)
> Used Dev Size : 3906681600 (3725.70 GiB 4000.44 GB)
> Raid Devices : 7
> Total Devices : 7
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon May 11 11:55:07 2015
> State : clean, reshaping
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Reshape Status : 0% complete
> Delta Devices : 3, (4->7)
>
> Name : anaNAS:0 (local to host anaNAS)
> UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
> Events : 51839
>
> Number Major Minor RaidDevice State
> 0 8 33 0 active sync
> /dev/sdc1
> 1 8 113 1 active sync
> /dev/sdh1
> 3 8 97 2 active sync
> /dev/sdg1
> 4 8 17 3 active sync
> /dev/sdb1
> 7 8 81 4 active sync
> /dev/sdf1
> 6 8 65 5 active sync
> /dev/sde1
> 5 8 49 6 active sync
> /dev/sdd1
>
> As it looked like it wouldn't be ready until long after my death and I
> also wrote a backup file, somehow restarting and continuing afterwards
> seemed reasonable to me.
> The source I was reading suggested running /mdadm /dev/md0 --continue
> --backup-file=$FILE/. Apparently this command was wrong, and I couldn't
> reassamble the array:
>
> # mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
> --backup-file=/root/grow7backup.bak
>
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
> mdadm: :/dev/md0 has an active reshape - checking if critical
> section needs to be restored
> mdadm: No backup metadata on /root/grow7backup.bak
> mdadm: No backup metadata on device-4
> mdadm: No backup metadata on device-5
> mdadm: No backup metadata on device-6
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
> I started searching for answers but didn't find anything helpful except
> the hint on the raid.wiki.kernel.org page to send an email here. The
> last sentence from mdadm sounds a bit pessimistic, but I hope someone in
> here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
> attachment.
>
> Thanks in advance,
>
> Marco
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-05-18 0:03 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-05-17 16:44 RAID5 assemble fails after reboot while reshaping Marco Fuckner
2015-05-17 18:51 ` Phil Turmel
2015-05-17 23:14 ` Marco Fuckner
2015-05-18 0:03 ` NeilBrown
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).