Re: How to assemble 4-disk raid5 with one broken disk and one marked as spare by operator error?

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3255 bytes --]

On Sun, 8 Dec 2013 21:04:42 +0100 (CET) Tomas Agartz <tlund@nxs.se> wrote:

> After booting a server that had been powered off for some time, the 4-disk 
> raid5 device was up and running in read-only mode with one disk missing. 
> After a, in hindsight, hasty decision, "mdadm --manage --add /dev/md0 
> /dev/sdd" was executed to re-add the missing device to the array.
> 
> At this time, all hell broke loose :) The first thing that happened was 
> that sdd was added as a spare instead of re-added as expected. The second 
> thing was that a different disk, sdb, was kicked from the array because of 
> read/sata-bus errors. The root disk also bailed and the system had to be 
> powercycled.

If you want to re-add, it is safest to ask mdadm to --re-add, not to --add.

> 
> The real problem, from the start, was probably that sdb was bad all along, 
> but from some reason sdd was the device missing from the array after the 
> initial boot.
> 
> Trying to read data from sdb gives read errors and timeouts, but I was 
> able to do "mdadm --examine" after resetting the sata port.
> 
> The current state is that, out of 4 disks two are good (sde and sdf), one 
> is (in error) marked as a spare (sdd), and the fourth device is unusable 
> (sdb).
> 
> What is the correct method do change the spare disk back to a data disk 
> and try to restart the array with 3 out of 4 devices (sdd, sde and sdf)?
> 

The only real option at this point is to --create the array.  There isn't
enough information for mdadm to be able  to do anything clever.

> The device has never had a spare, so I think that sdd used to be "Active 
> device 0" before this happened?
> 
> Possibly relevant data from mdadm --examine on the four devices:
> 
> sdb          State : clean
> sdb         Events : 333560
> sdb   Device Role : Active device 3
> sdb   Array State : .AAA ('A' == active, '.' == missing)
> 
> sdd          State : clean
> sdd         Events : 333562
> sdd   Device Role : spare
> sdd   Array State : .AA. ('A' == active, '.' == missing)
> 
> sde          State : clean
> sde         Events : 333562
> sde   Device Role : Active device 1
> sde   Array State : .AA. ('A' == active, '.' == missing)
> 
> sdf          State : clean
> sdf         Events : 333562
> sdf   Device Role : Active device 2
> sdf   Array State : .AA. ('A' == active, '.' == missing)
> 
> If no one else has any better suggestions, my best guess would be to: 
> "mdadm --create /dev/md0 --level=5 --raid-devices=4 --assume-clean 
> /dev/sdd /dev/sde /dev/sdf missing" (the device was created with default 
> values, metadata 1.2, chunk size 512K, layout left-symmetric).

Check the "Data Offset" of the devices and make sure the newly created array
gets the same "Data Offset" (it can explicitly be set with the latest mdadm).

NeilBrown


> 
> (Other crazy ideas involve editing the superblock of sdd and making it 
> device 0 and then trying to start the array after that).
> 
> Best regards,
> Tomas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]