Re: mdadm -add doesn't start rebuilding array

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 8339 bytes --]

On Tue, 21 Aug 2012 08:45:50 +0200 Sergiusz Brzeziński
<Sergiusz.Brzezinski@supersystem.pl> wrote:

> 
> 
> W dniu 21.08.2012 00:30, NeilBrown pisze:
> > On Mon, 20 Aug 2012 11:36:10 +0200 Sergiusz Brzeziński
> > <Sergiusz.Brzezinski@supersystem.pl>  wrote:
> >
> >> Hi,
> >>
> >> My system is Ubuntu 12.04, kernel 3.2.0-27. mdadm: 3.2.3
> >>
> >> If I do:
> >>
> >> # mdadm /dev/md0 -a /dev/sdc3
> >>
> >> then nothing happen! No message on command line, no info in logs!
> >>
> >> Man mdadm says:
> >>
> >> -a, --add
> >> hot-add listed devices.  If a device appears to have recently been part of the
> >> array (possibly it failed or was removed) the device is  re-added  as  describe
> >>    in  the  next point.   If  that  fails  or the device was never part of the
> >> array, the device is added as a hot-spare.  If the array is degraded, it will
> >> immediately start to rebuild data onto that spare.
> >>
> >> But array doesn't wont to rebuild.
> >>
> >> But not exactly. The bad info is, that sometimes it works and sometimes it doesn't.
> >>
> >> And if I restart the system, the array SOMETIMES start rebuilding and somethimes
> >> doesn't!
> >>
> >> There is no information what's up (comnand line or logs) so I don't know what to do.
> >>
> >>
> >>
> >> I describe bellow the whole procedure:
> >>
> >> 1.
> >> There ist a working Raid1 array:
> >>
> >> # cat /proc/mdstat
> >>
> >> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4]
> >> [raid10]
> >> md0 : active raid1 sda3[2] sdb3[3]
> >>         115999672 blocks super 1.0 [2/2] [UU]
> >>         bitmap: 0/1 pages [0KB], 65536KB chunk
> >>
> >> # mdadm --detail /dev/md0
> >>
> >> /dev/md0:
> >>           Version : 1.0
> >>     Creation Time : Wed Aug  1 15:45:56 2012
> >>        Raid Level : raid1
> >>        Array Size : 115999672 (110.63 GiB 118.78 GB)
> >>     Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
> >>      Raid Devices : 2
> >>     Total Devices : 2
> >>       Persistence : Superblock is persistent
> >>
> >>     Intent Bitmap : Internal
> >>
> >>       Update Time : Mon Aug 20 11:10:00 2012
> >>             State : active
> >>    Active Devices : 2
> >> Working Devices : 2
> >>    Failed Devices : 0
> >>     Spare Devices : 0
> >>
> >>              Name : linux:0
> >>              UUID : b7407176:2e88c73d:2c85e940:05ff7e06
> >>            Events : 5897
> >>
> >>       Number   Major   Minor   RaidDevice State
> >>          3       8       19        0      active sync   /dev/sdb3
> >>          2       8        3        1      active sync   /dev/sda3
> >>
> >>
> >> 2.
> >> I remove (phisicaly) the "/dev/sdb" drive of the box (hot-swap)
> >>
> >> # cat /proc/mdstat
> >>
> >> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4]
> >> [raid10]
> >> md0 : active raid1 sda3[2] sdb3[3](F)
> >>         115999672 blocks super 1.0 [2/1] [_U]
> >>         bitmap: 1/1 pages [4KB], 65536KB chunk
> >>
> >> # mdadm --detail /dev/md0
> >> /dev/md0:
> >>           Version : 1.0
> >>     Creation Time : Wed Aug  1 15:45:56 2012
> >>        Raid Level : raid1
> >>        Array Size : 115999672 (110.63 GiB 118.78 GB)
> >>     Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
> >>      Raid Devices : 2
> >>     Total Devices : 2
> >>       Persistence : Superblock is persistent
> >>
> >>     Intent Bitmap : Internal
> >>
> >>       Update Time : Mon Aug 20 11:11:58 2012
> >>             State : active, degraded
> >>    Active Devices : 1
> >> Working Devices : 1
> >>    Failed Devices : 1
> >>     Spare Devices : 0
> >>
> >>              Name : linux:0
> >>              UUID : b7407176:2e88c73d:2c85e940:05ff7e06
> >>            Events : 5908
> >>
> >>       Number   Major   Minor   RaidDevice State
> >>          0       0        0        0      removed
> >>          2       8        3        1      active sync   /dev/sda3
> >>
> >>          3       8       19        -      faulty spare
> >>
> >>
> >> 3.
> >> I insert the drive again - it become "/dev/sdc" instead of "/dev/sdb"
> >>
> >> 4.
> >> And now I do the following to rebuild the array:
> >>
> >> # mdadm /dev/md0 -a /dev/sdc3
> >>
> >> mdadm: /dev/sdc3 reports being an active member for /dev/md0, but a --re-add fails.
> >> mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
> >> mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.
> >>
> >> I have never seen this messages before (on another systems). But this is not a
> >> problem. I do, what they suggest:
> >>
> >> # mdadm --zero-superblock /dev/sdc3
> >>
> >> # mdadm --examine /dev/sdc3
> >> mdadm: No md superblock detected on /dev/sdc3.
> >>
> >> 5.
> >> And now I try again rebuild the array:
> >>
> >> # mdadm /dev/md0 -a /dev/sdc3
> >>
> >> ... and nothing happen! No message, no info. I repeat some times the command.
> >>
> >> I also try this:
> >>
> >> # mdadm /dev/md0 -a -vv --force /dev/sdc3
> >>
> >> ... also nothing. Somethimes, if I repeat the command one by one, in logs appear
> >> the line:
> >>
> >> Aug 20 11:30:52 serwer-linmot kernel: [ 1978.111282] md: export_rdev(sdc3)
> >>
> >> ... and nothing else
> >>
> >>
> >> # mdadm --examine /dev/sdc3
> >> /dev/sdc3:
> >>             Magic : a92b4efc
> >>           Version : 1.0
> >>       Feature Map : 0x1
> >>        Array UUID : b7407176:2e88c73d:2c85e940:05ff7e06
> >>              Name : linux:0
> >>     Creation Time : Wed Aug  1 15:45:56 2012
> >>        Raid Level : raid1
> >>      Raid Devices : 2
> >>
> >>    Avail Dev Size : 231999344 (110.63 GiB 118.78 GB)
> >>        Array Size : 231999344 (110.63 GiB 118.78 GB)
> >>      Super Offset : 231999472 sectors
> >>             State : clean
> >>       Device UUID : f1a1f291:e10d85cc:b574c6d2:45fc0c5d
> >>
> >> Internal Bitmap : -8 sectors from superblock
> >>       Update Time : Mon Aug 20 11:17:07 2012
> >>          Checksum : 71bcac04 - correct
> >>            Events : 0
> >>
> >>
> >>      Device Role : spare
> >>      Array State : .A ('A' == active, '.' == missing)
> >>
> >>
> >> # mdadm --detail /dev/md0
> >>
> >> /dev/md0:
> >>           Version : 1.0
> >>     Creation Time : Wed Aug  1 15:45:56 2012
> >>        Raid Level : raid1
> >>        Array Size : 115999672 (110.63 GiB 118.78 GB)
> >>     Used Dev Size : 115999672 (110.63 GiB 118.78 GB)
> >>      Raid Devices : 2
> >>     Total Devices : 2
> >>       Persistence : Superblock is persistent
> >>
> >>     Intent Bitmap : Internal
> >>
> >>       Update Time : Mon Aug 20 11:18:43 2012
> >>             State : active, degraded
> >>    Active Devices : 1
> >> Working Devices : 1
> >>    Failed Devices : 1
> >>     Spare Devices : 0
> >>
> >>              Name : linux:0
> >>              UUID : b7407176:2e88c73d:2c85e940:05ff7e06
> >>            Events : 5946
> >>
> >>       Number   Major   Minor   RaidDevice State
> >>          0       0        0        0      removed
> >>          2       8        3        1      active sync   /dev/sda3
> >>
> >>          3       8       19        -      faulty spare
> >>
> >>
> >> Can anyone help?
> >>
> >
> > I cannot see why that would happen.
> > It seems to be failing somewhere in bind_rdev_to_array, but I don't know
> > where.
> > Can you run
> >    strace -o /tmp/strace mdadm /dev/md0 --add /dev/sdc3
> > and post the output ... or at last the tail end of the output.  There should
> > be an ioctl which fails and I need to know what the error was (EEXIST,
> > EINVAL, ENOSPC are the likely ones).  But don't just post that line, include
> > at least the last 100 lines.
> >
> > NeilBrown
> 
> Hi,
> 
> Thank You for help. The complete strace output is in attachement.
> 
> Sergiusz
> 
> 
lseek(4, 118783725568, SEEK_SET)        = 118783725568
ioctl(4, BLKSSZGET, 0x7fff73de3fec)     = 0
write(4, "bitm\4\0\0\0\267@qv.\210\307=,\205\351@\5\377~\6\303-\0\0\0\0\0\0"..., 512) = -1 EINVAL (Invalid argument)


Oh, I remember now.  That bug.
Fixed by commit 6ef89052d85b8137b8a7100f which is in 3.2.4.

http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=6ef89052d85b8137b8a7100f

NeilBrown



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]