Help - Urgent!

Help - Urgent!

[Rhorer, Leslie]

Damn!!!

	I was doing some testing at the behest of Doug, when I made a
critical error which locked up the system and now the main array won't
assemble, at all.  I issued the command

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`

	And the system complained it was an invalid chunk-size, so I
tried:

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`

	This time there was no error, but the command line just hung
solid.  I tired everything I could to recover, but the system just went
downhill more and more.  Finally I tried a shutdown, but even that would
not work.  I had to hit the big red switch, and now when I rebooted the
system, mdadm will not assemble the array no matter what I try.  I tried


`mdadm --assemble /dev/md0 -f`

`mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx`

	etc.  It just keeps saying, "No devices found for md0".  When I
issue the --examine switch, the information all looks good for all the
drives, just no RAID.  What do I do, now?

	Please respond to me at leslie.rhorer@twtelecom.com, because
with the server down I do not have access to the list.

---


The content contained in this electronic message is not intended to constitute
formation of a contract binding tw telecom.  tw telecom will be contractually
bound only upon execution, by an authorized officer, of a contract including
agreed terms and conditions or by express application of its tariffs.  This message
is intended only for the use of the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, or the employee or agent
responsible for delivering the message to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this message is strictly
prohibited. If you have received this communication in error, please notify us
immediately by replying to the sender of this E-Mail or by telephone.

Help - Urgent!!!

[lrhorer]

Damn!!!

        I was doing some testing at the behest of Doug, when I made a
critical error which locked up the system and now the main array won't
assemble, at all.  I issued the command

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`

        And the system complained it was an invalid chunk-size, so I
tried:

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`

        This time there was no error, but the command line just hung
solid.  I tired everything I could to recover, but the system just went
downhill more and more.  Finally I tried a shutdown, but even that would
not work.  I had to hit the big red switch, and now when I rebooted the
system, mdadm will not assemble the array no matter what I try.  I tried


`mdadm --assemble /dev/md0 -f`

`mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx`

        etc.  It just keeps saying, "No devices found for md0".  When I
issue the --examine switch, the information all looks good for all the
drives, just no RAID.  What do I do, now?

Re: Help - Urgent!!!

[Neil Brown]

On Thursday November 12, lrhorer@satx.rr.com wrote:
> Damn!!!
> 
>         I was doing some testing at the behest of Doug, when I made a
> critical error which locked up the system and now the main array won't
> assemble, at all.  I issued the command
> 
> `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`
> 
>         And the system complained it was an invalid chunk-size, so I
> tried:
> 
> `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`
> 
>         This time there was no error, but the command line just hung
> solid.  I tired everything I could to recover, but the system just went
> downhill more and more.  Finally I tried a shutdown, but even that would
> not work.  I had to hit the big red switch, and now when I rebooted the
> system, mdadm will not assemble the array no matter what I try.  I tried
> 
> 
> `mdadm --assemble /dev/md0 -f`
> 
> `mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx`
> 
>         etc.  It just keeps saying, "No devices found for md0".  When I
> issue the --examine switch, the information all looks good for all the
> drives, just no RAID.  What do I do, now?


1/ show us what the result of --examine was.
2/ add -vvv to the --assemble and report the result of that.

NeilBrown

Re: Help - Urgent!!!

[Neil Brown]

On Friday November 13, neilb@suse.de wrote:
> 
> 1/ show us what the result of --examine was.
> 2/ add -vvv to the --assemble and report the result of that.
  3/ report the output of "mdadm -X /dev/list-of-devices"

NeilBrown

Re: Help - Urgent!!!

[lrhorer]

On Thursday 12 November 2009 11:12:30 pm you wrote:
> On Friday November 13, neilb@suse.de wrote:
> > 1/ show us what the result of --examine was.
> > 2/ add -vvv to the --assemble and report the result of that.
>
>   3/ report the output of "mdadm -X /dev/list-of-devices"



RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j]
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : RAID-Server:0  (local to host RAID-Server)
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : d40c9255:cef0739f:966d448d:e549ada8

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 9a4e478a - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 0 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : Uuuuuuuuuu
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : b38a638b:ec546588:4f9a4c8b:16886fe6

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 24e4ca52 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 1 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uUuuuuuuuu
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 5158e257:4f166c7f:f5abf1b6:5341dff8

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 267f2437 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 2 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuUuuuuuuu
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : bd1b5e0a:a34494e8:c445f3e0:7cf0bd2d

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : a1035eef - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 3 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuUuuuuuu
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : baf86255:b30938e9:edaf8c86:87b4f49a

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : ff7c2f31 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 4 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuUuuuuu
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 8e2c9eac:617f70f4:5bd00aa3:f59ef3cf

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : b36ce391 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 5 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuuUuuuu
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 0b5819b7:82845162:3e06a649:f7d4664b

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 4dd78014 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 6 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuuuUuuu
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 654fa4ee:290ddbd4:c171e2c5:7b57fdd8

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 1beee1f - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 7 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuuuuUuu
/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 51f8418f:e2dd7c93:6a898b0c:74d6661a

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : e910fe64 - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 8 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuuuuuUu
/dev/sdj:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
           Name : 'RAID-Server':0
  Creation Time : Sat Apr 25 01:17:12 2009
     Raid Level : raid6
   Raid Devices : 10

 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB)
     Array Size : 15628197888 (7452.11 GiB 8001.64 GB)
  Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 42993978:e422dd03:9417b109:cdfd6b07

    Update Time : Thu Nov 12 21:34:18 2009
       Checksum : 2c9399db - correct
         Events : 489251

     Chunk Size : 256K

    Array Slot : 9 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
   Array State : uuuuuuuuuU

Re: Help - Urgent!!!

[lrhorer]

On Thursday 12 November 2009 11:12:30 pm you wrote:
> mdadm -X /dev/list-of-devices


RAID-Server:/etc/cron.d# mdadm -X /dev/sd[a-j]
        Filename : /dev/sda
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdb
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdc
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdd
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sde
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdf
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdg
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdh
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdi
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)
        Filename : /dev/sdj
           Magic : 6d746962
         Version : 4
            UUID : 5ff10d73:a096195f:7a646bba:a68986ca
          Events : 0
  Events Cleared : 0
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 976762368 (931.51 GiB 1000.20 GB)
          Bitmap : 14905 bits (chunks), 14904 dirty (100.0%)

Re: Help - Urgent!!!

[Neil Brown]

On Thursday November 12, lrhorer@satx.rr.com wrote:
> On Thursday 12 November 2009 11:12:30 pm you wrote:
> > On Friday November 13, neilb@suse.de wrote:
> > > 1/ show us what the result of --examine was.
> > > 2/ add -vvv to the --assemble and report the result of that.
> >
> >   3/ report the output of "mdadm -X /dev/list-of-devices"
> 
> 
> 
> RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j]
> /dev/sda:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
>            Name : RAID-Server:0  (local to host RAID-Server)

Strangely this line is unique.  Every other device has:
>            Name : 'RAID-Server':0

Where do those "'"s come from???

What exactly is listed in mdadm.conf for /dev/md0??


Other than that everything looks OK.  The array hasn't been marked as
having a bitmap, but the bitmap seems to have been created correctly,
but not activated yet.

And you didn't provide the output of
   mdadm --assemble -vvv ....

That really should explain what is happening.

NeilBrown

Re: Help - Urgent!!!

[lrhorer]

On Friday 13 November 2009 12:08:47 am you wrote:
> On Thursday November 12, lrhorer@satx.rr.com wrote:
> > On Thursday 12 November 2009 11:12:30 pm you wrote:
> > > On Friday November 13, neilb@suse.de wrote:
> > > > 1/ show us what the result of --examine was.
> > > > 2/ add -vvv to the --assemble and report the result of that.
> > >
> > >   3/ report the output of "mdadm -X /dev/list-of-devices"
> >
> > RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j]
> > /dev/sda:
> >           Magic : a92b4efc
> >         Version : 1.2
> >     Feature Map : 0x0
> >      Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
> >            Name : RAID-Server:0  (local to host RAID-Server)
>
> Strangely this line is unique.  Every other device has:
> >            Name : 'RAID-Server':0
>
> Where do those "'"s come from???

	I have no idea.

> What exactly is listed in mdadm.conf for /dev/md0??

cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE /dev/sd[b-j]

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR lrhorer@satx.rr.com

# definitions of existing MD arrays
# ARRAY /dev/md0 level=raid6 num-devices=10 
UUID=f3b19017:f584ad54:4cb502e9:37bde716

# This file was auto-generated on Fri, 21 Nov 2008 22:35:30 -0600
# by mkconf $Id$
PROGRAM /usr/bin/mdadm_notify
ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=10 
UUID=5ff10d73:a096195f:7a646bba:a68986ca name=RAID-Server:0

I just spotted something not quite right.  Line 7 is missing a drive 
(/dev/sda).   That shouldn't prevent it from assembling, though.

> Other than that everything looks OK.  The array hasn't been marked as
> having a bitmap, but the bitmap seems to have been created correctly,
> but not activated yet.
>
> And you didn't provide the output of
>    mdadm --assemble -vvv ....

	Oh, sorry!  I thought the fact I got it to assemble meant I didn't need to 
issue the command again...

	Indeed, it won't let me issue it again.  it says:

RAID-Server:/etc/cron.d# mdadm -A /dev/md0 /dev/sd[a-j] -vvv
mdadm: device /dev/md0 already active - cannot assemble it

	Should I stop the arraya nd re-assemble it with -vvv?

	Also, what about my question about the bitmap?  Should I delete it, despite 
the fact mdadm says there isn't one, or should I attempt to add it again, or 
should I wait until I do the checkarrray?

Re: Help - Urgent!!!

[Neil Brown]

(cced: to debian mdadm maintainers as I there is a problem caused by
an older version of the mdadm package that you should be aware of).

On Friday November 13, lrhorer@satx.rr.com wrote:
> On Friday 13 November 2009 12:08:47 am you wrote:
> > On Thursday November 12, lrhorer@satx.rr.com wrote:
> > > On Thursday 12 November 2009 11:12:30 pm you wrote:
> > > > On Friday November 13, neilb@suse.de wrote:
> > > > > 1/ show us what the result of --examine was.
> > > > > 2/ add -vvv to the --assemble and report the result of that.
> > > >
> > > >   3/ report the output of "mdadm -X /dev/list-of-devices"
> > >
> > > RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j]
> > > /dev/sda:
> > >           Magic : a92b4efc
> > >         Version : 1.2
> > >     Feature Map : 0x0
> > >      Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca
> > >            Name : RAID-Server:0  (local to host RAID-Server)
> >
> > Strangely this line is unique.  Every other device has:
> > >            Name : 'RAID-Server':0
> >
> > Where do those "'"s come from???
> 
> 	I have no idea.

Well we need to find out because I think they are the problem.
This is Debian, isn't it???

> 
> > What exactly is listed in mdadm.conf for /dev/md0??
> 
> cat /etc/mdadm/mdadm.conf
> # mdadm.conf
> #
> # Please refer to mdadm.conf(5) for information about this file.
> #
> 
> # by default, scan all partitions (/proc/partitions) for MD superblocks.
> # alternatively, specify devices to scan, using wildcards if desired.
> DEVICE /dev/sd[b-j]

As you note, this a problem.  mdadm will only look at these devices,
not at /dev/sda.
And...

> ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=10 
> UUID=5ff10d73:a096195f:7a646bba:a68986ca name=RAID-Server:0

/dev/sda is the only one called
      RAID-Server:0
The others are all called
      'RAID-Server':0

so they won't match.  That is why it won't find these devices.

I can see how this happened now.
There Debian mdadm package (not the most recent, but some fairly
recent) has 

extra_args="--homehost='$MD_HOMEHOST' --auto-update-homehost"
in
/usr/share/initramfs-tools/scripts/local-top/mdadm

This shows a misunderstanding of shell quoting and exactly caused your
problem.  It updated the homehost on all device that were listed in
mdadm.conf to be   'RAID-Server'.  So b-j got updated but a did not.
Presumably the last time you booted /dev/sda didn't get included, so
you had to add it by hand.


> 
> I just spotted something not quite right.  Line 7 is missing a drive 
> (/dev/sda).   That shouldn't prevent it from assembling, though.
> 
> > Other than that everything looks OK.  The array hasn't been marked as
> > having a bitmap, but the bitmap seems to have been created correctly,
> > but not activated yet.
> >
> > And you didn't provide the output of
> >    mdadm --assemble -vvv ....
> 
> 	Oh, sorry!  I thought the fact I got it to assemble meant I didn't need to 
> issue the command again...

I didn't realise that you had got it to assemble.   There is no need
to repeat the command as I see exactly what is happening.
However to fix it you will need to stop the array and assemble with
   --update=homehost


> 
> 	Indeed, it won't let me issue it again.  it says:
> 
> RAID-Server:/etc/cron.d# mdadm -A /dev/md0 /dev/sd[a-j] -vvv
> mdadm: device /dev/md0 already active - cannot assemble it
> 
> 	Should I stop the arraya nd re-assemble it with -vvv?
> 
> 	Also, what about my question about the bitmap?  Should I delete it, despite 
> the fact mdadm says there isn't one, or should I attempt to add it again, or 
> should I wait until I do the checkarrray?
> 

There is nothing to delete.  Just try to add it again.
Though I would really like to know why it hangs.  If that happens
again, see if alt-sysrq-T shows anything useful.


NeilBrown

Re: Help - Urgent!!!

[Janek Kozicki]

Neil Brown said:     (by the date of Fri, 13 Nov 2009 17:08:47 +1100)

> Where do those "'"s come from???

I also have those "'" on one (out of 3) md's. Maybe it's debian
specific? Or only debian etch specific? Or only debian /dev/md0
specific.... It's only on /dev/md0, like this:

backup:~# mdadm -D /dev/md0
/dev/md0:
        Version : 01.00.03
  Creation Time : Wed Oct 31 09:13:41 2007
     Raid Level : raid1
     Array Size : 979924 (957.12 MiB 1003.44 MB)
    Device Size : 979924 (957.12 MiB 1003.44 MB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Fri Nov 13 10:04:02 2009
          State : active
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

           Name : 'backup':0
           UUID : 75b0f878:79539d6c:eef22092:f47a6e6f
         Events : 482808

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       5      22        1        1      active sync   /dev/hdc1
       2       3        1        2      active sync   /dev/hda1
       4       8       33        3      active sync   /dev/sdc1
       6       8       17        4      active sync   /dev/sdb1


> 
> What exactly is listed in mdadm.conf for /dev/md0??

nothing.

backup:~# cat /etc/mdadm/mdadm.conf 
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Tue, 23 Oct 2007 22:18:08 +0200
# by mkconf $Id: mkconf 261 2006-11-09 13:32:35Z madduck $



-- 
Janek Kozicki                                                         |

Re: Help - Urgent!!!

[Sujit K M]

Hi,

This seems to be an reported bug that neil seems to be following. Not
much helpfull
because(Kindly Read the entire mail chain) responder had not updated
on the mail.
But seems to have the solution/seems to be closed.

http://lists.alioth.debian.org/pipermail/pkg-mdadm-devel/2009-June/002945.html

Thanks,
Sujit


On Fri, Nov 13, 2009 at 2:36 PM, Janek Kozicki <janek_listy@wp.pl> wrote:
> Neil Brown said:     (by the date of Fri, 13 Nov 2009 17:08:47 +1100)
>
>> Where do those "'"s come from???
>
> I also have those "'" on one (out of 3) md's. Maybe it's debian
> specific? Or only debian etch specific? Or only debian /dev/md0
> specific.... It's only on /dev/md0, like this:
>
> backup:~# mdadm -D /dev/md0
> /dev/md0:
>        Version : 01.00.03
>  Creation Time : Wed Oct 31 09:13:41 2007
>     Raid Level : raid1
>     Array Size : 979924 (957.12 MiB 1003.44 MB)
>    Device Size : 979924 (957.12 MiB 1003.44 MB)
>   Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>    Persistence : Superblock is persistent
>
>  Intent Bitmap : Internal
>
>    Update Time : Fri Nov 13 10:04:02 2009
>          State : active
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>
>           Name : 'backup':0
>           UUID : 75b0f878:79539d6c:eef22092:f47a6e6f
>         Events : 482808
>
>    Number   Major   Minor   RaidDevice State
>       0       8        1        0      active sync   /dev/sda1
>       5      22        1        1      active sync   /dev/hdc1
>       2       3        1        2      active sync   /dev/hda1
>       4       8       33        3      active sync   /dev/sdc1
>       6       8       17        4      active sync   /dev/sdb1
>
>
>>
>> What exactly is listed in mdadm.conf for /dev/md0??
>
> nothing.
>
> backup:~# cat /etc/mdadm/mdadm.conf
> # mdadm.conf
> #
> # Please refer to mdadm.conf(5) for information about this file.
> #
>
> # by default, scan all partitions (/proc/partitions) for MD superblocks.
> # alternatively, specify devices to scan, using wildcards if desired.
> DEVICE partitions
>
> # auto-create devices with Debian standard permissions
> CREATE owner=root group=disk mode=0660 auto=yes
>
> # automatically tag new arrays as belonging to the local system
> HOMEHOST <system>
>
> # instruct the monitoring daemon where to send mail alerts
> MAILADDR root
>
> # definitions of existing MD arrays
>
> # This file was auto-generated on Tue, 23 Oct 2007 22:18:08 +0200
> # by mkconf $Id: mkconf 261 2006-11-09 13:32:35Z madduck $
>
>
>
> --
> Janek Kozicki                                                         |
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help - Urgent!!!

[lrhorer]

	I updated homehost, but I keep getting the following:



RAID-Server:/etc/cron.d# mdadm -G /dev/md0 -b internal
mdadm: failed to set internal bitmap.
RAID-Server:/etc/cron.d# /usr/share/mdadm/checkarray /dev/md0
/usr/share/mdadm/checkarray: line 167: echo: write error: Device or resource 
busy

Re: Help - Urgent!!!

[lrhorer]

On Friday 13 November 2009 04:13:58 pm you wrote:
> On Friday November 13, lrhorer@satx.rr.com wrote:
> > > There is nothing to delete.  Just try to add it again.
> > > Though I would really like to know why it hangs.  If that happens
> > > again, see if alt-sysrq-T shows anything useful.
> >
> > 	sysrq?
>
> It is a key on the keyboard - up near "Prnt Scrn"
>
> Hold the 'alt' key, hold the 'sysrq' key, then type 't'.
> It spews lot of stack traces on to the screen.  It is really only
> useful with a series console, of if the machine is working well enough
> for all the text to go in to log files.
>
> alt-sysrq-P is less verbose and can also be useful.

	I'm on an X login.  It produces nothing.  Does it only work with a hung 
command line?

Re: Help - Urgent!!!

[Majed B.]

Hey there,

Seems like this is an old bug. Check this:
http://www.issociate.de/board/post/328238/Unable_to_add_write-intent_bitmap.html

I faced some weird problems before and when posting here, Neil Brown
suggested that I try mdadm 3.0 -- I compiled and everything went
smooth from there.

On Sat, Nov 14, 2009 at 3:30 AM, lrhorer@satx.rr.com
<lrhorer@satx.rr.com> wrote:
> On Friday 13 November 2009 04:13:58 pm you wrote:
>> On Friday November 13, lrhorer@satx.rr.com wrote:
>> > > There is nothing to delete.  Just try to add it again.
>> > > Though I would really like to know why it hangs.  If that happens
>> > > again, see if alt-sysrq-T shows anything useful.
>> >
>> >     sysrq?
>>
>> It is a key on the keyboard - up near "Prnt Scrn"
>>
>> Hold the 'alt' key, hold the 'sysrq' key, then type 't'.
>> It spews lot of stack traces on to the screen.  It is really only
>> useful with a series console, of if the machine is working well enough
>> for all the text to go in to log files.
>>
>> alt-sysrq-P is less verbose and can also be useful.
>
>        I'm on an X login.  It produces nothing.  Does it only work with a hung
> command line?
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Help - Urgent!!!

[Leslie Rhorer]

> > 	Should I stop the arraya nd re-assemble it with -vvv?
> >
> > 	Also, what about my question about the bitmap?  Should I delete it,
> despite
> > the fact mdadm says there isn't one, or should I attempt to add it
> again, or
> > should I wait until I do the checkarrray?
> >
> 
> There is nothing to delete.  Just try to add it again.
> Though I would really like to know why it hangs.  If that happens
> again, see if alt-sysrq-T shows anything useful.

	Well, it hasn't hung again, but now that I can catch my breath I can
provide a little detail.  As I mentioned before, the original hang occurred
when I first issued:

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`

	Mdadm complained about the bitmap chunk being an incorrect size.  I
don't know if it actually created a bitmap anyway using the default size, or
what, but I assumed the error meant the bitmap was not created.  Going on
this assumption, I immediately issued the command again, changing only the
bitmap chunk size:

`mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`

	That's when it hung.  I tried stopping the array, but I couldn't
umount the file system because it was dirty.  Any commands to mdadm - even
informational ones and even 

`cat /proc/mdadm`

	would cause the xterm to hang.  Even a shutdown would not work.  I
had to shut down  the server with the power switch.

	When the system booted, mdadm would not assemble the array.  I think
Neil has figured out why.  Subsequent attempts to assemble the array also
failed, at least in part because /dev/md0 had been created without the array
being active.  I deleted /dev/md0, but subsequent failed assembly attempts
again created the file.  Finally I deleted the file once again and then
assembled using

`mdadm -A /dev/sd[a-j]`

	and it worked, although the array was marked as dirty, and the first
attempt to access it caused a resync.  When the resync was done, I fixed the
"homehost" issue Neil had spotted and corrected /etc/mdadm/mdadm.conf to
show all 10 drives.  Finally, I added back the bitmap, and now everything
seems fine.  I rebooted the server, and it comes up with all 10 drives in an
assembled array with an internal bitmap.

Re: Help - Urgent!!!

[Majed B.]

Congratulations :D

Out of curiosity, how did you fix the homehost issue?

On Sun, Nov 15, 2009 at 12:45 AM, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>> >     Should I stop the arraya nd re-assemble it with -vvv?
>> >
>> >     Also, what about my question about the bitmap?  Should I delete it,
>> despite
>> > the fact mdadm says there isn't one, or should I attempt to add it
>> again, or
>> > should I wait until I do the checkarrray?
>> >
>>
>> There is nothing to delete.  Just try to add it again.
>> Though I would really like to know why it hangs.  If that happens
>> again, see if alt-sysrq-T shows anything useful.
>
>        Well, it hasn't hung again, but now that I can catch my breath I can
> provide a little detail.  As I mentioned before, the original hang occurred
> when I first issued:
>
> `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`
>
>        Mdadm complained about the bitmap chunk being an incorrect size.  I
> don't know if it actually created a bitmap anyway using the default size, or
> what, but I assumed the error meant the bitmap was not created.  Going on
> this assumption, I immediately issued the command again, changing only the
> bitmap chunk size:
>
> `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`
>
>        That's when it hung.  I tried stopping the array, but I couldn't
> umount the file system because it was dirty.  Any commands to mdadm - even
> informational ones and even
>
> `cat /proc/mdadm`
>
>        would cause the xterm to hang.  Even a shutdown would not work.  I
> had to shut down  the server with the power switch.
>
>        When the system booted, mdadm would not assemble the array.  I think
> Neil has figured out why.  Subsequent attempts to assemble the array also
> failed, at least in part because /dev/md0 had been created without the array
> being active.  I deleted /dev/md0, but subsequent failed assembly attempts
> again created the file.  Finally I deleted the file once again and then
> assembled using
>
> `mdadm -A /dev/sd[a-j]`
>
>        and it worked, although the array was marked as dirty, and the first
> attempt to access it caused a resync.  When the resync was done, I fixed the
> "homehost" issue Neil had spotted and corrected /etc/mdadm/mdadm.conf to
> show all 10 drives.  Finally, I added back the bitmap, and now everything
> seems fine.  I rebooted the server, and it comes up with all 10 drives in an
> assembled array with an internal bitmap.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Help - Urgent!!!

[Leslie Rhorer]

> Congratulations :D

	Thanks.

> Out of curiosity, how did you fix the homehost issue?

`mdadm -A /dev/md0 /dev/sd[a-j] --update=homehost`