* Help - Urgent!!!
@ 2009-11-13 4:53 lrhorer
2009-11-13 5:11 ` Neil Brown
[not found] ` <200911130913.48292.lrhorer@satx.rr.com>
0 siblings, 2 replies; 17+ messages in thread
From: lrhorer @ 2009-11-13 4:53 UTC (permalink / raw)
To: linux-raid
Damn!!!
I was doing some testing at the behest of Doug, when I made a
critical error which locked up the system and now the main array won't
assemble, at all. I issued the command
`mdadm -G /dev/md0 -b internal --bitmap-chunk=65535`
And the system complained it was an invalid chunk-size, so I
tried:
`mdadm -G /dev/md0 -b internal --bitmap-chunk=65536`
This time there was no error, but the command line just hung
solid. I tired everything I could to recover, but the system just went
downhill more and more. Finally I tried a shutdown, but even that would
not work. I had to hit the big red switch, and now when I rebooted the
system, mdadm will not assemble the array no matter what I try. I tried
`mdadm --assemble /dev/md0 -f`
`mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx`
etc. It just keeps saying, "No devices found for md0". When I
issue the --examine switch, the information all looks good for all the
drives, just no RAID. What do I do, now?
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: Help - Urgent!!! 2009-11-13 4:53 Help - Urgent!!! lrhorer @ 2009-11-13 5:11 ` Neil Brown 2009-11-13 5:12 ` Neil Brown [not found] ` <200911130913.48292.lrhorer@satx.rr.com> 1 sibling, 1 reply; 17+ messages in thread From: Neil Brown @ 2009-11-13 5:11 UTC (permalink / raw) To: lrhorer@satx.rr.com; +Cc: linux-raid On Thursday November 12, lrhorer@satx.rr.com wrote: > Damn!!! > > I was doing some testing at the behest of Doug, when I made a > critical error which locked up the system and now the main array won't > assemble, at all. I issued the command > > `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535` > > And the system complained it was an invalid chunk-size, so I > tried: > > `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536` > > This time there was no error, but the command line just hung > solid. I tired everything I could to recover, but the system just went > downhill more and more. Finally I tried a shutdown, but even that would > not work. I had to hit the big red switch, and now when I rebooted the > system, mdadm will not assemble the array no matter what I try. I tried > > > `mdadm --assemble /dev/md0 -f` > > `mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx` > > etc. It just keeps saying, "No devices found for md0". When I > issue the --examine switch, the information all looks good for all the > drives, just no RAID. What do I do, now? 1/ show us what the result of --examine was. 2/ add -vvv to the --assemble and report the result of that. NeilBrown ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 5:11 ` Neil Brown @ 2009-11-13 5:12 ` Neil Brown 2009-11-13 5:14 ` lrhorer 2009-11-13 5:16 ` lrhorer 0 siblings, 2 replies; 17+ messages in thread From: Neil Brown @ 2009-11-13 5:12 UTC (permalink / raw) To: lrhorer@satx.rr.com, linux-raid On Friday November 13, neilb@suse.de wrote: > > 1/ show us what the result of --examine was. > 2/ add -vvv to the --assemble and report the result of that. 3/ report the output of "mdadm -X /dev/list-of-devices" NeilBrown ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 5:12 ` Neil Brown @ 2009-11-13 5:14 ` lrhorer 2009-11-13 6:08 ` Neil Brown 2009-11-13 5:16 ` lrhorer 1 sibling, 1 reply; 17+ messages in thread From: lrhorer @ 2009-11-13 5:14 UTC (permalink / raw) To: linux-raid On Thursday 12 November 2009 11:12:30 pm you wrote: > On Friday November 13, neilb@suse.de wrote: > > 1/ show us what the result of --examine was. > > 2/ add -vvv to the --assemble and report the result of that. > > 3/ report the output of "mdadm -X /dev/list-of-devices" RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j] /dev/sda: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : RAID-Server:0 (local to host RAID-Server) Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : d40c9255:cef0739f:966d448d:e549ada8 Update Time : Thu Nov 12 21:34:18 2009 Checksum : 9a4e478a - correct Events : 489251 Chunk Size : 256K Array Slot : 0 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : Uuuuuuuuuu /dev/sdb: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : b38a638b:ec546588:4f9a4c8b:16886fe6 Update Time : Thu Nov 12 21:34:18 2009 Checksum : 24e4ca52 - correct Events : 489251 Chunk Size : 256K Array Slot : 1 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uUuuuuuuuu /dev/sdc: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 5158e257:4f166c7f:f5abf1b6:5341dff8 Update Time : Thu Nov 12 21:34:18 2009 Checksum : 267f2437 - correct Events : 489251 Chunk Size : 256K Array Slot : 2 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuUuuuuuuu /dev/sdd: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : bd1b5e0a:a34494e8:c445f3e0:7cf0bd2d Update Time : Thu Nov 12 21:34:18 2009 Checksum : a1035eef - correct Events : 489251 Chunk Size : 256K Array Slot : 3 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuUuuuuuu /dev/sde: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : baf86255:b30938e9:edaf8c86:87b4f49a Update Time : Thu Nov 12 21:34:18 2009 Checksum : ff7c2f31 - correct Events : 489251 Chunk Size : 256K Array Slot : 4 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuUuuuuu /dev/sdf: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 8e2c9eac:617f70f4:5bd00aa3:f59ef3cf Update Time : Thu Nov 12 21:34:18 2009 Checksum : b36ce391 - correct Events : 489251 Chunk Size : 256K Array Slot : 5 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuuUuuuu /dev/sdg: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 0b5819b7:82845162:3e06a649:f7d4664b Update Time : Thu Nov 12 21:34:18 2009 Checksum : 4dd78014 - correct Events : 489251 Chunk Size : 256K Array Slot : 6 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuuuUuuu /dev/sdh: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 654fa4ee:290ddbd4:c171e2c5:7b57fdd8 Update Time : Thu Nov 12 21:34:18 2009 Checksum : 1beee1f - correct Events : 489251 Chunk Size : 256K Array Slot : 7 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuuuuUuu /dev/sdi: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 51f8418f:e2dd7c93:6a898b0c:74d6661a Update Time : Thu Nov 12 21:34:18 2009 Checksum : e910fe64 - correct Events : 489251 Chunk Size : 256K Array Slot : 8 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuuuuuUu /dev/sdj: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca Name : 'RAID-Server':0 Creation Time : Sat Apr 25 01:17:12 2009 Raid Level : raid6 Raid Devices : 10 Avail Dev Size : 1953524896 (931.51 GiB 1000.20 GB) Array Size : 15628197888 (7452.11 GiB 8001.64 GB) Used Dev Size : 1953524736 (931.51 GiB 1000.20 GB) Data Offset : 272 sectors Super Offset : 8 sectors State : active Device UUID : 42993978:e422dd03:9417b109:cdfd6b07 Update Time : Thu Nov 12 21:34:18 2009 Checksum : 2c9399db - correct Events : 489251 Chunk Size : 256K Array Slot : 9 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) Array State : uuuuuuuuuU ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 5:14 ` lrhorer @ 2009-11-13 6:08 ` Neil Brown 2009-11-13 6:22 ` lrhorer 2009-11-13 9:06 ` Janek Kozicki 0 siblings, 2 replies; 17+ messages in thread From: Neil Brown @ 2009-11-13 6:08 UTC (permalink / raw) To: lrhorer@satx.rr.com; +Cc: linux-raid On Thursday November 12, lrhorer@satx.rr.com wrote: > On Thursday 12 November 2009 11:12:30 pm you wrote: > > On Friday November 13, neilb@suse.de wrote: > > > 1/ show us what the result of --examine was. > > > 2/ add -vvv to the --assemble and report the result of that. > > > > 3/ report the output of "mdadm -X /dev/list-of-devices" > > > > RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j] > /dev/sda: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x0 > Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca > Name : RAID-Server:0 (local to host RAID-Server) Strangely this line is unique. Every other device has: > Name : 'RAID-Server':0 Where do those "'"s come from??? What exactly is listed in mdadm.conf for /dev/md0?? Other than that everything looks OK. The array hasn't been marked as having a bitmap, but the bitmap seems to have been created correctly, but not activated yet. And you didn't provide the output of mdadm --assemble -vvv .... That really should explain what is happening. NeilBrown ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 6:08 ` Neil Brown @ 2009-11-13 6:22 ` lrhorer 2009-11-13 7:03 ` Neil Brown 2009-11-13 9:06 ` Janek Kozicki 1 sibling, 1 reply; 17+ messages in thread From: lrhorer @ 2009-11-13 6:22 UTC (permalink / raw) To: linux-raid On Friday 13 November 2009 12:08:47 am you wrote: > On Thursday November 12, lrhorer@satx.rr.com wrote: > > On Thursday 12 November 2009 11:12:30 pm you wrote: > > > On Friday November 13, neilb@suse.de wrote: > > > > 1/ show us what the result of --examine was. > > > > 2/ add -vvv to the --assemble and report the result of that. > > > > > > 3/ report the output of "mdadm -X /dev/list-of-devices" > > > > RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j] > > /dev/sda: > > Magic : a92b4efc > > Version : 1.2 > > Feature Map : 0x0 > > Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca > > Name : RAID-Server:0 (local to host RAID-Server) > > Strangely this line is unique. Every other device has: > > Name : 'RAID-Server':0 > > Where do those "'"s come from??? I have no idea. > What exactly is listed in mdadm.conf for /dev/md0?? cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default, scan all partitions (/proc/partitions) for MD superblocks. # alternatively, specify devices to scan, using wildcards if desired. DEVICE /dev/sd[b-j] # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR lrhorer@satx.rr.com # definitions of existing MD arrays # ARRAY /dev/md0 level=raid6 num-devices=10 UUID=f3b19017:f584ad54:4cb502e9:37bde716 # This file was auto-generated on Fri, 21 Nov 2008 22:35:30 -0600 # by mkconf $Id$ PROGRAM /usr/bin/mdadm_notify ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=10 UUID=5ff10d73:a096195f:7a646bba:a68986ca name=RAID-Server:0 I just spotted something not quite right. Line 7 is missing a drive (/dev/sda). That shouldn't prevent it from assembling, though. > Other than that everything looks OK. The array hasn't been marked as > having a bitmap, but the bitmap seems to have been created correctly, > but not activated yet. > > And you didn't provide the output of > mdadm --assemble -vvv .... Oh, sorry! I thought the fact I got it to assemble meant I didn't need to issue the command again... Indeed, it won't let me issue it again. it says: RAID-Server:/etc/cron.d# mdadm -A /dev/md0 /dev/sd[a-j] -vvv mdadm: device /dev/md0 already active - cannot assemble it Should I stop the arraya nd re-assemble it with -vvv? Also, what about my question about the bitmap? Should I delete it, despite the fact mdadm says there isn't one, or should I attempt to add it again, or should I wait until I do the checkarrray? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 6:22 ` lrhorer @ 2009-11-13 7:03 ` Neil Brown 2009-11-13 15:09 ` lrhorer 2009-11-14 21:45 ` Leslie Rhorer 0 siblings, 2 replies; 17+ messages in thread From: Neil Brown @ 2009-11-13 7:03 UTC (permalink / raw) To: lrhorer@satx.rr.com; +Cc: linux-raid, Debian mdadm maintainers (cced: to debian mdadm maintainers as I there is a problem caused by an older version of the mdadm package that you should be aware of). On Friday November 13, lrhorer@satx.rr.com wrote: > On Friday 13 November 2009 12:08:47 am you wrote: > > On Thursday November 12, lrhorer@satx.rr.com wrote: > > > On Thursday 12 November 2009 11:12:30 pm you wrote: > > > > On Friday November 13, neilb@suse.de wrote: > > > > > 1/ show us what the result of --examine was. > > > > > 2/ add -vvv to the --assemble and report the result of that. > > > > > > > > 3/ report the output of "mdadm -X /dev/list-of-devices" > > > > > > RAID-Server:/etc/cron.d# mdadm --examine /dev/sd[a-j] > > > /dev/sda: > > > Magic : a92b4efc > > > Version : 1.2 > > > Feature Map : 0x0 > > > Array UUID : 5ff10d73:a096195f:7a646bba:a68986ca > > > Name : RAID-Server:0 (local to host RAID-Server) > > > > Strangely this line is unique. Every other device has: > > > Name : 'RAID-Server':0 > > > > Where do those "'"s come from??? > > I have no idea. Well we need to find out because I think they are the problem. This is Debian, isn't it??? > > > What exactly is listed in mdadm.conf for /dev/md0?? > > cat /etc/mdadm/mdadm.conf > # mdadm.conf > # > # Please refer to mdadm.conf(5) for information about this file. > # > > # by default, scan all partitions (/proc/partitions) for MD superblocks. > # alternatively, specify devices to scan, using wildcards if desired. > DEVICE /dev/sd[b-j] As you note, this a problem. mdadm will only look at these devices, not at /dev/sda. And... > ARRAY /dev/md/0 level=raid6 metadata=1.2 num-devices=10 > UUID=5ff10d73:a096195f:7a646bba:a68986ca name=RAID-Server:0 /dev/sda is the only one called RAID-Server:0 The others are all called 'RAID-Server':0 so they won't match. That is why it won't find these devices. I can see how this happened now. There Debian mdadm package (not the most recent, but some fairly recent) has extra_args="--homehost='$MD_HOMEHOST' --auto-update-homehost" in /usr/share/initramfs-tools/scripts/local-top/mdadm This shows a misunderstanding of shell quoting and exactly caused your problem. It updated the homehost on all device that were listed in mdadm.conf to be 'RAID-Server'. So b-j got updated but a did not. Presumably the last time you booted /dev/sda didn't get included, so you had to add it by hand. > > I just spotted something not quite right. Line 7 is missing a drive > (/dev/sda). That shouldn't prevent it from assembling, though. > > > Other than that everything looks OK. The array hasn't been marked as > > having a bitmap, but the bitmap seems to have been created correctly, > > but not activated yet. > > > > And you didn't provide the output of > > mdadm --assemble -vvv .... > > Oh, sorry! I thought the fact I got it to assemble meant I didn't need to > issue the command again... I didn't realise that you had got it to assemble. There is no need to repeat the command as I see exactly what is happening. However to fix it you will need to stop the array and assemble with --update=homehost > > Indeed, it won't let me issue it again. it says: > > RAID-Server:/etc/cron.d# mdadm -A /dev/md0 /dev/sd[a-j] -vvv > mdadm: device /dev/md0 already active - cannot assemble it > > Should I stop the arraya nd re-assemble it with -vvv? > > Also, what about my question about the bitmap? Should I delete it, despite > the fact mdadm says there isn't one, or should I attempt to add it again, or > should I wait until I do the checkarrray? > There is nothing to delete. Just try to add it again. Though I would really like to know why it hangs. If that happens again, see if alt-sysrq-T shows anything useful. NeilBrown ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 7:03 ` Neil Brown @ 2009-11-13 15:09 ` lrhorer 2009-11-14 21:45 ` Leslie Rhorer 1 sibling, 0 replies; 17+ messages in thread From: lrhorer @ 2009-11-13 15:09 UTC (permalink / raw) To: linux-raid I updated homehost, but I keep getting the following: RAID-Server:/etc/cron.d# mdadm -G /dev/md0 -b internal mdadm: failed to set internal bitmap. RAID-Server:/etc/cron.d# /usr/share/mdadm/checkarray /dev/md0 /usr/share/mdadm/checkarray: line 167: echo: write error: Device or resource busy ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: Help - Urgent!!! 2009-11-13 7:03 ` Neil Brown 2009-11-13 15:09 ` lrhorer @ 2009-11-14 21:45 ` Leslie Rhorer 2009-11-15 0:05 ` Majed B. 1 sibling, 1 reply; 17+ messages in thread From: Leslie Rhorer @ 2009-11-14 21:45 UTC (permalink / raw) To: 'Neil Brown'; +Cc: linux-raid, 'Debian mdadm maintainers' > > Should I stop the arraya nd re-assemble it with -vvv? > > > > Also, what about my question about the bitmap? Should I delete it, > despite > > the fact mdadm says there isn't one, or should I attempt to add it > again, or > > should I wait until I do the checkarrray? > > > > There is nothing to delete. Just try to add it again. > Though I would really like to know why it hangs. If that happens > again, see if alt-sysrq-T shows anything useful. Well, it hasn't hung again, but now that I can catch my breath I can provide a little detail. As I mentioned before, the original hang occurred when I first issued: `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535` Mdadm complained about the bitmap chunk being an incorrect size. I don't know if it actually created a bitmap anyway using the default size, or what, but I assumed the error meant the bitmap was not created. Going on this assumption, I immediately issued the command again, changing only the bitmap chunk size: `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536` That's when it hung. I tried stopping the array, but I couldn't umount the file system because it was dirty. Any commands to mdadm - even informational ones and even `cat /proc/mdadm` would cause the xterm to hang. Even a shutdown would not work. I had to shut down the server with the power switch. When the system booted, mdadm would not assemble the array. I think Neil has figured out why. Subsequent attempts to assemble the array also failed, at least in part because /dev/md0 had been created without the array being active. I deleted /dev/md0, but subsequent failed assembly attempts again created the file. Finally I deleted the file once again and then assembled using `mdadm -A /dev/sd[a-j]` and it worked, although the array was marked as dirty, and the first attempt to access it caused a resync. When the resync was done, I fixed the "homehost" issue Neil had spotted and corrected /etc/mdadm/mdadm.conf to show all 10 drives. Finally, I added back the bitmap, and now everything seems fine. I rebooted the server, and it comes up with all 10 drives in an assembled array with an internal bitmap. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-14 21:45 ` Leslie Rhorer @ 2009-11-15 0:05 ` Majed B. 2009-11-15 1:05 ` Leslie Rhorer 0 siblings, 1 reply; 17+ messages in thread From: Majed B. @ 2009-11-15 0:05 UTC (permalink / raw) To: Leslie Rhorer; +Cc: linux-raid Congratulations :D Out of curiosity, how did you fix the homehost issue? On Sun, Nov 15, 2009 at 12:45 AM, Leslie Rhorer <lrhorer@satx.rr.com> wrote: >> > Should I stop the arraya nd re-assemble it with -vvv? >> > >> > Also, what about my question about the bitmap? Should I delete it, >> despite >> > the fact mdadm says there isn't one, or should I attempt to add it >> again, or >> > should I wait until I do the checkarrray? >> > >> >> There is nothing to delete. Just try to add it again. >> Though I would really like to know why it hangs. If that happens >> again, see if alt-sysrq-T shows anything useful. > > Well, it hasn't hung again, but now that I can catch my breath I can > provide a little detail. As I mentioned before, the original hang occurred > when I first issued: > > `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535` > > Mdadm complained about the bitmap chunk being an incorrect size. I > don't know if it actually created a bitmap anyway using the default size, or > what, but I assumed the error meant the bitmap was not created. Going on > this assumption, I immediately issued the command again, changing only the > bitmap chunk size: > > `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536` > > That's when it hung. I tried stopping the array, but I couldn't > umount the file system because it was dirty. Any commands to mdadm - even > informational ones and even > > `cat /proc/mdadm` > > would cause the xterm to hang. Even a shutdown would not work. I > had to shut down the server with the power switch. > > When the system booted, mdadm would not assemble the array. I think > Neil has figured out why. Subsequent attempts to assemble the array also > failed, at least in part because /dev/md0 had been created without the array > being active. I deleted /dev/md0, but subsequent failed assembly attempts > again created the file. Finally I deleted the file once again and then > assembled using > > `mdadm -A /dev/sd[a-j]` > > and it worked, although the array was marked as dirty, and the first > attempt to access it caused a resync. When the resync was done, I fixed the > "homehost" issue Neil had spotted and corrected /etc/mdadm/mdadm.conf to > show all 10 drives. Finally, I added back the bitmap, and now everything > seems fine. I rebooted the server, and it comes up with all 10 drives in an > assembled array with an internal bitmap. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Majed B. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: Help - Urgent!!! 2009-11-15 0:05 ` Majed B. @ 2009-11-15 1:05 ` Leslie Rhorer 0 siblings, 0 replies; 17+ messages in thread From: Leslie Rhorer @ 2009-11-15 1:05 UTC (permalink / raw) To: 'Majed B.'; +Cc: linux-raid > Congratulations :D Thanks. > Out of curiosity, how did you fix the homehost issue? `mdadm -A /dev/md0 /dev/sd[a-j] --update=homehost` ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 6:08 ` Neil Brown 2009-11-13 6:22 ` lrhorer @ 2009-11-13 9:06 ` Janek Kozicki 2009-11-13 9:44 ` Sujit K M 1 sibling, 1 reply; 17+ messages in thread From: Janek Kozicki @ 2009-11-13 9:06 UTC (permalink / raw) To: linux-raid Neil Brown said: (by the date of Fri, 13 Nov 2009 17:08:47 +1100) > Where do those "'"s come from??? I also have those "'" on one (out of 3) md's. Maybe it's debian specific? Or only debian etch specific? Or only debian /dev/md0 specific.... It's only on /dev/md0, like this: backup:~# mdadm -D /dev/md0 /dev/md0: Version : 01.00.03 Creation Time : Wed Oct 31 09:13:41 2007 Raid Level : raid1 Array Size : 979924 (957.12 MiB 1003.44 MB) Device Size : 979924 (957.12 MiB 1003.44 MB) Raid Devices : 5 Total Devices : 5 Preferred Minor : 0 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Nov 13 10:04:02 2009 State : active Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Name : 'backup':0 UUID : 75b0f878:79539d6c:eef22092:f47a6e6f Events : 482808 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 5 22 1 1 active sync /dev/hdc1 2 3 1 2 active sync /dev/hda1 4 8 33 3 active sync /dev/sdc1 6 8 17 4 active sync /dev/sdb1 > > What exactly is listed in mdadm.conf for /dev/md0?? nothing. backup:~# cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default, scan all partitions (/proc/partitions) for MD superblocks. # alternatively, specify devices to scan, using wildcards if desired. DEVICE partitions # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays # This file was auto-generated on Tue, 23 Oct 2007 22:18:08 +0200 # by mkconf $Id: mkconf 261 2006-11-09 13:32:35Z madduck $ -- Janek Kozicki | ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 9:06 ` Janek Kozicki @ 2009-11-13 9:44 ` Sujit K M 0 siblings, 0 replies; 17+ messages in thread From: Sujit K M @ 2009-11-13 9:44 UTC (permalink / raw) To: linux-raid Hi, This seems to be an reported bug that neil seems to be following. Not much helpfull because(Kindly Read the entire mail chain) responder had not updated on the mail. But seems to have the solution/seems to be closed. http://lists.alioth.debian.org/pipermail/pkg-mdadm-devel/2009-June/002945.html Thanks, Sujit On Fri, Nov 13, 2009 at 2:36 PM, Janek Kozicki <janek_listy@wp.pl> wrote: > Neil Brown said: (by the date of Fri, 13 Nov 2009 17:08:47 +1100) > >> Where do those "'"s come from??? > > I also have those "'" on one (out of 3) md's. Maybe it's debian > specific? Or only debian etch specific? Or only debian /dev/md0 > specific.... It's only on /dev/md0, like this: > > backup:~# mdadm -D /dev/md0 > /dev/md0: > Version : 01.00.03 > Creation Time : Wed Oct 31 09:13:41 2007 > Raid Level : raid1 > Array Size : 979924 (957.12 MiB 1003.44 MB) > Device Size : 979924 (957.12 MiB 1003.44 MB) > Raid Devices : 5 > Total Devices : 5 > Preferred Minor : 0 > Persistence : Superblock is persistent > > Intent Bitmap : Internal > > Update Time : Fri Nov 13 10:04:02 2009 > State : active > Active Devices : 5 > Working Devices : 5 > Failed Devices : 0 > Spare Devices : 0 > > Name : 'backup':0 > UUID : 75b0f878:79539d6c:eef22092:f47a6e6f > Events : 482808 > > Number Major Minor RaidDevice State > 0 8 1 0 active sync /dev/sda1 > 5 22 1 1 active sync /dev/hdc1 > 2 3 1 2 active sync /dev/hda1 > 4 8 33 3 active sync /dev/sdc1 > 6 8 17 4 active sync /dev/sdb1 > > >> >> What exactly is listed in mdadm.conf for /dev/md0?? > > nothing. > > backup:~# cat /etc/mdadm/mdadm.conf > # mdadm.conf > # > # Please refer to mdadm.conf(5) for information about this file. > # > > # by default, scan all partitions (/proc/partitions) for MD superblocks. > # alternatively, specify devices to scan, using wildcards if desired. > DEVICE partitions > > # auto-create devices with Debian standard permissions > CREATE owner=root group=disk mode=0660 auto=yes > > # automatically tag new arrays as belonging to the local system > HOMEHOST <system> > > # instruct the monitoring daemon where to send mail alerts > MAILADDR root > > # definitions of existing MD arrays > > # This file was auto-generated on Tue, 23 Oct 2007 22:18:08 +0200 > # by mkconf $Id: mkconf 261 2006-11-09 13:32:35Z madduck $ > > > > -- > Janek Kozicki | > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- -- Sujit K M blog(http://kmsujit.blogspot.com/) -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-13 5:12 ` Neil Brown 2009-11-13 5:14 ` lrhorer @ 2009-11-13 5:16 ` lrhorer 1 sibling, 0 replies; 17+ messages in thread From: lrhorer @ 2009-11-13 5:16 UTC (permalink / raw) To: linux-raid On Thursday 12 November 2009 11:12:30 pm you wrote: > mdadm -X /dev/list-of-devices RAID-Server:/etc/cron.d# mdadm -X /dev/sd[a-j] Filename : /dev/sda Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdb Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdc Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdd Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sde Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdf Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdg Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdh Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdi Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) Filename : /dev/sdj Magic : 6d746962 Version : 4 UUID : 5ff10d73:a096195f:7a646bba:a68986ca Events : 0 Events Cleared : 0 State : OK Chunksize : 64 MB Daemon : 5s flush period Write Mode : Normal Sync Size : 976762368 (931.51 GiB 1000.20 GB) Bitmap : 14905 bits (chunks), 14904 dirty (100.0%) ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <200911130913.48292.lrhorer@satx.rr.com>]
[parent not found: <19197.55846.756558.21650@notabene.brown>]
* Re: Help - Urgent!!! [not found] ` <19197.55846.756558.21650@notabene.brown> @ 2009-11-14 0:30 ` lrhorer 2009-11-14 0:34 ` Majed B. 0 siblings, 1 reply; 17+ messages in thread From: lrhorer @ 2009-11-14 0:30 UTC (permalink / raw) To: linux-raid On Friday 13 November 2009 04:13:58 pm you wrote: > On Friday November 13, lrhorer@satx.rr.com wrote: > > > There is nothing to delete. Just try to add it again. > > > Though I would really like to know why it hangs. If that happens > > > again, see if alt-sysrq-T shows anything useful. > > > > sysrq? > > It is a key on the keyboard - up near "Prnt Scrn" > > Hold the 'alt' key, hold the 'sysrq' key, then type 't'. > It spews lot of stack traces on to the screen. It is really only > useful with a series console, of if the machine is working well enough > for all the text to go in to log files. > > alt-sysrq-P is less verbose and can also be useful. I'm on an X login. It produces nothing. Does it only work with a hung command line? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Help - Urgent!!! 2009-11-14 0:30 ` lrhorer @ 2009-11-14 0:34 ` Majed B. 0 siblings, 0 replies; 17+ messages in thread From: Majed B. @ 2009-11-14 0:34 UTC (permalink / raw) To: lrhorer@satx.rr.com; +Cc: linux-raid Hey there, Seems like this is an old bug. Check this: http://www.issociate.de/board/post/328238/Unable_to_add_write-intent_bitmap.html I faced some weird problems before and when posting here, Neil Brown suggested that I try mdadm 3.0 -- I compiled and everything went smooth from there. On Sat, Nov 14, 2009 at 3:30 AM, lrhorer@satx.rr.com <lrhorer@satx.rr.com> wrote: > On Friday 13 November 2009 04:13:58 pm you wrote: >> On Friday November 13, lrhorer@satx.rr.com wrote: >> > > There is nothing to delete. Just try to add it again. >> > > Though I would really like to know why it hangs. If that happens >> > > again, see if alt-sysrq-T shows anything useful. >> > >> > sysrq? >> >> It is a key on the keyboard - up near "Prnt Scrn" >> >> Hold the 'alt' key, hold the 'sysrq' key, then type 't'. >> It spews lot of stack traces on to the screen. It is really only >> useful with a series console, of if the machine is working well enough >> for all the text to go in to log files. >> >> alt-sysrq-P is less verbose and can also be useful. > > I'm on an X login. It produces nothing. Does it only work with a hung > command line? > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Majed B. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 17+ messages in thread
* Help - Urgent! @ 2009-11-13 4:34 Rhorer, Leslie 0 siblings, 0 replies; 17+ messages in thread From: Rhorer, Leslie @ 2009-11-13 4:34 UTC (permalink / raw) To: linux-raid Damn!!! I was doing some testing at the behest of Doug, when I made a critical error which locked up the system and now the main array won't assemble, at all. I issued the command `mdadm -G /dev/md0 -b internal --bitmap-chunk=65535` And the system complained it was an invalid chunk-size, so I tried: `mdadm -G /dev/md0 -b internal --bitmap-chunk=65536` This time there was no error, but the command line just hung solid. I tired everything I could to recover, but the system just went downhill more and more. Finally I tried a shutdown, but even that would not work. I had to hit the big red switch, and now when I rebooted the system, mdadm will not assemble the array no matter what I try. I tried `mdadm --assemble /dev/md0 -f` `mdadm --assemble /dev/md0 -u xxxxxxxx:xxxxxxxxx:xxxxxx:xxxxxx` etc. It just keeps saying, "No devices found for md0". When I issue the --examine switch, the information all looks good for all the drives, just no RAID. What do I do, now? Please respond to me at leslie.rhorer@twtelecom.com, because with the server down I do not have access to the list. --- The content contained in this electronic message is not intended to constitute formation of a contract binding tw telecom. tw telecom will be contractually bound only upon execution, by an authorized officer, of a contract including agreed terms and conditions or by express application of its tariffs. This message is intended only for the use of the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the sender of this E-Mail or by telephone. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2009-11-15 1:05 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-11-13 4:53 Help - Urgent!!! lrhorer
2009-11-13 5:11 ` Neil Brown
2009-11-13 5:12 ` Neil Brown
2009-11-13 5:14 ` lrhorer
2009-11-13 6:08 ` Neil Brown
2009-11-13 6:22 ` lrhorer
2009-11-13 7:03 ` Neil Brown
2009-11-13 15:09 ` lrhorer
2009-11-14 21:45 ` Leslie Rhorer
2009-11-15 0:05 ` Majed B.
2009-11-15 1:05 ` Leslie Rhorer
2009-11-13 9:06 ` Janek Kozicki
2009-11-13 9:44 ` Sujit K M
2009-11-13 5:16 ` lrhorer
[not found] ` <200911130913.48292.lrhorer@satx.rr.com>
[not found] ` <19197.55846.756558.21650@notabene.brown>
2009-11-14 0:30 ` lrhorer
2009-11-14 0:34 ` Majed B.
-- strict thread matches above, loose matches on Subject: below --
2009-11-13 4:34 Help - Urgent! Rhorer, Leslie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox