* raid related kernel hang in 2.6.36
@ 2010-11-24 13:48 Du Jun
2010-11-24 14:19 ` Mathias Burén
0 siblings, 1 reply; 8+ messages in thread
From: Du Jun @ 2010-11-24 13:48 UTC (permalink / raw)
To: linux-raid
Hi,
We just quick tested the raid stability under the vanilla 2.6.36 kernel
and got an unstable result.
first, we create a raid5 array using 16 sata disks:
mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
then use dd to stress the io:
dd if=/dev/zero of=/dev/md10 bs=1M
After a while, usually several minutes, the system hangs. It looks like
some kind of kernel deadlock. ping to this machine could get timely
response, however, any other process just hangs.
It is easily reproducible and everytime the test result is a system hang.
Johnson
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-24 13:48 raid related kernel hang in 2.6.36 Du Jun
@ 2010-11-24 14:19 ` Mathias Burén
2010-11-24 14:34 ` Du Jun
0 siblings, 1 reply; 8+ messages in thread
From: Mathias Burén @ 2010-11-24 14:19 UTC (permalink / raw)
To: Du Jun; +Cc: linux-raid
Hi,
I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds),
mdadm -D md10, zcat /proc/config.gz, mdadm version)
Regards,
// Mathias
2010/11/24 Du Jun <dujun@perabytes.com>:
> Hi,
> We just quick tested the raid stability under the vanilla 2.6.36 kernel
> and got an unstable result.
>
> first, we create a raid5 array using 16 sata disks:
>
> mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
>
> then use dd to stress the io:
>
> dd if=/dev/zero of=/dev/md10 bs=1M
>
> After a while, usually several minutes, the system hangs. It looks like
> some kind of kernel deadlock. ping to this machine could get timely
> response, however, any other process just hangs.
>
> It is easily reproducible and everytime the test result is a system hang.
>
> Johnson
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-24 14:19 ` Mathias Burén
@ 2010-11-24 14:34 ` Du Jun
2010-11-24 20:01 ` Neil Brown
0 siblings, 1 reply; 8+ messages in thread
From: Du Jun @ 2010-11-24 14:34 UTC (permalink / raw)
To: Mathias Burén; +Cc: linux-raid
Their's no dmesg information at all before the hang. The system just
hangs suddenly without any kernel printk or oops.
mdadm version is debian lenny version 2.6.7.2-3
I will get mdadm -E & mdadm -D informations tomorrow but I doubt they
are useful.
Johnson
on 2010/11/24 22:19, Mathias Burén wrote:
> Hi,
>
> I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds),
> mdadm -D md10, zcat /proc/config.gz, mdadm version)
>
> Regards,
> // Mathias
>
> 2010/11/24 Du Jun<dujun@perabytes.com>:
>
>> Hi,
>> We just quick tested the raid stability under the vanilla 2.6.36 kernel
>> and got an unstable result.
>>
>> first, we create a raid5 array using 16 sata disks:
>>
>> mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
>>
>> then use dd to stress the io:
>>
>> dd if=/dev/zero of=/dev/md10 bs=1M
>>
>> After a while, usually several minutes, the system hangs. It looks like
>> some kind of kernel deadlock. ping to this machine could get timely
>> response, however, any other process just hangs.
>>
>> It is easily reproducible and everytime the test result is a system hang.
>>
>> Johnson
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-24 14:34 ` Du Jun
@ 2010-11-24 20:01 ` Neil Brown
2010-11-24 21:15 ` Carlos Carvalho
2010-11-25 3:12 ` dujun
0 siblings, 2 replies; 8+ messages in thread
From: Neil Brown @ 2010-11-24 20:01 UTC (permalink / raw)
To: Du Jun; +Cc: Mathias Burén, linux-raid
On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <dujun@perabytes.com> wrote:
> Their's no dmesg information at all before the hang. The system just
> hangs suddenly without any kernel printk or oops.
>
> mdadm version is debian lenny version 2.6.7.2-3
>
> I will get mdadm -E & mdadm -D informations tomorrow but I doubt they
> are useful.
Probably not, but they wouldn't hurt either.
What would really help is if you could capture the result of
sysrq-T
which I accept might be a challenge depending on how hard the machine really
has frozen.
Also, make sure the LOCKUP_DETECTOR config options are set, and wait a few
minutes after the hand to see if something pops up.
I'll try to see if I can reproduce, but I don't have 16 disks so it may not
work.
NeilBrown
>
> Johnson
> on 2010/11/24 22:19, Mathias Burén wrote:
> > Hi,
> >
> > I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds),
> > mdadm -D md10, zcat /proc/config.gz, mdadm version)
> >
> > Regards,
> > // Mathias
> >
> > 2010/11/24 Du Jun<dujun@perabytes.com>:
> >
> >> Hi,
> >> We just quick tested the raid stability under the vanilla 2.6.36 kernel
> >> and got an unstable result.
> >>
> >> first, we create a raid5 array using 16 sata disks:
> >>
> >> mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
> >>
> >> then use dd to stress the io:
> >>
> >> dd if=/dev/zero of=/dev/md10 bs=1M
> >>
> >> After a while, usually several minutes, the system hangs. It looks like
> >> some kind of kernel deadlock. ping to this machine could get timely
> >> response, however, any other process just hangs.
> >>
> >> It is easily reproducible and everytime the test result is a system hang.
> >>
> >> Johnson
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-24 20:01 ` Neil Brown
@ 2010-11-24 21:15 ` Carlos Carvalho
2010-11-25 3:12 ` dujun
1 sibling, 0 replies; 8+ messages in thread
From: Carlos Carvalho @ 2010-11-24 21:15 UTC (permalink / raw)
To: Du Jun, linux-raid
Neil Brown (neilb@suse.de) wrote on 25 November 2010 07:01:
>On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <dujun@perabytes.com> wrote:
>
>> Their's no dmesg information at all before the hang. The system just
>> hangs suddenly without any kernel printk or oops.
>>
>> mdadm version is debian lenny version 2.6.7.2-3
>>
>> I will get mdadm -E & mdadm -D informations tomorrow but I doubt they
>> are useful.
>
>Probably not, but they wouldn't hurt either.
>
>What would really help is if you could capture the result of
> sysrq-T
>
>which I accept might be a challenge depending on how hard the machine really
>has frozen.
>
>Also, make sure the LOCKUP_DETECTOR config options are set, and wait a few
>minutes after the hand to see if something pops up.
You could also boot with the NMI watchdog. Using netconsole to capture
the full traces would be useful as well.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-24 20:01 ` Neil Brown
2010-11-24 21:15 ` Carlos Carvalho
@ 2010-11-25 3:12 ` dujun
2011-02-10 16:28 ` Iordan Iordanov
1 sibling, 1 reply; 8+ messages in thread
From: dujun @ 2010-11-25 3:12 UTC (permalink / raw)
To: Neil Brown; +Cc: Mathias Burén, linux-raid
[-- Attachment #1: Type: text/plain, Size: 6586 bytes --]
Hi, Neil,
attached is the mdadm -E and mdadm -D information for md10(raid5 with 16
disks) and md11 (raid6 with 16 disks).
I recompiled the 2.6.36 kernel with kernel sysrq and softlock dectet
option, then setup two mds to reproduce the softlock bug.
First create two md using mdadm:
mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
mdadm -C /dev/md10 -l 6 -n 16 /dev/sd[r-z] /dev/sda[a-g]
then use dd :
dd if=/dev/zero of=/dev/md10 bs=1M &
dd if=/dev/zero of=/dev/md11 bs=1M&
After a while, the system is hung with lots of softlock kernel trace on
screen. following are some netconsole output:
Nov 25 11:01:35 10.10.20.20 [ 1236.027001] BUG: soft lockup - CPU#0
stuck for 67s! [top:5690]
Nov 25 11:01:40 10.10.20.20 [ 1236.027001] Stack:
Nov 25 11:01:45 10.10.20.20 [ 1236.027001] Call Trace:
Nov 25 11:01:50 10.10.20.20 [ 1236.027001] Code:
Nov 25 11:01:55 10.10.20.20 03
Nov 25 11:02:00 10.10.20.20 81
Nov 25 11:02:05 10.10.20.20 48
Nov 25 11:02:10 10.10.20.20 c7
Nov 25 11:02:15 10.10.20.20 c1
Nov 25 11:02:20 10.10.20.20 91
Nov 25 11:02:25 10.10.20.20 19
Nov 25 11:02:30 10.10.20.20 03
Nov 25 11:02:35 10.10.20.20 81
Nov 25 11:02:40 10.10.20.20 e9
Nov 25 11:02:45 10.10.20.20 dd
Nov 25 11:02:50 10.10.20.20 fe
Nov 25 11:02:55 10.10.20.20 ff
Nov 25 11:03:00 10.10.20.20 ff
Nov 25 11:03:05 10.10.20.20 90
Nov 25 11:03:10 10.10.20.20 90
Nov 25 11:03:15 10.10.20.20 90
Nov 25 11:03:20 10.10.20.20 90
Nov 25 11:03:25 10.10.20.20 90
Nov 25 11:03:30 10.10.20.20 90
Nov 25 11:03:35 10.10.20.20 90
Nov 25 11:03:40 10.10.20.20 90
Nov 25 11:03:45 10.10.20.20 90
Nov 25 11:03:50 10.10.20.20 90
Nov 25 11:03:55 10.10.20.20 90
Nov 25 11:04:00 10.10.20.20 90
Nov 25 11:04:05 10.10.20.20 90
Nov 25 11:04:10 10.10.20.20 b8
Nov 25 11:04:15 10.10.20.20 00
Nov 25 11:04:20 10.10.20.20 01
Nov 25 11:04:25 10.10.20.20 00
Nov 25 11:04:30 10.10.20.20 00
Nov 25 11:04:35 10.10.20.20 f0
Nov 25 11:04:40 10.10.20.20 66
Nov 25 11:04:45 10.10.20.20 0f
Nov 25 11:04:50 10.10.20.20 c1
Nov 25 11:04:55 10.10.20.20 07
Nov 25 11:05:00 10.10.20.20 38
Nov 25 11:05:05 10.10.20.20 e0
Nov 25 11:05:10 10.10.20.20 74
Nov 25 11:05:15 10.10.20.20 06
Nov 25 11:05:20 10.10.20.20 f3
Nov 25 11:05:25 10.10.20.20 90
Message from syslogd@10.10.20.20 at Nov 25 11:05:30 ...
a>
Nov 25 11:05:35 10.10.20.20 07
Nov 25 11:05:40 10.10.20.20 eb
Nov 25 11:05:45 10.10.20.20 f6
Nov 25 11:05:50 10.10.20.20 c3
Nov 25 11:05:55 10.10.20.20 66
Nov 25 11:06:00 10.10.20.20 66
Nov 25 11:06:05 10.10.20.20 2e
Nov 25 11:06:10 10.10.20.20 0f
Nov 25 11:06:15 10.10.20.20 1f
Nov 25 11:06:20 10.10.20.20 84
Nov 25 11:06:25 10.10.20.20 00
Nov 25 11:06:30 10.10.20.20 00
Nov 25 11:06:35 10.10.20.20 00
Nov 25 11:06:40 10.10.20.20 00
Nov 25 11:06:45 10.10.20.20 00
Nov 25 11:06:50 10.10.20.20 0f
Nov 25 11:06:55 10.10.20.20 b7
Nov 25 11:07:00 10.10.20.20 07
Nov 25 11:07:05 10.10.20.20 38
Nov 25 11:07:10 10.10.20.20 e0
Nov 25 11:07:15 10.10.20.20
Nov 25 11:07:21 10.10.20.20 [ 1236.099000] BUG: soft lockup - CPU#1
stuck for 67s! [pvscan:6172]
Nov 25 11:07:26 10.10.20.20 [ 1236.099001] Stack:
Nov 25 11:07:31 10.10.20.20 [ 1236.099001] Call Trace:
Nov 25 11:07:36 10.10.20.20 [ 1236.099001] Code:
Nov 25 11:07:41 10.10.20.20 39
Nov 25 11:07:46 10.10.20.20 ea
Nov 25 11:07:51 10.10.20.20 0f
Nov 25 11:07:56 10.10.20.20 1f
Nov 25 11:08:01 10.10.20.20 00
Nov 25 11:08:06 10.10.20.20 77
Nov 25 11:08:11 10.10.20.20 a8
Nov 25 11:08:16 10.10.20.20 89
Nov 25 11:08:21 10.10.20.20 de
Nov 25 11:08:26 10.10.20.20 89
Nov 25 11:08:31 10.10.20.20 e9
Nov 25 11:08:36 10.10.20.20 49
Nov 25 11:08:41 10.10.20.20 8d
Nov 25 11:08:46 10.10.20.20 34
Nov 25 11:08:51 10.10.20.20 f7
Nov 25 11:08:56 10.10.20.20 29
Nov 25 11:09:01 10.10.20.20 d9
Nov 25 11:09:06 10.10.20.20 45
Nov 25 11:09:11 10.10.20.20 89
Nov 25 11:09:16 10.10.20.20 e1
Nov 25 11:09:21 10.10.20.20 4c
Nov 25 11:09:26 10.10.20.20 8d
Nov 25 11:09:31 10.10.20.20 44
Nov 25 11:09:36 10.10.20.20 24
Nov 25 11:09:41 10.10.20.20 08
the sysrq-T just print "showstate" on screen. no more useful
information.
thanks.
Johnson
在 2010-11-25四的 07:01 +1100，Neil Brown写道：
> On Wed, 24 Nov 2010 22:34:21 +0800 Du Jun <dujun@perabytes.com> wrote:
>
> > Their's no dmesg information at all before the hang. The system just
> > hangs suddenly without any kernel printk or oops.
> >
> > mdadm version is debian lenny version 2.6.7.2-3
> >
> > I will get mdadm -E & mdadm -D informations tomorrow but I doubt they
> > are useful.
>
> Probably not, but they wouldn't hurt either.
>
> What would really help is if you could capture the result of
> sysrq-T
>
> which I accept might be a challenge depending on how hard the machine really
> has frozen.
>
> Also, make sure the LOCKUP_DETECTOR config options are set, and wait a few
> minutes after the hand to see if something pops up.
>
> I'll try to see if I can reproduce, but I don't have 16 disks so it may not
> work.
>
> NeilBrown
>
>
> >
> > Johnson
> > on 2010/11/24 22:19, Mathias Burén wrote:
> > > Hi,
> > >
> > > I suppose a few logs would be interesting. (dmesg, mdadm -E (hdds),
> > > mdadm -D md10, zcat /proc/config.gz, mdadm version)
> > >
> > > Regards,
> > > // Mathias
> > >
> > > 2010/11/24 Du Jun<dujun@perabytes.com>:
> > >
> > >> Hi,
> > >> We just quick tested the raid stability under the vanilla 2.6.36 kernel
> > >> and got an unstable result.
> > >>
> > >> first, we create a raid5 array using 16 sata disks:
> > >>
> > >> mdadm -C /dev/md10 -l 5 -n 16 /dev/sd[b-q]
> > >>
> > >> then use dd to stress the io:
> > >>
> > >> dd if=/dev/zero of=/dev/md10 bs=1M
> > >>
> > >> After a while, usually several minutes, the system hangs. It looks like
> > >> some kind of kernel deadlock. ping to this machine could get timely
> > >> response, however, any other process just hangs.
> > >>
> > >> It is easily reproducible and everytime the test result is a system hang.
> > >>
> > >> Johnson
> > >>
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > >> the body of a message to majordomo@vger.kernel.org
> > >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >>
> > >>
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> > >
> > >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
[-- Attachment #2: md10.txt --]
[-- Type: text/plain, Size: 1739 bytes --]
/dev/md10:
Version : 00.90
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Persistence : Superblock is persistent
Update Time : Thu Nov 25 10:42:10 2010
State : clean, degraded, recovering
Active Devices : 15
Working Devices : 16
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 2% complete
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Events : 0.4
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 8 64 3 active sync /dev/sde
4 8 80 4 active sync /dev/sdf
5 8 96 5 active sync /dev/sdg
6 8 112 6 active sync /dev/sdh
7 8 128 7 active sync /dev/sdi
8 8 144 8 active sync /dev/sdj
9 8 160 9 active sync /dev/sdk
10 8 176 10 active sync /dev/sdl
11 8 192 11 active sync /dev/sdm
12 8 208 12 active sync /dev/sdn
13 8 224 13 active sync /dev/sdo
14 8 240 14 active sync /dev/sdp
16 65 0 15 spare rebuilding /dev/sdq
[-- Attachment #3: md11.txt --]
[-- Type: text/plain, Size: 1746 bytes --]
/dev/md11:
Version : 00.90
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Persistence : Superblock is persistent
Update Time : Thu Nov 25 10:43:51 2010
State : clean, degraded, recovering
Active Devices : 15
Working Devices : 16
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 2% complete
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Events : 0.2
Number Major Minor RaidDevice State
0 65 16 0 active sync /dev/sdr
1 65 32 1 active sync /dev/sds
2 65 48 2 active sync /dev/sdt
3 65 64 3 active sync /dev/sdu
4 65 80 4 active sync /dev/sdv
5 65 96 5 active sync /dev/sdw
6 65 112 6 active sync /dev/sdx
7 65 128 7 active sync /dev/sdy
8 65 144 8 active sync /dev/sdz
9 65 160 9 active sync /dev/sdaa
10 65 176 10 active sync /dev/sdab
11 65 192 11 active sync /dev/sdac
12 65 208 12 active sync /dev/sdad
13 65 224 13 active sync /dev/sdae
14 65 240 14 active sync /dev/sdaf
16 66 0 15 spare rebuilding /dev/sdag
[-- Attachment #4: sdds.txt --]
[-- Type: text/plain, Size: 59378 bytes --]
/dev/sdaa:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d61 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 65 160 9 active sync /dev/sdaa
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdab:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d73 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 10 65 176 10 active sync /dev/sdab
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdac:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d85 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 11 65 192 11 active sync /dev/sdac
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdad:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d97 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 12 65 208 12 active sync /dev/sdad
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdae:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62da9 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 13 65 224 13 active sync /dev/sdae
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdaf:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62dbb - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 14 65 240 14 active sync /dev/sdaf
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdag:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cca - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 16 66 0 16 spare /dev/sdag
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdb:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4090 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 16 0 active sync /dev/sdb
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdc:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40a2 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 32 1 active sync /dev/sdc
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdd:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40b4 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 48 2 active sync /dev/sdd
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sde:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40c6 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 64 3 active sync /dev/sde
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdf:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40d8 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 80 4 active sync /dev/sdf
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdg:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40ea - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 96 5 active sync /dev/sdg
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdh:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40fc - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 112 6 active sync /dev/sdh
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdi:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a410e - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 128 7 active sync /dev/sdi
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdj:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4120 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 144 8 active sync /dev/sdj
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdk:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4132 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 8 160 9 active sync /dev/sdk
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdl:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4144 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 10 8 176 10 active sync /dev/sdl
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdm:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4156 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 11 8 192 11 active sync /dev/sdm
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdn:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a4168 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 12 8 208 12 active sync /dev/sdn
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdo:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a417a - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 13 8 224 13 active sync /dev/sdo
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdp:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a418c - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 14 8 240 14 active sync /dev/sdp
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdq:
Magic : a92b4efc
Version : 00.90.00
UUID : f7187a5e:3e368fe2:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:24:11 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 10
Update Time : Thu Nov 25 10:51:49 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : aa5a40d3 - correct
Events : 6
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 16 65 0 16 spare /dev/sdq
0 0 8 16 0 active sync /dev/sdb
1 1 8 32 1 active sync /dev/sdc
2 2 8 48 2 active sync /dev/sdd
3 3 8 64 3 active sync /dev/sde
4 4 8 80 4 active sync /dev/sdf
5 5 8 96 5 active sync /dev/sdg
6 6 8 112 6 active sync /dev/sdh
7 7 8 128 7 active sync /dev/sdi
8 8 8 144 8 active sync /dev/sdj
9 9 8 160 9 active sync /dev/sdk
10 10 8 176 10 active sync /dev/sdl
11 11 8 192 11 active sync /dev/sdm
12 12 8 208 12 active sync /dev/sdn
13 13 8 224 13 active sync /dev/sdo
14 14 8 240 14 active sync /dev/sdp
15 15 0 0 15 faulty removed
16 16 65 0 16 spare /dev/sdq
/dev/sdr:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cbf - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 65 16 0 active sync /dev/sdr
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sds:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cd1 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 65 32 1 active sync /dev/sds
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdt:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62ce3 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 65 48 2 active sync /dev/sdt
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdu:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62cf5 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 65 64 3 active sync /dev/sdu
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdv:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d07 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 65 80 4 active sync /dev/sdv
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdw:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d19 - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 65 96 5 active sync /dev/sdw
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdx:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d2b - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 65 112 6 active sync /dev/sdx
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdy:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d3d - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 65 128 7 active sync /dev/sdy
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
/dev/sdz:
Magic : a92b4efc
Version : 00.90.00
UUID : 012dcc5b:f09d21e4:d64d9ae0:217d735e (local to host DigiOcean)
Creation Time : Thu Nov 25 10:43:51 2010
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 14651437440 (13972.70 GiB 15003.07 GB)
Raid Devices : 16
Total Devices : 16
Preferred Minor : 11
Update Time : Thu Nov 25 10:51:55 2010
State : clean
Active Devices : 15
Working Devices : 16
Failed Devices : 1
Spare Devices : 1
Checksum : 66d62d4f - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 65 144 8 active sync /dev/sdz
0 0 65 16 0 active sync /dev/sdr
1 1 65 32 1 active sync /dev/sds
2 2 65 48 2 active sync /dev/sdt
3 3 65 64 3 active sync /dev/sdu
4 4 65 80 4 active sync /dev/sdv
5 5 65 96 5 active sync /dev/sdw
6 6 65 112 6 active sync /dev/sdx
7 7 65 128 7 active sync /dev/sdy
8 8 65 144 8 active sync /dev/sdz
9 9 65 160 9 active sync /dev/sdaa
10 10 65 176 10 active sync /dev/sdab
11 11 65 192 11 active sync /dev/sdac
12 12 65 208 12 active sync /dev/sdad
13 13 65 224 13 active sync /dev/sdae
14 14 65 240 14 active sync /dev/sdaf
15 15 0 0 15 faulty removed
16 16 66 0 16 spare /dev/sdag
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2010-11-25 3:12 ` dujun
@ 2011-02-10 16:28 ` Iordan Iordanov
2011-02-11 15:52 ` Iordan Iordanov
0 siblings, 1 reply; 8+ messages in thread
From: Iordan Iordanov @ 2011-02-10 16:28 UTC (permalink / raw)
To: dujun; +Cc: Neil Brown, Mathias Burén, linux-raid, CDF Admin
Hi Neil,
We are currently using 2.6.36 on a production machine running mdadm, and
this morning we experienced a very similar issue, but we didn't have the
LOCKUP_DETECTOR config options set, so there was absolutely no output.
We also experienced this on a non-production test platform a few days
ago (same hardware and software as our production machine).
Has this bug been diagnosed/resolved? If so, past which kernel version?
If not, can you suggest some recourse for us, like perhaps moving back
to 2.6.35 for our production server, and moving up to the latest stable
on our test platform, and enabling the LOCKUP_DETECTOR options?
Is this bug totally reproducible Du Jun? If so, should I just try what
you outlined in your previous message?
Many thanks,
Iordan Iordanov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: raid related kernel hang in 2.6.36
2011-02-10 16:28 ` Iordan Iordanov
@ 2011-02-11 15:52 ` Iordan Iordanov
0 siblings, 0 replies; 8+ messages in thread
From: Iordan Iordanov @ 2011-02-11 15:52 UTC (permalink / raw)
To: dujun; +Cc: Neil Brown, Mathias Burén, linux-raid, CDF Admin
Hi Neil,
As requested, here is more background.
On 02/10/11 11:28, Iordan Iordanov wrote:
> We are currently using 2.6.36 on a production machine running mdadm
Our distribution is Debian Lenny. We have 10 3-way RAID1 mirrors (30 1TB
disks) striped together with LVM in one large volume group, out of which
we create logical drives. The actual drives are iscsi targets we import
from separate machines. The system is serving about 3500 active (not
logged-in at the same time) accounts to our department. Also, we have a
"development" platform, with no users using it, also with lvm, mdadm,
and iscsi, but with 5 disks instead of 30. The development platform was
sitting idle the whole time.
> this morning we experienced a very similar issue, but we didn't have the
> LOCKUP_DETECTOR config options set, so there was absolutely no output.
What I mean by a similar issue is that our development platform hung
without any output in the logs and was completely unresponsive on Feb 7,
and our production system hung and was completely unresponsive, save for
pings, on Feb 10. I don't know if the two hangs were caused by the same
issue.
I thought that the potential lock-up reported by Du Jun was acknowledged
as reality. I was wrong. We will recompile a longterm kernel (this time)
with the LOCKUP_DETECTOR flags set, and will wait to see if the issue
recurs. Do you have any suggestions to us?
Many thanks,
Iordan
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-02-11 15:52 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-11-24 13:48 raid related kernel hang in 2.6.36 Du Jun
2010-11-24 14:19 ` Mathias Burén
2010-11-24 14:34 ` Du Jun
2010-11-24 20:01 ` Neil Brown
2010-11-24 21:15 ` Carlos Carvalho
2010-11-25 3:12 ` dujun
2011-02-10 16:28 ` Iordan Iordanov
2011-02-11 15:52 ` Iordan Iordanov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).