* md rotates RAID5 spare at boot
@ 2008-01-09 20:07 Jed Davidow
2008-01-10 18:03 ` Jed Davidow
2008-01-10 20:47 ` Bill Davidsen
0 siblings, 2 replies; 12+ messages in thread
From: Jed Davidow @ 2008-01-09 20:07 UTC (permalink / raw)
To: linux-raid
I have a RAID5 (5+1spare) setup that works perfectly well until I
reboot. I have 6 drives (two different models) partitioned to give me 2
arrays, md0 and md1, that I use for /home and /var respectively.
When I reboot, the system assembles each array, but swaps out what was
the spare with one of the member drives. It then immediately detects a
degraded array and rebuilds. After that, all is fine and testing has
shown things to be working like they should. Until I reboot.
Example:
Built two arrays: /dev/md0 -> /dev/sd[abcef]1 and /dev/md1 ->
/dev/sd[abcef]2
Added /dev/sdg1 and /dev/sdg2 as spares, and this works.
One scenario when I reboot:
md0 is assembled from sd[abceg]1; it's degraded and reports a
"spares missing" event.
md1 assembles correctly, spare is not missing
Any ideas? I have asked about this on various boards (some said UDEV
rules would help, some thought the issue had to do with the /dev/sdX
names changing, etc). I don't think those are applicable since dmesg
reports the arrays assemble as soon as the disks are detected.
Thanks in advance!
INFO:
(currently the boot drive (non raid) is sdd, otherwise all sd devices
are part of the raid)
fdisk:
$ sudo fdisk -l
Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 1 1521 12217401 fd Linux raid
autodetect
/dev/sda2 1522 30401 231978600 fd Linux raid
autodetect
Disk /dev/sdb: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 1 1521 12217401 fd Linux raid
autodetect
/dev/sdb2 1522 30401 231978600 fd Linux raid
autodetect
Disk /dev/sdc: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdc1 1 1521 12217401 fd Linux raid
autodetect
/dev/sdc2 1522 30401 231978600 fd Linux raid
autodetect
Disk /dev/md0: 50.0 GB, 50041978880 bytes
2 heads, 4 sectors/track, 12217280 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1: 950.1 GB, 950183919616 bytes
2 heads, 4 sectors/track, 231978496 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/sdd: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x535bfd7a
Device Boot Start End Blocks Id System
/dev/sdd1 * 1 14219 114214086 83 Linux
/dev/sdd2 14220 14593 3004155 5 Extended
/dev/sdd5 14220 14593 3004123+ 82 Linux swap /
Solaris
Disk /dev/sde: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sde1 1 1521 12217401 fd Linux raid
autodetect
/dev/sde2 1522 30401 231978600 fd Linux raid
autodetect
Disk /dev/sdf: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdf1 1 1521 12217401 fd Linux raid
autodetect
/dev/sdf2 1522 30401 231978600 fd Linux raid
autodetect
Disk /dev/sdg: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdg1 1 1521 12217401 fd Linux raid
autodetect
/dev/sdg2 1522 30401 231978600 fd Linux raid
autodetect
$ sudo mdadm --detail /dev/md0 (md1 shows similar info)
/dev/md0:
Version : 00.90.03
Creation Time : Sat Apr 7 23:32:58 2007
Raid Level : raid5
Array Size : 48869120 (46.61 GiB 50.04 GB)
Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Jan 9 11:35:54 2008
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
Events : 0.2601918
Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sde1
1 8 81 1 active sync /dev/sdf1
2 8 17 2 active sync /dev/sdb1
3 8 1 3 active sync /dev/sda1
4 8 33 4 active sync /dev/sdc1
5 8 97 - spare /dev/sdg1
$ sudo mdadm --examine /dev/sda1 (The other SBs contain info that looks OK)
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
Creation Time : Sat Apr 7 23:32:58 2007
Raid Level : raid5
Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
Array Size : 48869120 (46.61 GiB 50.04 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Wed Jan 9 11:41:29 2008
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : c50df29 - correct
Events : 0.2601918
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 1 3 active sync /dev/sda1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 17 2 active sync /dev/sdb1
3 3 8 1 3 active sync /dev/sda1
4 4 8 33 4 active sync /dev/sdc1
5 5 8 97 5 spare /dev/sdg1
/proc/mdstat:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : active raid5 sdg2[5](S) sde2[0] sda2[3] sdc2[4] sdb2[2] sdf2[1]
927913984 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
md0 : active raid5 sdg1[5](S) sde1[0] sdc1[4] sda1[3] sdb1[2] sdf1[1]
48869120 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
unused devices: <none>
/etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=5 spares=1
UUID=e7356e2b:71e53a26:94b87bc7:e6a9e6b2
ARRAY /dev/md1 level=raid5 num-devices=5 spares=1
UUID=aa0264a3:5fb0396b:04071607:a713ba9
Dmesg (edited):
[ 36.112449] md: linear personality registered for level -1
[ 36.117197] md: multipath personality registered for level -4
[ 36.121795] md: raid0 personality registered for level 0
[ 36.126950] md: raid1 personality registered for level 1
[ 36.131424] raid5: automatically using best checksumming
function: pIII_sse
[ 36.149971] pIII_sse : 4564.000 MB/sec
[ 36.150020] raid5: using function: pIII_sse (4564.000 MB/sec)
[ 36.218015] raid6: int32x1 780 MB/s
[ 36.285943] raid6: int32x2 902 MB/s
[ 36.353961] raid6: int32x4 667 MB/s
[ 36.421869] raid6: int32x8 528 MB/s
[ 36.489811] raid6: mmxx1 1813 MB/s
[ 36.557775] raid6: mmxx2 2123 MB/s
[ 36.625763] raid6: sse1x1 1101 MB/s
[ 36.693717] raid6: sse1x2 1898 MB/s
[ 36.761688] raid6: sse2x1 2227 MB/s
[ 36.829647] raid6: sse2x2 3178 MB/s
[ 36.829695] raid6: using algorithm sse2x2 (3178 MB/s)
[ 36.829744] md: raid6 personality registered for level 6
[ 36.829793] md: raid5 personality registered for level 5
[ 36.829842] md: raid4 personality registered for level 4
[ 36.853475] md: raid10 personality registered for level 10
[ 39.634924] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
(250059 MB)
[ 39.634995] sd 0:0:0:0: [sda] Write Protect is off
[ 39.635048] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 39.635076] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.635218] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
(250059 MB)
[ 39.635292] sd 0:0:0:0: [sda] Write Protect is off
[ 39.635350] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 39.635380] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.635462] sda: sda1 sda2
[ 39.650092] sd 0:0:0:0: [sda] Attached SCSI disk
[ 39.650226] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.650296] sd 1:0:0:0: [sdb] Write Protect is off
[ 39.650348] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 39.650379] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.650505] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.650573] sd 1:0:0:0: [sdb] Write Protect is off
[ 39.650625] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 39.650657] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.650727] sdb: sdb1 sdb2
[ 39.667599] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 39.667719] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.667788] sd 3:0:0:0: [sdc] Write Protect is off
[ 39.667840] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 39.667871] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.667997] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.668064] sd 3:0:0:0: [sdc] Write Protect is off
[ 39.668116] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 39.668146] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.668213] sdc: sdc1 sdc2
[ 39.692703] sd 3:0:0:0: [sdc] Attached SCSI disk
[ 39.699348] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 39.699570] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 39.699786] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 39.834560] md: md0 stopped.
[ 39.870361] md: bind<sdc1>
[ 39.870527] md: md1 stopped.
[ 39.910999] md: md0 stopped.
[ 39.911064] md: unbind<sdc1>
[ 39.911120] md: export_rdev(sdc1)
[ 39.929760] md: bind<sda1>
[ 39.929953] md: bind<sdc1>
[ 39.930139] md: bind<sdb1>
[ 39.930231] md: md1 stopped.
[ 39.932468] md: bind<sdc2>
[ 39.932674] md: bind<sda2>
[ 39.932860] md: bind<sdb2>
[ 40.880217] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
(120034 MB)
[ 40.880288] sd 6:0:0:0: [sdd] Write Protect is off
[ 40.880340] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 40.880367] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.880504] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
(120034 MB)
[ 40.880572] sd 6:0:0:0: [sdd] Write Protect is off
[ 40.880623] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 40.880651] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.880718] sdd: sdd1 sdd2 <<7>ieee1394: Host added:
ID:BUS[0-00:1023] GUID[001485000012704c]
[ 40.908264] sdd5 >
[ 40.908479] sd 6:0:0:0: [sdd] Attached SCSI disk
[ 40.908579] sd 6:0:0:0: Attached scsi generic sg4 type 0
[ 40.908747] scsi 6:0:1:0: Direct-Access ATA Maxtor
7L250S0 BACE PQ: 0 ANSI: 5
[ 40.908899] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
(251000 MB)
[ 40.908968] sd 6:0:1:0: [sde] Write Protect is off
[ 40.909020] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
[ 40.909050] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.909174] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
(251000 MB)
[ 40.909243] sd 6:0:1:0: [sde] Write Protect is off
[ 40.909294] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
[ 40.909324] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.909391] sde: sde1 sde2
[ 40.930218] sd 6:0:1:0: [sde] Attached SCSI disk
[ 40.930318] sd 6:0:1:0: Attached scsi generic sg5 type 0
[ 40.930480] scsi 7:0:0:0: Direct-Access ATA WDC
WD2500JD-00H 08.0 PQ: 0 ANSI: 5
[ 40.930621] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.930689] sd 7:0:0:0: [sdf] Write Protect is off
[ 40.930742] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 40.930769] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.930894] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.930963] sd 7:0:0:0: [sdf] Write Protect is off
[ 40.931015] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 40.931044] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.931111] sdf: sdf1 sdf2
[ 40.948846] sd 7:0:0:0: [sdf] Attached SCSI disk
[ 40.948946] sd 7:0:0:0: Attached scsi generic sg6 type 0
[ 40.949106] scsi 7:0:1:0: Direct-Access ATA WDC
WD2500JS-00M 02.0 PQ: 0 ANSI: 5
[ 40.949248] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.949317] sd 7:0:1:0: [sdg] Write Protect is off
[ 40.949368] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
[ 40.949396] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.949519] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.949588] sd 7:0:1:0: [sdg] Write Protect is off
[ 40.949640] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
[ 40.949668] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.949734] sdg: sdg1 sdg2
[ 40.969827] sd 7:0:1:0: [sdg] Attached SCSI disk
[ 40.969926] sd 7:0:1:0: Attached scsi generic sg7 type 0
[ 41.206078] md: md0 stopped.
[ 41.206137] md: unbind<sdb1>
[ 41.206187] md: export_rdev(sdb1)
[ 41.206253] md: unbind<sdc1>
[ 41.206302] md: export_rdev(sdc1)
[ 41.206360] md: unbind<sda1>
[ 41.206408] md: export_rdev(sda1)
[ 41.247389] md: bind<sdf1>
[ 41.247584] md: bind<sdb1>
[ 41.247787] md: bind<sda1>
[ 41.247971] md: bind<sdc1>
[ 41.248151] md: bind<sdg1>
[ 41.248325] md: bind<sde1>
[ 41.256718] raid5: device sde1 operational as raid disk 0
[ 41.256771] raid5: device sdc1 operational as raid disk 4
[ 41.256821] raid5: device sda1 operational as raid disk 3
[ 41.256870] raid5: device sdb1 operational as raid disk 2
[ 41.256919] raid5: device sdf1 operational as raid disk 1
[ 41.257426] raid5: allocated 5245kB for md0
[ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
devices, algorithm 2
[ 41.257538] RAID5 conf printout:
[ 41.257584] --- rd:5 wd:5
[ 41.257631] disk 0, o:1, dev:sde1
[ 41.257677] disk 1, o:1, dev:sdf1
[ 41.257724] disk 2, o:1, dev:sdb1
[ 41.257771] disk 3, o:1, dev:sda1
[ 41.257817] disk 4, o:1, dev:sdc1
[ 41.257952] md: md1 stopped.
[ 41.258009] md: unbind<sdb2>
[ 41.258060] md: export_rdev(sdb2)
[ 41.258128] md: unbind<sda2>
[ 41.258179] md: export_rdev(sda2)
[ 41.258248] md: unbind<sdc2>
[ 41.258306] md: export_rdev(sdc2)
[ 41.283067] md: bind<sdc2>
[ 41.283297] md: bind<sda2>
[ 41.285235] md: bind<sdb2>
[ 41.306753] md: md1 stopped.
[ 41.306818] md: unbind<sdb2>
[ 41.306878] md: export_rdev(sdb2)
[ 41.306956] md: unbind<sda2>
[ 41.307007] md: export_rdev(sda2)
[ 41.307075] md: unbind<sdc2>
[ 41.307130] md: export_rdev(sdc2)
[ 41.312250] md: bind<sdf2>
[ 41.312476] md: bind<sdb2>
[ 41.312711] md: bind<sdg2>
[ 41.312922] md: bind<sdc2>
[ 41.313138] md: bind<sda2>
[ 41.313343] md: bind<sde2>
[ 41.313452] md: md1: raid array is not clean -- starting
background reconstruction
[ 41.322189] raid5: device sde2 operational as raid disk 0
[ 41.322243] raid5: device sdc2 operational as raid disk 4
[ 41.322292] raid5: device sdg2 operational as raid disk 3
[ 41.322342] raid5: device sdb2 operational as raid disk 2
[ 41.322391] raid5: device sdf2 operational as raid disk 1
[ 41.322823] raid5: allocated 5245kB for md1
[ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
devices, algorithm 2
[ 41.322934] RAID5 conf printout:
[ 41.322980] --- rd:5 wd:5
[ 41.323026] disk 0, o:1, dev:sde2
[ 41.323073] disk 1, o:1, dev:sdf2
[ 41.323119] disk 2, o:1, dev:sdb2
[ 41.323165] disk 3, o:1, dev:sdg2
[ 41.323212] disk 4, o:1, dev:sdc2
[ 41.323316] md: resync of RAID array md1
[ 41.323364] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 41.323415] md: using maximum available idle IO bandwidth (but
not more than 200000 KB/sec) for resync.
[ 41.323492] md: using 128k window, over a total of 231978496 blocks.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-09 20:07 md rotates RAID5 spare at boot Jed Davidow
@ 2008-01-10 18:03 ` Jed Davidow
2008-01-10 21:23 ` this goes go my megaraid probs too was: " Eric S. Johansson
2008-01-10 20:47 ` Bill Davidsen
1 sibling, 1 reply; 12+ messages in thread
From: Jed Davidow @ 2008-01-10 18:03 UTC (permalink / raw)
To: linux-raid
I'm sorry- is this an inappropriate list to ask for help? There seemed
to be a fair amount of that when I searched the archives, but I don't
want to bug developers with my problems!
Please let me know if I should find another place to ask for help (and
please let me know where that might be!).
Thanks!
Jed Davidow wrote:
> I have a RAID5 (5+1spare) setup that works perfectly well until I
> reboot. I have 6 drives (two different models) partitioned to give me
> 2 arrays, md0 and md1, that I use for /home and /var respectively.
>
> When I reboot, the system assembles each array, but swaps out what was
> the spare with one of the member drives. It then immediately detects
> a degraded array and rebuilds. After that, all is fine and testing
> has shown things to be working like they should. Until I reboot.
>
> Example:
> Built two arrays: /dev/md0 -> /dev/sd[abcef]1 and /dev/md1 ->
> /dev/sd[abcef]2
> Added /dev/sdg1 and /dev/sdg2 as spares, and this works.
>
> One scenario when I reboot:
> md0 is assembled from sd[abceg]1; it's degraded and reports a
> "spares missing" event.
> md1 assembles correctly, spare is not missing
>
> Any ideas? I have asked about this on various boards (some said UDEV
> rules would help, some thought the issue had to do with the /dev/sdX
> names changing, etc). I don't think those are applicable since dmesg
> reports the arrays assemble as soon as the disks are detected.
>
>
> Thanks in advance!
>
>
> INFO:
> (currently the boot drive (non raid) is sdd, otherwise all sd devices
> are part of the raid)
>
> fdisk:
>
> $ sudo fdisk -l
>
> Disk /dev/sda: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sda2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdb: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdb2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdc: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdc2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/md0: 50.0 GB, 50041978880 bytes
> 2 heads, 4 sectors/track, 12217280 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
>
> Disk /dev/md1: 950.1 GB, 950183919616 bytes
> 2 heads, 4 sectors/track, 231978496 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md1 doesn't contain a valid partition table
>
> Disk /dev/sdd: 120.0 GB, 120034123776 bytes
> 255 heads, 63 sectors/track, 14593 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x535bfd7a
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 * 1 14219 114214086 83 Linux
> /dev/sdd2 14220 14593 3004155 5 Extended
> /dev/sdd5 14220 14593 3004123+ 82 Linux swap /
> Solaris
>
> Disk /dev/sde: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sde1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sde2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdf: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdf1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdf2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdg: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdg1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdg2 1522 30401 231978600 fd Linux raid
> autodetect
>
>
> $ sudo mdadm --detail /dev/md0 (md1 shows similar info)
>
> /dev/md0:
> Version : 00.90.03
> Creation Time : Sat Apr 7 23:32:58 2007
> Raid Level : raid5
> Array Size : 48869120 (46.61 GiB 50.04 GB)
> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
> Raid Devices : 5
> Total Devices : 6
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Wed Jan 9 11:35:54 2008
> State : clean
> Active Devices : 5
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> Events : 0.2601918
>
> Number Major Minor RaidDevice State
> 0 8 65 0 active sync /dev/sde1
> 1 8 81 1 active sync /dev/sdf1
> 2 8 17 2 active sync /dev/sdb1
> 3 8 1 3 active sync /dev/sda1
> 4 8 33 4 active sync /dev/sdc1
>
> 5 8 97 - spare /dev/sdg1
>
> $ sudo mdadm --examine /dev/sda1 (The other SBs contain info that
> looks OK)
>
> /dev/sda1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> Creation Time : Sat Apr 7 23:32:58 2007
> Raid Level : raid5
> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
> Array Size : 48869120 (46.61 GiB 50.04 GB)
> Raid Devices : 5
> Total Devices : 6
> Preferred Minor : 0
>
> Update Time : Wed Jan 9 11:41:29 2008
> State : clean
> Active Devices : 5
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 1
> Checksum : c50df29 - correct
> Events : 0.2601918
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 1 3 active sync /dev/sda1
>
> 0 0 8 65 0 active sync /dev/sde1
> 1 1 8 81 1 active sync /dev/sdf1
> 2 2 8 17 2 active sync /dev/sdb1
> 3 3 8 1 3 active sync /dev/sda1
> 4 4 8 33 4 active sync /dev/sdc1
> 5 5 8 97 5 spare /dev/sdg1
>
>
> /proc/mdstat:
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid5 sdg2[5](S) sde2[0] sda2[3] sdc2[4] sdb2[2] sdf2[1]
> 927913984 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>
> md0 : active raid5 sdg1[5](S) sde1[0] sdc1[4] sda1[3] sdb1[2] sdf1[1]
> 48869120 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>
> unused devices: <none>
>
> /etc/mdadm/mdadm.conf
>
> DEVICE partitions
> ARRAY /dev/md0 level=raid5 num-devices=5 spares=1
> UUID=e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> ARRAY /dev/md1 level=raid5 num-devices=5 spares=1
> UUID=aa0264a3:5fb0396b:04071607:a713ba9
>
>
> Dmesg (edited):
>
> [ 36.112449] md: linear personality registered for level -1
> [ 36.117197] md: multipath personality registered for level -4
> [ 36.121795] md: raid0 personality registered for level 0
> [ 36.126950] md: raid1 personality registered for level 1
> [ 36.131424] raid5: automatically using best checksumming
> function: pIII_sse
> [ 36.149971] pIII_sse : 4564.000 MB/sec
> [ 36.150020] raid5: using function: pIII_sse (4564.000 MB/sec)
> [ 36.218015] raid6: int32x1 780 MB/s
> [ 36.285943] raid6: int32x2 902 MB/s
> [ 36.353961] raid6: int32x4 667 MB/s
> [ 36.421869] raid6: int32x8 528 MB/s
> [ 36.489811] raid6: mmxx1 1813 MB/s
> [ 36.557775] raid6: mmxx2 2123 MB/s
> [ 36.625763] raid6: sse1x1 1101 MB/s
> [ 36.693717] raid6: sse1x2 1898 MB/s
> [ 36.761688] raid6: sse2x1 2227 MB/s
> [ 36.829647] raid6: sse2x2 3178 MB/s
> [ 36.829695] raid6: using algorithm sse2x2 (3178 MB/s)
> [ 36.829744] md: raid6 personality registered for level 6
> [ 36.829793] md: raid5 personality registered for level 5
> [ 36.829842] md: raid4 personality registered for level 4
> [ 36.853475] md: raid10 personality registered for level 10
>
> [ 39.634924] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 39.634995] sd 0:0:0:0: [sda] Write Protect is off
> [ 39.635048] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [ 39.635076] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.635218] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 39.635292] sd 0:0:0:0: [sda] Write Protect is off
> [ 39.635350] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [ 39.635380] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.635462] sda: sda1 sda2
> [ 39.650092] sd 0:0:0:0: [sda] Attached SCSI disk
>
> [ 39.650226] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.650296] sd 1:0:0:0: [sdb] Write Protect is off
> [ 39.650348] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [ 39.650379] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.650505] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.650573] sd 1:0:0:0: [sdb] Write Protect is off
> [ 39.650625] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [ 39.650657] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.650727] sdb: sdb1 sdb2
> [ 39.667599] sd 1:0:0:0: [sdb] Attached SCSI disk
>
> [ 39.667719] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.667788] sd 3:0:0:0: [sdc] Write Protect is off
> [ 39.667840] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [ 39.667871] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.667997] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.668064] sd 3:0:0:0: [sdc] Write Protect is off
> [ 39.668116] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [ 39.668146] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.668213] sdc: sdc1 sdc2
> [ 39.692703] sd 3:0:0:0: [sdc] Attached SCSI disk
>
> [ 39.699348] sd 0:0:0:0: Attached scsi generic sg0 type 0
> [ 39.699570] sd 1:0:0:0: Attached scsi generic sg1 type 0
> [ 39.699786] sd 3:0:0:0: Attached scsi generic sg2 type 0
>
> [ 39.834560] md: md0 stopped.
> [ 39.870361] md: bind<sdc1>
> [ 39.870527] md: md1 stopped.
> [ 39.910999] md: md0 stopped.
> [ 39.911064] md: unbind<sdc1>
> [ 39.911120] md: export_rdev(sdc1)
> [ 39.929760] md: bind<sda1>
> [ 39.929953] md: bind<sdc1>
> [ 39.930139] md: bind<sdb1>
> [ 39.930231] md: md1 stopped.
> [ 39.932468] md: bind<sdc2>
> [ 39.932674] md: bind<sda2>
> [ 39.932860] md: bind<sdb2>
>
> [ 40.880217] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
> (120034 MB)
> [ 40.880288] sd 6:0:0:0: [sdd] Write Protect is off
> [ 40.880340] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
> [ 40.880367] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.880504] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
> (120034 MB)
> [ 40.880572] sd 6:0:0:0: [sdd] Write Protect is off
> [ 40.880623] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
> [ 40.880651] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.880718] sdd: sdd1 sdd2 <<7>ieee1394: Host added:
> ID:BUS[0-00:1023] GUID[001485000012704c]
> [ 40.908264] sdd5 >
> [ 40.908479] sd 6:0:0:0: [sdd] Attached SCSI disk
> [ 40.908579] sd 6:0:0:0: Attached scsi generic sg4 type 0
>
> [ 40.908747] scsi 6:0:1:0: Direct-Access ATA Maxtor
> 7L250S0 BACE PQ: 0 ANSI: 5
> [ 40.908899] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 40.908968] sd 6:0:1:0: [sde] Write Protect is off
> [ 40.909020] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
> [ 40.909050] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.909174] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 40.909243] sd 6:0:1:0: [sde] Write Protect is off
> [ 40.909294] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
> [ 40.909324] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.909391] sde: sde1 sde2
> [ 40.930218] sd 6:0:1:0: [sde] Attached SCSI disk
> [ 40.930318] sd 6:0:1:0: Attached scsi generic sg5 type 0
>
> [ 40.930480] scsi 7:0:0:0: Direct-Access ATA WDC
> WD2500JD-00H 08.0 PQ: 0 ANSI: 5
> [ 40.930621] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.930689] sd 7:0:0:0: [sdf] Write Protect is off
> [ 40.930742] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
> [ 40.930769] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.930894] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.930963] sd 7:0:0:0: [sdf] Write Protect is off
> [ 40.931015] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
> [ 40.931044] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.931111] sdf: sdf1 sdf2
> [ 40.948846] sd 7:0:0:0: [sdf] Attached SCSI disk
> [ 40.948946] sd 7:0:0:0: Attached scsi generic sg6 type 0
>
> [ 40.949106] scsi 7:0:1:0: Direct-Access ATA WDC
> WD2500JS-00M 02.0 PQ: 0 ANSI: 5
> [ 40.949248] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.949317] sd 7:0:1:0: [sdg] Write Protect is off
> [ 40.949368] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
> [ 40.949396] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.949519] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.949588] sd 7:0:1:0: [sdg] Write Protect is off
> [ 40.949640] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
> [ 40.949668] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.949734] sdg: sdg1 sdg2
> [ 40.969827] sd 7:0:1:0: [sdg] Attached SCSI disk
> [ 40.969926] sd 7:0:1:0: Attached scsi generic sg7 type 0
>
> [ 41.206078] md: md0 stopped.
> [ 41.206137] md: unbind<sdb1>
> [ 41.206187] md: export_rdev(sdb1)
> [ 41.206253] md: unbind<sdc1>
> [ 41.206302] md: export_rdev(sdc1)
> [ 41.206360] md: unbind<sda1>
> [ 41.206408] md: export_rdev(sda1)
> [ 41.247389] md: bind<sdf1>
> [ 41.247584] md: bind<sdb1>
> [ 41.247787] md: bind<sda1>
> [ 41.247971] md: bind<sdc1>
> [ 41.248151] md: bind<sdg1>
> [ 41.248325] md: bind<sde1>
> [ 41.256718] raid5: device sde1 operational as raid disk 0
> [ 41.256771] raid5: device sdc1 operational as raid disk 4
> [ 41.256821] raid5: device sda1 operational as raid disk 3
> [ 41.256870] raid5: device sdb1 operational as raid disk 2
> [ 41.256919] raid5: device sdf1 operational as raid disk 1
> [ 41.257426] raid5: allocated 5245kB for md0
> [ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
> devices, algorithm 2
> [ 41.257538] RAID5 conf printout:
> [ 41.257584] --- rd:5 wd:5
> [ 41.257631] disk 0, o:1, dev:sde1
> [ 41.257677] disk 1, o:1, dev:sdf1
> [ 41.257724] disk 2, o:1, dev:sdb1
> [ 41.257771] disk 3, o:1, dev:sda1
> [ 41.257817] disk 4, o:1, dev:sdc1
>
> [ 41.257952] md: md1 stopped.
> [ 41.258009] md: unbind<sdb2>
> [ 41.258060] md: export_rdev(sdb2)
> [ 41.258128] md: unbind<sda2>
> [ 41.258179] md: export_rdev(sda2)
> [ 41.258248] md: unbind<sdc2>
> [ 41.258306] md: export_rdev(sdc2)
> [ 41.283067] md: bind<sdc2>
> [ 41.283297] md: bind<sda2>
> [ 41.285235] md: bind<sdb2>
> [ 41.306753] md: md1 stopped.
> [ 41.306818] md: unbind<sdb2>
> [ 41.306878] md: export_rdev(sdb2)
> [ 41.306956] md: unbind<sda2>
> [ 41.307007] md: export_rdev(sda2)
> [ 41.307075] md: unbind<sdc2>
> [ 41.307130] md: export_rdev(sdc2)
> [ 41.312250] md: bind<sdf2>
> [ 41.312476] md: bind<sdb2>
> [ 41.312711] md: bind<sdg2>
> [ 41.312922] md: bind<sdc2>
> [ 41.313138] md: bind<sda2>
> [ 41.313343] md: bind<sde2>
> [ 41.313452] md: md1: raid array is not clean -- starting
> background reconstruction
> [ 41.322189] raid5: device sde2 operational as raid disk 0
> [ 41.322243] raid5: device sdc2 operational as raid disk 4
> [ 41.322292] raid5: device sdg2 operational as raid disk 3
> [ 41.322342] raid5: device sdb2 operational as raid disk 2
> [ 41.322391] raid5: device sdf2 operational as raid disk 1
> [ 41.322823] raid5: allocated 5245kB for md1
> [ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
> devices, algorithm 2
> [ 41.322934] RAID5 conf printout:
> [ 41.322980] --- rd:5 wd:5
> [ 41.323026] disk 0, o:1, dev:sde2
> [ 41.323073] disk 1, o:1, dev:sdf2
> [ 41.323119] disk 2, o:1, dev:sdb2
> [ 41.323165] disk 3, o:1, dev:sdg2
> [ 41.323212] disk 4, o:1, dev:sdc2
>
> [ 41.323316] md: resync of RAID array md1
> [ 41.323364] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 41.323415] md: using maximum available idle IO bandwidth (but
> not more than 200000 KB/sec) for resync.
> [ 41.323492] md: using 128k window, over a total of 231978496
> blocks.
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* this goes go my megaraid probs too was: Re: md rotates RAID5 spare at boot
2008-01-10 18:03 ` Jed Davidow
@ 2008-01-10 21:23 ` Eric S. Johansson
0 siblings, 0 replies; 12+ messages in thread
From: Eric S. Johansson @ 2008-01-10 21:23 UTC (permalink / raw)
To: linux-raid
Jed Davidow wrote:
> I'm sorry- is this an inappropriate list to ask for help? There seemed
> to be a fair amount of that when I searched the archives, but I don't
> want to bug developers with my problems!
>
> Please let me know if I should find another place to ask for help (and
> please let me know where that might be!).
I could also use help with my mega-raid 150 question. Don't know if I asked it
wrong or it was the color shirt I was wearing. I am unfortunately running with
such a dearth of knowledge on the topic that I don't really know the right
questions to ask when diagnosing a performance problem. All I know is that
there's very little documentation on this card, is even less documentation on
the commandline tool to access/control the card and if it makes the most sense,
I am perfectly willing to deep six the card on eBay and pick up a couple of
reasonable speed serial ATA controller cards in its stead.
the only reason I want to try and learn more about the hardware raid is because
the problems I'm experiencing with my virtual machines on this platform mimic
problems a customer of mine is experiencing and if I can fix them just by
changing how the raid controller uses the discs, then that is a huge win.
Personally, I think it's something a little deeper because VMware server seems
to go out to lunch whenever there is a backup in the disk I/O queue. I'm
seriously thinking about picking up esx as soon as the budget allows.
I just need some good solid advice on what path I should take.
---eric
--
Speech-recognition in use. It makes mistakes, I correct some.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: md rotates RAID5 spare at boot
2008-01-09 20:07 md rotates RAID5 spare at boot Jed Davidow
2008-01-10 18:03 ` Jed Davidow
@ 2008-01-10 20:47 ` Bill Davidsen
2008-01-10 20:53 ` Jed Davidow
1 sibling, 1 reply; 12+ messages in thread
From: Bill Davidsen @ 2008-01-10 20:47 UTC (permalink / raw)
To: Jed Davidow; +Cc: linux-raid
Jed Davidow wrote:
> I have a RAID5 (5+1spare) setup that works perfectly well until I
> reboot. I have 6 drives (two different models) partitioned to give me
> 2 arrays, md0 and md1, that I use for /home and /var respectively.
>
> When I reboot, the system assembles each array, but swaps out what was
> the spare with one of the member drives. It then immediately detects
> a degraded array and rebuilds. After that, all is fine and testing
> has shown things to be working like they should. Until I reboot.
>
> Example:
> Built two arrays: /dev/md0 -> /dev/sd[abcef]1 and /dev/md1 ->
> /dev/sd[abcef]2
> Added /dev/sdg1 and /dev/sdg2 as spares, and this works.
>
> One scenario when I reboot:
> md0 is assembled from sd[abceg]1; it's degraded and reports a
> "spares missing" event.
> md1 assembles correctly, spare is not missing
>
I'm looking at the dmesg which follows and seeing md1 reconstructing.
This seems to be at variance with "assembles correctly" here. That's the
only thing which has struck me as worth mentioning so far.
> Any ideas? I have asked about this on various boards (some said UDEV
> rules would help, some thought the issue had to do with the /dev/sdX
> names changing, etc). I don't think those are applicable since dmesg
> reports the arrays assemble as soon as the disks are detected.
>
>
> Thanks in advance!
>
>
> INFO:
> (currently the boot drive (non raid) is sdd, otherwise all sd devices
> are part of the raid)
>
> fdisk:
>
> $ sudo fdisk -l
>
> Disk /dev/sda: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sda2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdb: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdb2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdc: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdc2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/md0: 50.0 GB, 50041978880 bytes
> 2 heads, 4 sectors/track, 12217280 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
>
> Disk /dev/md1: 950.1 GB, 950183919616 bytes
> 2 heads, 4 sectors/track, 231978496 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md1 doesn't contain a valid partition table
>
> Disk /dev/sdd: 120.0 GB, 120034123776 bytes
> 255 heads, 63 sectors/track, 14593 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x535bfd7a
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 * 1 14219 114214086 83 Linux
> /dev/sdd2 14220 14593 3004155 5 Extended
> /dev/sdd5 14220 14593 3004123+ 82 Linux swap /
> Solaris
>
> Disk /dev/sde: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sde1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sde2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdf: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdf1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdf2 1522 30401 231978600 fd Linux raid
> autodetect
>
> Disk /dev/sdg: 250.0 GB, 250059350016 bytes
> 255 heads, 63 sectors/track, 30401 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdg1 1 1521 12217401 fd Linux raid
> autodetect
> /dev/sdg2 1522 30401 231978600 fd Linux raid
> autodetect
>
>
> $ sudo mdadm --detail /dev/md0 (md1 shows similar info)
>
> /dev/md0:
> Version : 00.90.03
> Creation Time : Sat Apr 7 23:32:58 2007
> Raid Level : raid5
> Array Size : 48869120 (46.61 GiB 50.04 GB)
> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
> Raid Devices : 5
> Total Devices : 6
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Wed Jan 9 11:35:54 2008
> State : clean
> Active Devices : 5
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> Events : 0.2601918
>
> Number Major Minor RaidDevice State
> 0 8 65 0 active sync /dev/sde1
> 1 8 81 1 active sync /dev/sdf1
> 2 8 17 2 active sync /dev/sdb1
> 3 8 1 3 active sync /dev/sda1
> 4 8 33 4 active sync /dev/sdc1
>
> 5 8 97 - spare /dev/sdg1
>
> $ sudo mdadm --examine /dev/sda1 (The other SBs contain info that
> looks OK)
>
> /dev/sda1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> Creation Time : Sat Apr 7 23:32:58 2007
> Raid Level : raid5
> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
> Array Size : 48869120 (46.61 GiB 50.04 GB)
> Raid Devices : 5
> Total Devices : 6
> Preferred Minor : 0
>
> Update Time : Wed Jan 9 11:41:29 2008
> State : clean
> Active Devices : 5
> Working Devices : 6
> Failed Devices : 0
> Spare Devices : 1
> Checksum : c50df29 - correct
> Events : 0.2601918
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 1 3 active sync /dev/sda1
>
> 0 0 8 65 0 active sync /dev/sde1
> 1 1 8 81 1 active sync /dev/sdf1
> 2 2 8 17 2 active sync /dev/sdb1
> 3 3 8 1 3 active sync /dev/sda1
> 4 4 8 33 4 active sync /dev/sdc1
> 5 5 8 97 5 spare /dev/sdg1
>
>
> /proc/mdstat:
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : active raid5 sdg2[5](S) sde2[0] sda2[3] sdc2[4] sdb2[2] sdf2[1]
> 927913984 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>
> md0 : active raid5 sdg1[5](S) sde1[0] sdc1[4] sda1[3] sdb1[2] sdf1[1]
> 48869120 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>
> unused devices: <none>
>
> /etc/mdadm/mdadm.conf
>
> DEVICE partitions
> ARRAY /dev/md0 level=raid5 num-devices=5 spares=1
> UUID=e7356e2b:71e53a26:94b87bc7:e6a9e6b2
> ARRAY /dev/md1 level=raid5 num-devices=5 spares=1
> UUID=aa0264a3:5fb0396b:04071607:a713ba9
>
>
> Dmesg (edited):
>
> [ 36.112449] md: linear personality registered for level -1
> [ 36.117197] md: multipath personality registered for level -4
> [ 36.121795] md: raid0 personality registered for level 0
> [ 36.126950] md: raid1 personality registered for level 1
> [ 36.131424] raid5: automatically using best checksumming
> function: pIII_sse
> [ 36.149971] pIII_sse : 4564.000 MB/sec
> [ 36.150020] raid5: using function: pIII_sse (4564.000 MB/sec)
> [ 36.218015] raid6: int32x1 780 MB/s
> [ 36.285943] raid6: int32x2 902 MB/s
> [ 36.353961] raid6: int32x4 667 MB/s
> [ 36.421869] raid6: int32x8 528 MB/s
> [ 36.489811] raid6: mmxx1 1813 MB/s
> [ 36.557775] raid6: mmxx2 2123 MB/s
> [ 36.625763] raid6: sse1x1 1101 MB/s
> [ 36.693717] raid6: sse1x2 1898 MB/s
> [ 36.761688] raid6: sse2x1 2227 MB/s
> [ 36.829647] raid6: sse2x2 3178 MB/s
> [ 36.829695] raid6: using algorithm sse2x2 (3178 MB/s)
> [ 36.829744] md: raid6 personality registered for level 6
> [ 36.829793] md: raid5 personality registered for level 5
> [ 36.829842] md: raid4 personality registered for level 4
> [ 36.853475] md: raid10 personality registered for level 10
>
> [ 39.634924] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 39.634995] sd 0:0:0:0: [sda] Write Protect is off
> [ 39.635048] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [ 39.635076] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.635218] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 39.635292] sd 0:0:0:0: [sda] Write Protect is off
> [ 39.635350] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [ 39.635380] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.635462] sda: sda1 sda2
> [ 39.650092] sd 0:0:0:0: [sda] Attached SCSI disk
>
> [ 39.650226] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.650296] sd 1:0:0:0: [sdb] Write Protect is off
> [ 39.650348] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [ 39.650379] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.650505] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.650573] sd 1:0:0:0: [sdb] Write Protect is off
> [ 39.650625] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [ 39.650657] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.650727] sdb: sdb1 sdb2
> [ 39.667599] sd 1:0:0:0: [sdb] Attached SCSI disk
>
> [ 39.667719] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.667788] sd 3:0:0:0: [sdc] Write Protect is off
> [ 39.667840] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [ 39.667871] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.667997] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 39.668064] sd 3:0:0:0: [sdc] Write Protect is off
> [ 39.668116] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [ 39.668146] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 39.668213] sdc: sdc1 sdc2
> [ 39.692703] sd 3:0:0:0: [sdc] Attached SCSI disk
>
> [ 39.699348] sd 0:0:0:0: Attached scsi generic sg0 type 0
> [ 39.699570] sd 1:0:0:0: Attached scsi generic sg1 type 0
> [ 39.699786] sd 3:0:0:0: Attached scsi generic sg2 type 0
>
> [ 39.834560] md: md0 stopped.
> [ 39.870361] md: bind<sdc1>
> [ 39.870527] md: md1 stopped.
> [ 39.910999] md: md0 stopped.
> [ 39.911064] md: unbind<sdc1>
> [ 39.911120] md: export_rdev(sdc1)
> [ 39.929760] md: bind<sda1>
> [ 39.929953] md: bind<sdc1>
> [ 39.930139] md: bind<sdb1>
> [ 39.930231] md: md1 stopped.
> [ 39.932468] md: bind<sdc2>
> [ 39.932674] md: bind<sda2>
> [ 39.932860] md: bind<sdb2>
>
> [ 40.880217] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
> (120034 MB)
> [ 40.880288] sd 6:0:0:0: [sdd] Write Protect is off
> [ 40.880340] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
> [ 40.880367] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.880504] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
> (120034 MB)
> [ 40.880572] sd 6:0:0:0: [sdd] Write Protect is off
> [ 40.880623] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
> [ 40.880651] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.880718] sdd: sdd1 sdd2 <<7>ieee1394: Host added:
> ID:BUS[0-00:1023] GUID[001485000012704c]
> [ 40.908264] sdd5 >
> [ 40.908479] sd 6:0:0:0: [sdd] Attached SCSI disk
> [ 40.908579] sd 6:0:0:0: Attached scsi generic sg4 type 0
>
> [ 40.908747] scsi 6:0:1:0: Direct-Access ATA Maxtor
> 7L250S0 BACE PQ: 0 ANSI: 5
> [ 40.908899] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 40.908968] sd 6:0:1:0: [sde] Write Protect is off
> [ 40.909020] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
> [ 40.909050] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.909174] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
> (251000 MB)
> [ 40.909243] sd 6:0:1:0: [sde] Write Protect is off
> [ 40.909294] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
> [ 40.909324] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.909391] sde: sde1 sde2
> [ 40.930218] sd 6:0:1:0: [sde] Attached SCSI disk
> [ 40.930318] sd 6:0:1:0: Attached scsi generic sg5 type 0
>
> [ 40.930480] scsi 7:0:0:0: Direct-Access ATA WDC
> WD2500JD-00H 08.0 PQ: 0 ANSI: 5
> [ 40.930621] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.930689] sd 7:0:0:0: [sdf] Write Protect is off
> [ 40.930742] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
> [ 40.930769] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.930894] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.930963] sd 7:0:0:0: [sdf] Write Protect is off
> [ 40.931015] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
> [ 40.931044] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.931111] sdf: sdf1 sdf2
> [ 40.948846] sd 7:0:0:0: [sdf] Attached SCSI disk
> [ 40.948946] sd 7:0:0:0: Attached scsi generic sg6 type 0
>
> [ 40.949106] scsi 7:0:1:0: Direct-Access ATA WDC
> WD2500JS-00M 02.0 PQ: 0 ANSI: 5
> [ 40.949248] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.949317] sd 7:0:1:0: [sdg] Write Protect is off
> [ 40.949368] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
> [ 40.949396] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.949519] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
> (250059 MB)
> [ 40.949588] sd 7:0:1:0: [sdg] Write Protect is off
> [ 40.949640] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
> [ 40.949668] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
> enabled, doesn't support DPO or FUA
> [ 40.949734] sdg: sdg1 sdg2
> [ 40.969827] sd 7:0:1:0: [sdg] Attached SCSI disk
> [ 40.969926] sd 7:0:1:0: Attached scsi generic sg7 type 0
>
> [ 41.206078] md: md0 stopped.
> [ 41.206137] md: unbind<sdb1>
> [ 41.206187] md: export_rdev(sdb1)
> [ 41.206253] md: unbind<sdc1>
> [ 41.206302] md: export_rdev(sdc1)
> [ 41.206360] md: unbind<sda1>
> [ 41.206408] md: export_rdev(sda1)
> [ 41.247389] md: bind<sdf1>
> [ 41.247584] md: bind<sdb1>
> [ 41.247787] md: bind<sda1>
> [ 41.247971] md: bind<sdc1>
> [ 41.248151] md: bind<sdg1>
> [ 41.248325] md: bind<sde1>
> [ 41.256718] raid5: device sde1 operational as raid disk 0
> [ 41.256771] raid5: device sdc1 operational as raid disk 4
> [ 41.256821] raid5: device sda1 operational as raid disk 3
> [ 41.256870] raid5: device sdb1 operational as raid disk 2
> [ 41.256919] raid5: device sdf1 operational as raid disk 1
> [ 41.257426] raid5: allocated 5245kB for md0
> [ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
> devices, algorithm 2
> [ 41.257538] RAID5 conf printout:
> [ 41.257584] --- rd:5 wd:5
> [ 41.257631] disk 0, o:1, dev:sde1
> [ 41.257677] disk 1, o:1, dev:sdf1
> [ 41.257724] disk 2, o:1, dev:sdb1
> [ 41.257771] disk 3, o:1, dev:sda1
> [ 41.257817] disk 4, o:1, dev:sdc1
>
> [ 41.257952] md: md1 stopped.
> [ 41.258009] md: unbind<sdb2>
> [ 41.258060] md: export_rdev(sdb2)
> [ 41.258128] md: unbind<sda2>
> [ 41.258179] md: export_rdev(sda2)
> [ 41.258248] md: unbind<sdc2>
> [ 41.258306] md: export_rdev(sdc2)
> [ 41.283067] md: bind<sdc2>
> [ 41.283297] md: bind<sda2>
> [ 41.285235] md: bind<sdb2>
> [ 41.306753] md: md1 stopped.
> [ 41.306818] md: unbind<sdb2>
> [ 41.306878] md: export_rdev(sdb2)
> [ 41.306956] md: unbind<sda2>
> [ 41.307007] md: export_rdev(sda2)
> [ 41.307075] md: unbind<sdc2>
> [ 41.307130] md: export_rdev(sdc2)
> [ 41.312250] md: bind<sdf2>
> [ 41.312476] md: bind<sdb2>
> [ 41.312711] md: bind<sdg2>
> [ 41.312922] md: bind<sdc2>
> [ 41.313138] md: bind<sda2>
> [ 41.313343] md: bind<sde2>
> [ 41.313452] md: md1: raid array is not clean -- starting
> background reconstruction
> [ 41.322189] raid5: device sde2 operational as raid disk 0
> [ 41.322243] raid5: device sdc2 operational as raid disk 4
> [ 41.322292] raid5: device sdg2 operational as raid disk 3
> [ 41.322342] raid5: device sdb2 operational as raid disk 2
> [ 41.322391] raid5: device sdf2 operational as raid disk 1
> [ 41.322823] raid5: allocated 5245kB for md1
> [ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
> devices, algorithm 2
> [ 41.322934] RAID5 conf printout:
> [ 41.322980] --- rd:5 wd:5
> [ 41.323026] disk 0, o:1, dev:sde2
> [ 41.323073] disk 1, o:1, dev:sdf2
> [ 41.323119] disk 2, o:1, dev:sdb2
> [ 41.323165] disk 3, o:1, dev:sdg2
> [ 41.323212] disk 4, o:1, dev:sdc2
>
> [ 41.323316] md: resync of RAID array md1
> [ 41.323364] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 41.323415] md: using maximum available idle IO bandwidth (but
> not more than 200000 KB/sec) for resync.
> [ 41.323492] md: using 128k window, over a total of 231978496
> blocks.
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Bill Davidsen <davidsen@tmr.com>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: md rotates RAID5 spare at boot
2008-01-10 20:47 ` Bill Davidsen
@ 2008-01-10 20:53 ` Jed Davidow
2008-01-10 22:01 ` Neil Brown
0 siblings, 1 reply; 12+ messages in thread
From: Jed Davidow @ 2008-01-10 20:53 UTC (permalink / raw)
To: Bill Davidsen; +Cc: linux-raid
Hi Bill,
Maybe I'm using the wrong words...
In this instance, on the previous boot, md1 was assembled from
sd[efbac]2 and sdg2 was the spare. When I rebooted it assembled from
sd[efbgc]2 and had no spare (appears that sdg was swapped in for sda).
Since sdg2 had been the spare, the array is degraded and it rebuilds. I
suppose this would be the case if, during the shutdown that sda2 was
compromised (although I see nothing about sda2 as being faulty- I can
manually add it immediately). But this happens just about every time I
reboot, sometimes to only one of the two arrays, sometimes with the
corresponding partitions on both arrays and sometimes with different
partitions on each array.
If something was physically wrong with one of the drives, I would expect
it to swap in the spare for that drive each time. But it seems to swap
in the spare randomly.
Note- last night I shutdown completely, restarted after 30 sec and for
the first time in a while did not have an issue. This time the drives
were recognized and assigned device nodes in the 'correct' order (MB
controller first, PCI controller next). Would device node assignments
have any affect on how the array was being assembled?
It looks to me like md inspects and attempts to assemble after each
drive controller is scanned (from dmesg, there appears to be a failed
bind on the first three devices after they are scanned, and then again
when the second controller is scanned). Would the scan order cause a
spare to be swapped in?
Bill Davidsen wrote:
> Jed Davidow wrote:
>> I have a RAID5 (5+1spare) setup that works perfectly well until I
>> reboot. I have 6 drives (two different models) partitioned to give
>> me 2 arrays, md0 and md1, that I use for /home and /var respectively.
>>
>> When I reboot, the system assembles each array, but swaps out what
>> was the spare with one of the member drives. It then immediately
>> detects a degraded array and rebuilds. After that, all is fine and
>> testing has shown things to be working like they should. Until I
>> reboot.
>>
>> Example:
>> Built two arrays: /dev/md0 -> /dev/sd[abcef]1 and /dev/md1 ->
>> /dev/sd[abcef]2
>> Added /dev/sdg1 and /dev/sdg2 as spares, and this works.
>>
>> One scenario when I reboot:
>> md0 is assembled from sd[abceg]1; it's degraded and reports a
>> "spares missing" event.
>> md1 assembles correctly, spare is not missing
>>
> I'm looking at the dmesg which follows and seeing md1 reconstructing.
> This seems to be at variance with "assembles correctly" here. That's
> the only thing which has struck me as worth mentioning so far.
>> Any ideas? I have asked about this on various boards (some said UDEV
>> rules would help, some thought the issue had to do with the /dev/sdX
>> names changing, etc). I don't think those are applicable since dmesg
>> reports the arrays assemble as soon as the disks are detected.
>>
>>
>> Thanks in advance!
>>
>>
>> INFO:
>> (currently the boot drive (non raid) is sdd, otherwise all sd devices
>> are part of the raid)
>>
>> fdisk:
>>
>> $ sudo fdisk -l
>>
>> Disk /dev/sda: 250.0 GB, 250059350016 bytes
>> 255 heads, 63 sectors/track, 30401 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sda1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sda2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>> Disk /dev/sdb: 251.0 GB, 251000193024 bytes
>> 255 heads, 63 sectors/track, 30515 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdb1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sdb2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>> Disk /dev/sdc: 251.0 GB, 251000193024 bytes
>> 255 heads, 63 sectors/track, 30515 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdc1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sdc2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>> Disk /dev/md0: 50.0 GB, 50041978880 bytes
>> 2 heads, 4 sectors/track, 12217280 cylinders
>> Units = cylinders of 8 * 512 = 4096 bytes
>> Disk identifier: 0x00000000
>>
>> Disk /dev/md0 doesn't contain a valid partition table
>>
>> Disk /dev/md1: 950.1 GB, 950183919616 bytes
>> 2 heads, 4 sectors/track, 231978496 cylinders
>> Units = cylinders of 8 * 512 = 4096 bytes
>> Disk identifier: 0x00000000
>>
>> Disk /dev/md1 doesn't contain a valid partition table
>>
>> Disk /dev/sdd: 120.0 GB, 120034123776 bytes
>> 255 heads, 63 sectors/track, 14593 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x535bfd7a
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdd1 * 1 14219 114214086 83 Linux
>> /dev/sdd2 14220 14593 3004155 5 Extended
>> /dev/sdd5 14220 14593 3004123+ 82 Linux swap /
>> Solaris
>>
>> Disk /dev/sde: 251.0 GB, 251000193024 bytes
>> 255 heads, 63 sectors/track, 30515 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sde1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sde2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>> Disk /dev/sdf: 250.0 GB, 250059350016 bytes
>> 255 heads, 63 sectors/track, 30401 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdf1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sdf2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>> Disk /dev/sdg: 250.0 GB, 250059350016 bytes
>> 255 heads, 63 sectors/track, 30401 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>> Disk identifier: 0x00000000
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdg1 1 1521 12217401 fd Linux raid
>> autodetect
>> /dev/sdg2 1522 30401 231978600 fd Linux raid
>> autodetect
>>
>>
>> $ sudo mdadm --detail /dev/md0 (md1 shows similar info)
>>
>> /dev/md0:
>> Version : 00.90.03
>> Creation Time : Sat Apr 7 23:32:58 2007
>> Raid Level : raid5
>> Array Size : 48869120 (46.61 GiB 50.04 GB)
>> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
>> Raid Devices : 5
>> Total Devices : 6
>> Preferred Minor : 0
>> Persistence : Superblock is persistent
>>
>> Update Time : Wed Jan 9 11:35:54 2008
>> State : clean
>> Active Devices : 5
>> Working Devices : 6
>> Failed Devices : 0
>> Spare Devices : 1
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
>> Events : 0.2601918
>>
>> Number Major Minor RaidDevice State
>> 0 8 65 0 active sync /dev/sde1
>> 1 8 81 1 active sync /dev/sdf1
>> 2 8 17 2 active sync /dev/sdb1
>> 3 8 1 3 active sync /dev/sda1
>> 4 8 33 4 active sync /dev/sdc1
>>
>> 5 8 97 - spare /dev/sdg1
>>
>> $ sudo mdadm --examine /dev/sda1 (The other SBs contain info that
>> looks OK)
>>
>> /dev/sda1:
>> Magic : a92b4efc
>> Version : 00.90.00
>> UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
>> Creation Time : Sat Apr 7 23:32:58 2007
>> Raid Level : raid5
>> Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
>> Array Size : 48869120 (46.61 GiB 50.04 GB)
>> Raid Devices : 5
>> Total Devices : 6
>> Preferred Minor : 0
>>
>> Update Time : Wed Jan 9 11:41:29 2008
>> State : clean
>> Active Devices : 5
>> Working Devices : 6
>> Failed Devices : 0
>> Spare Devices : 1
>> Checksum : c50df29 - correct
>> Events : 0.2601918
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 3 8 1 3 active sync /dev/sda1
>>
>> 0 0 8 65 0 active sync /dev/sde1
>> 1 1 8 81 1 active sync /dev/sdf1
>> 2 2 8 17 2 active sync /dev/sdb1
>> 3 3 8 1 3 active sync /dev/sda1
>> 4 4 8 33 4 active sync /dev/sdc1
>> 5 5 8 97 5 spare /dev/sdg1
>>
>>
>> /proc/mdstat:
>>
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md1 : active raid5 sdg2[5](S) sde2[0] sda2[3] sdc2[4] sdb2[2] sdf2[1]
>> 927913984 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>>
>> md0 : active raid5 sdg1[5](S) sde1[0] sdc1[4] sda1[3] sdb1[2] sdf1[1]
>> 48869120 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
>>
>> unused devices: <none>
>>
>> /etc/mdadm/mdadm.conf
>>
>> DEVICE partitions
>> ARRAY /dev/md0 level=raid5 num-devices=5 spares=1
>> UUID=e7356e2b:71e53a26:94b87bc7:e6a9e6b2
>> ARRAY /dev/md1 level=raid5 num-devices=5 spares=1
>> UUID=aa0264a3:5fb0396b:04071607:a713ba9
>>
>>
>> Dmesg (edited):
>>
>> [ 36.112449] md: linear personality registered for level -1
>> [ 36.117197] md: multipath personality registered for level -4
>> [ 36.121795] md: raid0 personality registered for level 0
>> [ 36.126950] md: raid1 personality registered for level 1
>> [ 36.131424] raid5: automatically using best checksumming
>> function: pIII_sse
>> [ 36.149971] pIII_sse : 4564.000 MB/sec
>> [ 36.150020] raid5: using function: pIII_sse (4564.000 MB/sec)
>> [ 36.218015] raid6: int32x1 780 MB/s
>> [ 36.285943] raid6: int32x2 902 MB/s
>> [ 36.353961] raid6: int32x4 667 MB/s
>> [ 36.421869] raid6: int32x8 528 MB/s
>> [ 36.489811] raid6: mmxx1 1813 MB/s
>> [ 36.557775] raid6: mmxx2 2123 MB/s
>> [ 36.625763] raid6: sse1x1 1101 MB/s
>> [ 36.693717] raid6: sse1x2 1898 MB/s
>> [ 36.761688] raid6: sse2x1 2227 MB/s
>> [ 36.829647] raid6: sse2x2 3178 MB/s
>> [ 36.829695] raid6: using algorithm sse2x2 (3178 MB/s)
>> [ 36.829744] md: raid6 personality registered for level 6
>> [ 36.829793] md: raid5 personality registered for level 5
>> [ 36.829842] md: raid4 personality registered for level 4
>> [ 36.853475] md: raid10 personality registered for level 10
>>
>> [ 39.634924] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 39.634995] sd 0:0:0:0: [sda] Write Protect is off
>> [ 39.635048] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
>> [ 39.635076] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.635218] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 39.635292] sd 0:0:0:0: [sda] Write Protect is off
>> [ 39.635350] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
>> [ 39.635380] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.635462] sda: sda1 sda2
>> [ 39.650092] sd 0:0:0:0: [sda] Attached SCSI disk
>>
>> [ 39.650226] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 39.650296] sd 1:0:0:0: [sdb] Write Protect is off
>> [ 39.650348] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
>> [ 39.650379] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.650505] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 39.650573] sd 1:0:0:0: [sdb] Write Protect is off
>> [ 39.650625] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
>> [ 39.650657] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.650727] sdb: sdb1 sdb2
>> [ 39.667599] sd 1:0:0:0: [sdb] Attached SCSI disk
>>
>> [ 39.667719] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 39.667788] sd 3:0:0:0: [sdc] Write Protect is off
>> [ 39.667840] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
>> [ 39.667871] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.667997] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 39.668064] sd 3:0:0:0: [sdc] Write Protect is off
>> [ 39.668116] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
>> [ 39.668146] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 39.668213] sdc: sdc1 sdc2
>> [ 39.692703] sd 3:0:0:0: [sdc] Attached SCSI disk
>>
>> [ 39.699348] sd 0:0:0:0: Attached scsi generic sg0 type 0
>> [ 39.699570] sd 1:0:0:0: Attached scsi generic sg1 type 0
>> [ 39.699786] sd 3:0:0:0: Attached scsi generic sg2 type 0
>>
>> [ 39.834560] md: md0 stopped.
>> [ 39.870361] md: bind<sdc1>
>> [ 39.870527] md: md1 stopped.
>> [ 39.910999] md: md0 stopped.
>> [ 39.911064] md: unbind<sdc1>
>> [ 39.911120] md: export_rdev(sdc1)
>> [ 39.929760] md: bind<sda1>
>> [ 39.929953] md: bind<sdc1>
>> [ 39.930139] md: bind<sdb1>
>> [ 39.930231] md: md1 stopped.
>> [ 39.932468] md: bind<sdc2>
>> [ 39.932674] md: bind<sda2>
>> [ 39.932860] md: bind<sdb2>
>>
>> [ 40.880217] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
>> (120034 MB)
>> [ 40.880288] sd 6:0:0:0: [sdd] Write Protect is off
>> [ 40.880340] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
>> [ 40.880367] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.880504] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
>> (120034 MB)
>> [ 40.880572] sd 6:0:0:0: [sdd] Write Protect is off
>> [ 40.880623] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
>> [ 40.880651] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.880718] sdd: sdd1 sdd2 <<7>ieee1394: Host added:
>> ID:BUS[0-00:1023] GUID[001485000012704c]
>> [ 40.908264] sdd5 >
>> [ 40.908479] sd 6:0:0:0: [sdd] Attached SCSI disk
>> [ 40.908579] sd 6:0:0:0: Attached scsi generic sg4 type 0
>>
>> [ 40.908747] scsi 6:0:1:0: Direct-Access ATA Maxtor
>> 7L250S0 BACE PQ: 0 ANSI: 5
>> [ 40.908899] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 40.908968] sd 6:0:1:0: [sde] Write Protect is off
>> [ 40.909020] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
>> [ 40.909050] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.909174] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
>> (251000 MB)
>> [ 40.909243] sd 6:0:1:0: [sde] Write Protect is off
>> [ 40.909294] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
>> [ 40.909324] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.909391] sde: sde1 sde2
>> [ 40.930218] sd 6:0:1:0: [sde] Attached SCSI disk
>> [ 40.930318] sd 6:0:1:0: Attached scsi generic sg5 type 0
>>
>> [ 40.930480] scsi 7:0:0:0: Direct-Access ATA WDC
>> WD2500JD-00H 08.0 PQ: 0 ANSI: 5
>> [ 40.930621] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 40.930689] sd 7:0:0:0: [sdf] Write Protect is off
>> [ 40.930742] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
>> [ 40.930769] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.930894] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 40.930963] sd 7:0:0:0: [sdf] Write Protect is off
>> [ 40.931015] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
>> [ 40.931044] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.931111] sdf: sdf1 sdf2
>> [ 40.948846] sd 7:0:0:0: [sdf] Attached SCSI disk
>> [ 40.948946] sd 7:0:0:0: Attached scsi generic sg6 type 0
>>
>> [ 40.949106] scsi 7:0:1:0: Direct-Access ATA WDC
>> WD2500JS-00M 02.0 PQ: 0 ANSI: 5
>> [ 40.949248] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 40.949317] sd 7:0:1:0: [sdg] Write Protect is off
>> [ 40.949368] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
>> [ 40.949396] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.949519] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
>> (250059 MB)
>> [ 40.949588] sd 7:0:1:0: [sdg] Write Protect is off
>> [ 40.949640] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
>> [ 40.949668] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
>> enabled, doesn't support DPO or FUA
>> [ 40.949734] sdg: sdg1 sdg2
>> [ 40.969827] sd 7:0:1:0: [sdg] Attached SCSI disk
>> [ 40.969926] sd 7:0:1:0: Attached scsi generic sg7 type 0
>>
>> [ 41.206078] md: md0 stopped.
>> [ 41.206137] md: unbind<sdb1>
>> [ 41.206187] md: export_rdev(sdb1)
>> [ 41.206253] md: unbind<sdc1>
>> [ 41.206302] md: export_rdev(sdc1)
>> [ 41.206360] md: unbind<sda1>
>> [ 41.206408] md: export_rdev(sda1)
>> [ 41.247389] md: bind<sdf1>
>> [ 41.247584] md: bind<sdb1>
>> [ 41.247787] md: bind<sda1>
>> [ 41.247971] md: bind<sdc1>
>> [ 41.248151] md: bind<sdg1>
>> [ 41.248325] md: bind<sde1>
>> [ 41.256718] raid5: device sde1 operational as raid disk 0
>> [ 41.256771] raid5: device sdc1 operational as raid disk 4
>> [ 41.256821] raid5: device sda1 operational as raid disk 3
>> [ 41.256870] raid5: device sdb1 operational as raid disk 2
>> [ 41.256919] raid5: device sdf1 operational as raid disk 1
>> [ 41.257426] raid5: allocated 5245kB for md0
>> [ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
>> devices, algorithm 2
>> [ 41.257538] RAID5 conf printout:
>> [ 41.257584] --- rd:5 wd:5
>> [ 41.257631] disk 0, o:1, dev:sde1
>> [ 41.257677] disk 1, o:1, dev:sdf1
>> [ 41.257724] disk 2, o:1, dev:sdb1
>> [ 41.257771] disk 3, o:1, dev:sda1
>> [ 41.257817] disk 4, o:1, dev:sdc1
>>
>> [ 41.257952] md: md1 stopped.
>> [ 41.258009] md: unbind<sdb2>
>> [ 41.258060] md: export_rdev(sdb2)
>> [ 41.258128] md: unbind<sda2>
>> [ 41.258179] md: export_rdev(sda2)
>> [ 41.258248] md: unbind<sdc2>
>> [ 41.258306] md: export_rdev(sdc2)
>> [ 41.283067] md: bind<sdc2>
>> [ 41.283297] md: bind<sda2>
>> [ 41.285235] md: bind<sdb2>
>> [ 41.306753] md: md1 stopped.
>> [ 41.306818] md: unbind<sdb2>
>> [ 41.306878] md: export_rdev(sdb2)
>> [ 41.306956] md: unbind<sda2>
>> [ 41.307007] md: export_rdev(sda2)
>> [ 41.307075] md: unbind<sdc2>
>> [ 41.307130] md: export_rdev(sdc2)
>> [ 41.312250] md: bind<sdf2>
>> [ 41.312476] md: bind<sdb2>
>> [ 41.312711] md: bind<sdg2>
>> [ 41.312922] md: bind<sdc2>
>> [ 41.313138] md: bind<sda2>
>> [ 41.313343] md: bind<sde2>
>> [ 41.313452] md: md1: raid array is not clean -- starting
>> background reconstruction
>> [ 41.322189] raid5: device sde2 operational as raid disk 0
>> [ 41.322243] raid5: device sdc2 operational as raid disk 4
>> [ 41.322292] raid5: device sdg2 operational as raid disk 3
>> [ 41.322342] raid5: device sdb2 operational as raid disk 2
>> [ 41.322391] raid5: device sdf2 operational as raid disk 1
>> [ 41.322823] raid5: allocated 5245kB for md1
>> [ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
>> devices, algorithm 2
>> [ 41.322934] RAID5 conf printout:
>> [ 41.322980] --- rd:5 wd:5
>> [ 41.323026] disk 0, o:1, dev:sde2
>> [ 41.323073] disk 1, o:1, dev:sdf2
>> [ 41.323119] disk 2, o:1, dev:sdb2
>> [ 41.323165] disk 3, o:1, dev:sdg2
>> [ 41.323212] disk 4, o:1, dev:sdc2
>>
>> [ 41.323316] md: resync of RAID array md1
>> [ 41.323364] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
>> [ 41.323415] md: using maximum available idle IO bandwidth (but
>> not more than 200000 KB/sec) for resync.
>> [ 41.323492] md: using 128k window, over a total of 231978496
>> blocks.
>>
>>
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: md rotates RAID5 spare at boot
2008-01-10 20:53 ` Jed Davidow
@ 2008-01-10 22:01 ` Neil Brown
2008-01-10 23:41 ` Jed Davidow
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: Neil Brown @ 2008-01-10 22:01 UTC (permalink / raw)
To: Jed Davidow; +Cc: Bill Davidsen, linux-raid
On Thursday January 10, jed@ultralame.com wrote:
>
> It looks to me like md inspects and attempts to assemble after each
> drive controller is scanned (from dmesg, there appears to be a failed
> bind on the first three devices after they are scanned, and then again
> when the second controller is scanned). Would the scan order cause a
> spare to be swapped in?
>
This suggests that "mdadm --incremental" is being used to assemble the
arrays. Every time udev finds a new device, it gets added to
whichever array is should be in.
If it is called as "mdadm --incremental --run", then it will get
started as soon as possible, even if it is degraded. With the
"--run", it will wait until all devices are available.
Even with "mdadm --incremental --run", you shouldn't get a resync if
the last device is added before the array is written to.
What distro are you running?
What does
grep -R mdadm /etc/udev
show?
NeilBrown
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-10 22:01 ` Neil Brown
@ 2008-01-10 23:41 ` Jed Davidow
2008-01-10 23:43 ` Jed Davidow
[not found] ` <47869D2C.7020805@ultralame.com>
2 siblings, 0 replies; 12+ messages in thread
From: Jed Davidow @ 2008-01-10 23:41 UTC (permalink / raw)
To: linux-raid; +Cc: Neil Brown, Bill Davidsen
distro: Ubuntu 7.10
Two files show up...
85-mdadm.rules:
# This file causes block devices with Linux RAID (mdadm) signatures to
# automatically cause mdadm to be run.
# See udev(8) for syntax
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
RUN+="watershed /sbin/mdadm --assemble --scan --no-degraded"
65-mdadm.vol_id.rules:
# This file causes Linux RAID (mdadm) block devices to be checked for
# further filesystems if the array is active.
# See udev(8) for syntax
SUBSYSTEM!="block", GOTO="mdadm_end"
KERNEL!="md[0-9]*", GOTO="mdadm_end"
ACTION!="add|change", GOTO="mdadm_end"
# Check array status
ATTR{md/array_state}=="|clear|inactive", GOTO="mdadm_end"
# Obtain array information
IMPORT{program}="/sbin/mdadm --detail --export $tempnode"
ENV{MD_NAME}=="?*", SYMLINK+="disk/by-id/md-name-$env{MD_NAME}"
ENV{MD_UUID}=="?*", SYMLINK+="disk/by-id/md-uuid-$env{MD_UUID}"
# by-uuid and by-label symlinks
IMPORT{program}="vol_id --export $tempnode"
OPTIONS="link_priority=-100"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", \
SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", \
SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
I see. So udev is invoking the assemble command as soon as it detects
the devices. So is it possible that the spare is not the last drive to
be detected and mdadm assembles too soon?
Neil Brown wrote:
> On Thursday January 10, jed@ultralame.com wrote:
>
>> It looks to me like md inspects and attempts to assemble after each
>> drive controller is scanned (from dmesg, there appears to be a failed
>> bind on the first three devices after they are scanned, and then again
>> when the second controller is scanned). Would the scan order cause a
>> spare to be swapped in?
>>
>>
>
> This suggests that "mdadm --incremental" is being used to assemble the
> arrays. Every time udev finds a new device, it gets added to
> whichever array is should be in.
> If it is called as "mdadm --incremental --run", then it will get
> started as soon as possible, even if it is degraded. With the
> "--run", it will wait until all devices are available.
>
> Even with "mdadm --incremental --run", you shouldn't get a resync if
> the last device is added before the array is written to.
>
> What distro are you running?
> What does
> grep -R mdadm /etc/udev
>
> show?
>
> NeilBrown
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-10 22:01 ` Neil Brown
2008-01-10 23:41 ` Jed Davidow
@ 2008-01-10 23:43 ` Jed Davidow
2008-01-11 0:24 ` Neil Brown
[not found] ` <47869D2C.7020805@ultralame.com>
2 siblings, 1 reply; 12+ messages in thread
From: Jed Davidow @ 2008-01-10 23:43 UTC (permalink / raw)
To: linux-raid
One quick question about those rules. The 65-mdadm rule looks like it
checks ACTIVE arrays for filesystems, and the 85 rule assembles arrays.
Shouldn't they run in the other order?
distro: Ubuntu 7.10
Two files show up...
85-mdadm.rules:
# This file causes block devices with Linux RAID (mdadm) signatures to
# automatically cause mdadm to be run.
# See udev(8) for syntax
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
RUN+="watershed /sbin/mdadm --assemble --scan --no-degraded"
65-mdadm.vol_id.rules:
# This file causes Linux RAID (mdadm) block devices to be checked for
# further filesystems if the array is active.
# See udev(8) for syntax
SUBSYSTEM!="block", GOTO="mdadm_end"
KERNEL!="md[0-9]*", GOTO="mdadm_end"
ACTION!="add|change", GOTO="mdadm_end"
# Check array status
ATTR{md/array_state}=="|clear|inactive", GOTO="mdadm_end"
# Obtain array information
IMPORT{program}="/sbin/mdadm --detail --export $tempnode"
ENV{MD_NAME}=="?*", SYMLINK+="disk/by-id/md-name-$env{MD_NAME}"
ENV{MD_UUID}=="?*", SYMLINK+="disk/by-id/md-uuid-$env{MD_UUID}"
# by-uuid and by-label symlinks
IMPORT{program}="vol_id --export $tempnode"
OPTIONS="link_priority=-100"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", \
SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", \
SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
I see. So udev is invoking the assemble command as soon as it detects
the devices. So is it possible that the spare is not the last drive to
be detected and mdadm assembles too soon?
Neil Brown wrote:
> On Thursday January 10, jed@ultralame.com wrote:
>
>> It looks to me like md inspects and attempts to assemble after each
>> drive controller is scanned (from dmesg, there appears to be a failed
>> bind on the first three devices after they are scanned, and then again
>> when the second controller is scanned). Would the scan order cause a
>> spare to be swapped in?
>>
>>
>
> This suggests that "mdadm --incremental" is being used to assemble the
> arrays. Every time udev finds a new device, it gets added to
> whichever array is should be in.
> If it is called as "mdadm --incremental --run", then it will get
> started as soon as possible, even if it is degraded. With the
> "--run", it will wait until all devices are available.
>
> Even with "mdadm --incremental --run", you shouldn't get a resync if
> the last device is added before the array is written to.
>
> What distro are you running?
> What does
> grep -R mdadm /etc/udev
>
> show?
>
> NeilBrown
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-10 23:43 ` Jed Davidow
@ 2008-01-11 0:24 ` Neil Brown
0 siblings, 0 replies; 12+ messages in thread
From: Neil Brown @ 2008-01-11 0:24 UTC (permalink / raw)
To: Jed Davidow; +Cc: linux-raid
On Thursday January 10, jed@ultralame.com wrote:
> One quick question about those rules. The 65-mdadm rule looks like it
> checks ACTIVE arrays for filesystems, and the 85 rule assembles arrays.
> Shouldn't they run in the other order?
>
They are fine. The '65' rule applies to arrays. I.e. it fires on an
array device once it has been started.
The '85' rule applies to component devices.
They are quite independent.
NeilBrown
>
>
>
> distro: Ubuntu 7.10
>
> Two files show up...
>
> 85-mdadm.rules:
> # This file causes block devices with Linux RAID (mdadm) signatures to
> # automatically cause mdadm to be run.
> # See udev(8) for syntax
>
> SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
> RUN+="watershed /sbin/mdadm --assemble --scan --no-degraded"
>
>
>
> 65-mdadm.vol_id.rules:
> # This file causes Linux RAID (mdadm) block devices to be checked for
> # further filesystems if the array is active.
> # See udev(8) for syntax
>
> SUBSYSTEM!="block", GOTO="mdadm_end"
> KERNEL!="md[0-9]*", GOTO="mdadm_end"
> ACTION!="add|change", GOTO="mdadm_end"
>
> # Check array status
> ATTR{md/array_state}=="|clear|inactive", GOTO="mdadm_end"
>
> # Obtain array information
> IMPORT{program}="/sbin/mdadm --detail --export $tempnode"
> ENV{MD_NAME}=="?*", SYMLINK+="disk/by-id/md-name-$env{MD_NAME}"
> ENV{MD_UUID}=="?*", SYMLINK+="disk/by-id/md-uuid-$env{MD_UUID}"
>
> # by-uuid and by-label symlinks
> IMPORT{program}="vol_id --export $tempnode"
> OPTIONS="link_priority=-100"
> ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", \
> SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
> ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", \
> SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
>
>
> I see. So udev is invoking the assemble command as soon as it detects
> the devices. So is it possible that the spare is not the last drive to
> be detected and mdadm assembles too soon?
>
>
>
> Neil Brown wrote:
> > On Thursday January 10, jed@ultralame.com wrote:
> >
> >> It looks to me like md inspects and attempts to assemble after each
> >> drive controller is scanned (from dmesg, there appears to be a failed
> >> bind on the first three devices after they are scanned, and then again
> >> when the second controller is scanned). Would the scan order cause a
> >> spare to be swapped in?
> >>
> >>
> >
> > This suggests that "mdadm --incremental" is being used to assemble the
> > arrays. Every time udev finds a new device, it gets added to
> > whichever array is should be in.
> > If it is called as "mdadm --incremental --run", then it will get
> > started as soon as possible, even if it is degraded. With the
> > "--run", it will wait until all devices are available.
> >
> > Even with "mdadm --incremental --run", you shouldn't get a resync if
> > the last device is added before the array is written to.
> >
> > What distro are you running?
> > What does
> > grep -R mdadm /etc/udev
> >
> > show?
> >
> > NeilBrown
> >
> >
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <47869D2C.7020805@ultralame.com>]
* Re: md rotates RAID5 spare at boot
[not found] ` <47869D2C.7020805@ultralame.com>
@ 2008-01-11 0:22 ` Neil Brown
2008-01-11 0:38 ` Jed Davidow
0 siblings, 1 reply; 12+ messages in thread
From: Neil Brown @ 2008-01-11 0:22 UTC (permalink / raw)
To: Jed Davidow; +Cc: linux-raid
On Thursday January 10, jed@ultralame.com wrote:
> distro: Ubuntu 7.10
>
> Two files show up...
>
> 85-mdadm.rules:
> # This file causes block devices with Linux RAID (mdadm) signatures to
> # automatically cause mdadm to be run.
> # See udev(8) for syntax
>
> SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
> RUN+="watershed /sbin/mdadm --assemble --scan --no-degraded"
....
>
> I see. So udev is invoking the assemble command as soon as it detects
> the devices. So is it possible that the spare is not the last drive to
> be detected and mdadm assembles too soon?
The "--no-degraded' should stop it from assembling until all expected
devices have been found. It could assemble before the spare is found,
but should not assemble before all the data devices have been found.
The "dmesg" trace you included in your first mail doesn't actually
show anything wrong - it never starts and incomplete array.
Can you try again and get a trace where there definitely is a rebuild
happening.
And please don't drop linux-raid from the 'cc' list.
NeilBrown
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-11 0:22 ` Neil Brown
@ 2008-01-11 0:38 ` Jed Davidow
2008-01-11 1:12 ` Neil Brown
0 siblings, 1 reply; 12+ messages in thread
From: Jed Davidow @ 2008-01-11 0:38 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
(Sorry- yes it looks like I posted an incorrect dmesg extract)
$egrep "sd|md|raid|scsi" /var/log/dmesg.0
[ 36.112449] md: linear personality registered for level -1
[ 36.117197] md: multipath personality registered for level -4
[ 36.121795] md: raid0 personality registered for level 0
[ 36.126950] md: raid1 personality registered for level 1
[ 36.131424] raid5: automatically using best checksumming function:
pIII_sse
[ 36.150020] raid5: using function: pIII_sse (4564.000 MB/sec)
[ 36.218015] raid6: int32x1 780 MB/s
[ 36.285943] raid6: int32x2 902 MB/s
[ 36.353961] raid6: int32x4 667 MB/s
[ 36.421869] raid6: int32x8 528 MB/s
[ 36.489811] raid6: mmxx1 1813 MB/s
[ 36.557775] raid6: mmxx2 2123 MB/s
[ 36.625763] raid6: sse1x1 1101 MB/s
[ 36.693717] raid6: sse1x2 1898 MB/s
[ 36.761688] raid6: sse2x1 2227 MB/s
[ 36.829647] raid6: sse2x2 3178 MB/s
[ 36.829695] raid6: using algorithm sse2x2 (3178 MB/s)
[ 36.829744] md: raid6 personality registered for level 6
[ 36.829793] md: raid5 personality registered for level 5
[ 36.829842] md: raid4 personality registered for level 4
[ 36.853475] md: raid10 personality registered for level 10
[ 37.781513] scsi0 : sata_sil
[ 37.781628] scsi1 : sata_sil
[ 37.781724] scsi2 : sata_sil
[ 37.781820] scsi3 : sata_sil
[ 37.781922] ata1: SATA max UDMA/100 cmd 0xf88c0080 ctl 0xf88c008a
bmdma 0xf88c0000 irq 20
[ 37.781997] ata2: SATA max UDMA/100 cmd 0xf88c00c0 ctl 0xf88c00ca
bmdma 0xf88c0008 irq 20
[ 37.782069] ata3: SATA max UDMA/100 cmd 0xf88c0280 ctl 0xf88c028a
bmdma 0xf88c0200 irq 20
[ 37.782142] ata4: SATA max UDMA/100 cmd 0xf88c02c0 ctl 0xf88c02ca
bmdma 0xf88c0208 irq 20
[ 39.577812] scsi 0:0:0:0: Direct-Access ATA WDC WD2500JD-00H
08.0 PQ: 0 ANSI: 5
[ 39.578027] scsi 1:0:0:0: Direct-Access ATA Maxtor 7L250S0
BACE PQ: 0 ANSI: 5
[ 39.578234] scsi 3:0:0:0: Direct-Access ATA Maxtor 7L250S0
BACE PQ: 0 ANSI: 5
[ 39.632483] scsi4 : ata_piix
[ 39.632591] scsi5 : ata_piix
[ 39.632812] ata5: PATA max UDMA/133 cmd 0x000101f0 ctl 0x000103f6
bmdma 0x0001f000 irq 14
[ 39.634522] ata6: PATA max UDMA/133 cmd 0x00010170 ctl 0x00010376
bmdma 0x0001f008 irq 15
[ 39.634924] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
(250059 MB)
[ 39.634995] sd 0:0:0:0: [sda] Write Protect is off
[ 39.635048] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 39.635076] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.635218] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors
(250059 MB)
[ 39.635292] sd 0:0:0:0: [sda] Write Protect is off
[ 39.635350] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 39.635380] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.635462] sda: sda1 sda2
[ 39.650092] sd 0:0:0:0: [sda] Attached SCSI disk
[ 39.650226] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.650296] sd 1:0:0:0: [sdb] Write Protect is off
[ 39.650348] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 39.650379] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.650505] sd 1:0:0:0: [sdb] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.650573] sd 1:0:0:0: [sdb] Write Protect is off
[ 39.650625] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 39.650657] sd 1:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.650727] sdb: sdb1 sdb2
[ 39.667599] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 39.667719] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.667788] sd 3:0:0:0: [sdc] Write Protect is off
[ 39.667840] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 39.667871] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.667997] sd 3:0:0:0: [sdc] 490234752 512-byte hardware sectors
(251000 MB)
[ 39.668064] sd 3:0:0:0: [sdc] Write Protect is off
[ 39.668116] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 39.668146] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 39.668213] sdc: sdc1 sdc2
[ 39.692703] sd 3:0:0:0: [sdc] Attached SCSI disk
[ 39.699348] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 39.699570] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 39.699786] sd 3:0:0:0: Attached scsi generic sg2 type 0
[ 39.834560] md: md0 stopped.
[ 39.870361] md: bind<sdc1>
[ 39.870527] md: md1 stopped.
[ 39.910999] md: md0 stopped.
[ 39.911064] md: unbind<sdc1>
[ 39.911120] md: export_rdev(sdc1)
[ 39.929760] md: bind<sda1>
[ 39.929953] md: bind<sdc1>
[ 39.930139] md: bind<sdb1>
[ 39.930231] md: md1 stopped.
[ 39.932468] md: bind<sdc2>
[ 39.932674] md: bind<sda2>
[ 39.932860] md: bind<sdb2>
[ 40.411001] scsi 4:0:1:0: CD-ROM LITE-ON DVDRW SOHW-1213S
TS09 PQ: 0 ANSI: 5
[ 40.411152] scsi 4:0:1:0: Attached scsi generic sg3 type 5
[ 40.411657] scsi6 : ata_piix
[ 40.411766] scsi7 : ata_piix
[ 40.411981] ata7: SATA max UDMA/133 cmd 0x0001d000 ctl 0x0001d402
bmdma 0x0001e000 irq 18
[ 40.412050] ata8: SATA max UDMA/133 cmd 0x0001d800 ctl 0x0001dc02
bmdma 0x0001e008 irq 18
[ 40.880062] scsi 6:0:0:0: Direct-Access ATA ST3120026AS
3.05 PQ: 0 ANSI: 5
[ 40.880217] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
(120034 MB)
[ 40.880288] sd 6:0:0:0: [sdd] Write Protect is off
[ 40.880340] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 40.880367] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.880504] sd 6:0:0:0: [sdd] 234441648 512-byte hardware sectors
(120034 MB)
[ 40.880572] sd 6:0:0:0: [sdd] Write Protect is off
[ 40.880623] sd 6:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 40.880651] sd 6:0:0:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.880718] sdd: sdd1 sdd2 <<7>ieee1394: Host added:
ID:BUS[0-00:1023] GUID[001485000012704c]
[ 40.908264] sdd5 >
[ 40.908479] sd 6:0:0:0: [sdd] Attached SCSI disk
[ 40.908579] sd 6:0:0:0: Attached scsi generic sg4 type 0
[ 40.908747] scsi 6:0:1:0: Direct-Access ATA Maxtor 7L250S0
BACE PQ: 0 ANSI: 5
[ 40.908899] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
(251000 MB)
[ 40.908968] sd 6:0:1:0: [sde] Write Protect is off
[ 40.909020] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
[ 40.909050] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.909174] sd 6:0:1:0: [sde] 490234752 512-byte hardware sectors
(251000 MB)
[ 40.909243] sd 6:0:1:0: [sde] Write Protect is off
[ 40.909294] sd 6:0:1:0: [sde] Mode Sense: 00 3a 00 00
[ 40.909324] sd 6:0:1:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.909391] sde: sde1 sde2
[ 40.930218] sd 6:0:1:0: [sde] Attached SCSI disk
[ 40.930318] sd 6:0:1:0: Attached scsi generic sg5 type 0
[ 40.930480] scsi 7:0:0:0: Direct-Access ATA WDC WD2500JD-00H
08.0 PQ: 0 ANSI: 5
[ 40.930621] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.930689] sd 7:0:0:0: [sdf] Write Protect is off
[ 40.930742] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 40.930769] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.930894] sd 7:0:0:0: [sdf] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.930963] sd 7:0:0:0: [sdf] Write Protect is off
[ 40.931015] sd 7:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 40.931044] sd 7:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.931111] sdf: sdf1 sdf2
[ 40.948846] sd 7:0:0:0: [sdf] Attached SCSI disk
[ 40.948946] sd 7:0:0:0: Attached scsi generic sg6 type 0
[ 40.949106] scsi 7:0:1:0: Direct-Access ATA WDC WD2500JS-00M
02.0 PQ: 0 ANSI: 5
[ 40.949248] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.949317] sd 7:0:1:0: [sdg] Write Protect is off
[ 40.949368] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
[ 40.949396] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.949519] sd 7:0:1:0: [sdg] 488397168 512-byte hardware sectors
(250059 MB)
[ 40.949588] sd 7:0:1:0: [sdg] Write Protect is off
[ 40.949640] sd 7:0:1:0: [sdg] Mode Sense: 00 3a 00 00
[ 40.949668] sd 7:0:1:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 40.949734] sdg: sdg1 sdg2
[ 40.969827] sd 7:0:1:0: [sdg] Attached SCSI disk
[ 40.969926] sd 7:0:1:0: Attached scsi generic sg7 type 0
[ 40.995542] sr0: scsi3-mmc drive: 12x/12x writer cd/rw xa/form2 cdda tray
[ 40.995965] sr 4:0:1:0: Attached scsi CD-ROM sr0
[ 41.206078] md: md0 stopped.
[ 41.206137] md: unbind<sdb1>
[ 41.206187] md: export_rdev(sdb1)
[ 41.206253] md: unbind<sdc1>
[ 41.206302] md: export_rdev(sdc1)
[ 41.206360] md: unbind<sda1>
[ 41.206408] md: export_rdev(sda1)
[ 41.247389] md: bind<sdf1>
[ 41.247584] md: bind<sdb1>
[ 41.247787] md: bind<sda1>
[ 41.247971] md: bind<sdc1>
[ 41.248151] md: bind<sdg1>
[ 41.248325] md: bind<sde1>
[ 41.256718] raid5: device sde1 operational as raid disk 0
[ 41.256771] raid5: device sdc1 operational as raid disk 4
[ 41.256821] raid5: device sda1 operational as raid disk 3
[ 41.256870] raid5: device sdb1 operational as raid disk 2
[ 41.256919] raid5: device sdf1 operational as raid disk 1
[ 41.257426] raid5: allocated 5245kB for md0
[ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
devices, algorithm 2
[ 41.257631] disk 0, o:1, dev:sde1
[ 41.257677] disk 1, o:1, dev:sdf1
[ 41.257724] disk 2, o:1, dev:sdb1
[ 41.257771] disk 3, o:1, dev:sda1
[ 41.257817] disk 4, o:1, dev:sdc1
[ 41.257952] md: md1 stopped.
[ 41.258009] md: unbind<sdb2>
[ 41.258060] md: export_rdev(sdb2)
[ 41.258128] md: unbind<sda2>
[ 41.258179] md: export_rdev(sda2)
[ 41.258248] md: unbind<sdc2>
[ 41.258306] md: export_rdev(sdc2)
[ 41.283067] md: bind<sdc2>
[ 41.283297] md: bind<sda2>
[ 41.285235] md: bind<sdb2>
[ 41.306753] md: md1 stopped.
[ 41.306818] md: unbind<sdb2>
[ 41.306878] md: export_rdev(sdb2)
[ 41.306956] md: unbind<sda2>
[ 41.307007] md: export_rdev(sda2)
[ 41.307075] md: unbind<sdc2>
[ 41.307130] md: export_rdev(sdc2)
[ 41.312250] md: bind<sdf2>
[ 41.312476] md: bind<sdb2>
[ 41.312711] md: bind<sdg2>
[ 41.312922] md: bind<sdc2>
[ 41.313138] md: bind<sda2>
[ 41.313343] md: bind<sde2>
[ 41.313452] md: md1: raid array is not clean -- starting background
reconstruction
[ 41.322189] raid5: device sde2 operational as raid disk 0
[ 41.322243] raid5: device sdc2 operational as raid disk 4
[ 41.322292] raid5: device sdg2 operational as raid disk 3
[ 41.322342] raid5: device sdb2 operational as raid disk 2
[ 41.322391] raid5: device sdf2 operational as raid disk 1
[ 41.322823] raid5: allocated 5245kB for md1
[ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
devices, algorithm 2
[ 41.323026] disk 0, o:1, dev:sde2
[ 41.323073] disk 1, o:1, dev:sdf2
[ 41.323119] disk 2, o:1, dev:sdb2
[ 41.323165] disk 3, o:1, dev:sdg2
[ 41.323212] disk 4, o:1, dev:sdc2
[ 41.323316] md: resync of RAID array md1
[ 41.323364] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 41.323415] md: using maximum available idle IO bandwidth (but not
more than 200000 KB/sec) for resync.
[ 41.323492] md: using 128k window, over a total of 231978496 blocks.
[ 50.321199] EXT3-fs: sdd1: orphan cleanup on readonly fs
[ 50.321323] EXT3-fs: sdd1: 5 orphan inodes deleted
[ 66.616890] Adding 3004112k swap on /dev/sdd5. Priority:-1 extents:1
across:3004112k
[ 66.773498] EXT3 FS on sdd1, internal journal
Neil Brown wrote:
> On Thursday January 10, jed@ultralame.com wrote:
>
>> distro: Ubuntu 7.10
>>
>> Two files show up...
>>
>> 85-mdadm.rules:
>> # This file causes block devices with Linux RAID (mdadm) signatures to
>> # automatically cause mdadm to be run.
>> # See udev(8) for syntax
>>
>> SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
>> RUN+="watershed /sbin/mdadm --assemble --scan --no-degraded"
>>
> ....
>
>> I see. So udev is invoking the assemble command as soon as it detects
>> the devices. So is it possible that the spare is not the last drive to
>> be detected and mdadm assembles too soon?
>>
>
> The "--no-degraded' should stop it from assembling until all expected
> devices have been found. It could assemble before the spare is found,
> but should not assemble before all the data devices have been found.
>
> The "dmesg" trace you included in your first mail doesn't actually
> show anything wrong - it never starts and incomplete array.
> Can you try again and get a trace where there definitely is a rebuild
> happening.
>
> And please don't drop linux-raid from the 'cc' list.
>
> NeilBrown
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: md rotates RAID5 spare at boot
2008-01-11 0:38 ` Jed Davidow
@ 2008-01-11 1:12 ` Neil Brown
0 siblings, 0 replies; 12+ messages in thread
From: Neil Brown @ 2008-01-11 1:12 UTC (permalink / raw)
To: Jed Davidow; +Cc: linux-raid
On Thursday January 10, jed@ultralame.com wrote:
> (Sorry- yes it looks like I posted an incorrect dmesg extract)
This still doesn't seem to match your description.
I see:
> [ 41.247389] md: bind<sdf1>
> [ 41.247584] md: bind<sdb1>
> [ 41.247787] md: bind<sda1>
> [ 41.247971] md: bind<sdc1>
> [ 41.248151] md: bind<sdg1>
> [ 41.248325] md: bind<sde1>
> [ 41.256718] raid5: device sde1 operational as raid disk 0
> [ 41.256771] raid5: device sdc1 operational as raid disk 4
> [ 41.256821] raid5: device sda1 operational as raid disk 3
> [ 41.256870] raid5: device sdb1 operational as raid disk 2
> [ 41.256919] raid5: device sdf1 operational as raid disk 1
> [ 41.257426] raid5: allocated 5245kB for md0
> [ 41.257476] raid5: raid level 5 set md0 active with 5 out of 5
> devices, algorithm 2
which looks like 'md0' started with 5 of 5 drives, plus g1 is there as
a spare. And
> [ 41.312250] md: bind<sdf2>
> [ 41.312476] md: bind<sdb2>
> [ 41.312711] md: bind<sdg2>
> [ 41.312922] md: bind<sdc2>
> [ 41.313138] md: bind<sda2>
> [ 41.313343] md: bind<sde2>
> [ 41.313452] md: md1: raid array is not clean -- starting background
> reconstruction
> [ 41.322189] raid5: device sde2 operational as raid disk 0
> [ 41.322243] raid5: device sdc2 operational as raid disk 4
> [ 41.322292] raid5: device sdg2 operational as raid disk 3
> [ 41.322342] raid5: device sdb2 operational as raid disk 2
> [ 41.322391] raid5: device sdf2 operational as raid disk 1
> [ 41.322823] raid5: allocated 5245kB for md1
> [ 41.322872] raid5: raid level 5 set md1 active with 5 out of 5
> devices, algorithm 2
md1 also assembled with 5/5 drives and sda2 as a spare.
This one was not shut down cleanly so it started a resync. But there
is not evidence of anything starting degraded.
????
NeilBrown
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2008-01-11 1:12 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-01-09 20:07 md rotates RAID5 spare at boot Jed Davidow
2008-01-10 18:03 ` Jed Davidow
2008-01-10 21:23 ` this goes go my megaraid probs too was: " Eric S. Johansson
2008-01-10 20:47 ` Bill Davidsen
2008-01-10 20:53 ` Jed Davidow
2008-01-10 22:01 ` Neil Brown
2008-01-10 23:41 ` Jed Davidow
2008-01-10 23:43 ` Jed Davidow
2008-01-11 0:24 ` Neil Brown
[not found] ` <47869D2C.7020805@ultralame.com>
2008-01-11 0:22 ` Neil Brown
2008-01-11 0:38 ` Jed Davidow
2008-01-11 1:12 ` Neil Brown
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).