* slow mdadm reshape, normal?
@ 2009-06-10 22:40 Michael Ole Olsen
2009-06-10 23:04 ` John Robinson
0 siblings, 1 reply; 12+ messages in thread
From: Michael Ole Olsen @ 2009-06-10 22:40 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 14158 bytes --]
is it normal that a raid5 reshape happens at only 7000kB/s on
newer disks? i created the array at 60MB/s stable over the whole array
(accordin to /proc/mdstat), sometimes up to 90MB/s
chunk size is 64kB, disks are 9xst1500 (1.5tb seagate with upgraded
firmware CC1H /SD1A)
the hardware is a pci 32bit 4port sata and onboard sata2 (6 port sata - non
ide host board) if that explains it?
modules are sata_nv and sata_sil
I have lvm2 and xfs+2TB data on top of md before the growing of two new
disks from 7 to 9.
perhaps this speed is normal?
I have tried echoing various things into proc and sys, no luck there
Any ideas?
my apology for the direct mailing, Niel :)
and sorry to everyone for wasting bandwidth with this lengthy mail if its normal :)
Here is the build+reshape log:
-------------------------------------------
while creating the raid5 as normal (the new disks has not been growed yet)
mfs:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7](S) sdb[8](S) sda[0] sdi[9] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks level 5, 64k chunk, algorithm 2 [7/6] [UUUUUU_]
[================>....] recovery = 82.5% (1209205112/1465138496) finish=71.5min speed=59647K/sec
unused devices: <none>
mfs:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
cpq:/diskless/mws 284388032 225657236 58730796 80% /
tmpfs 1817684 0 1817684 0% /lib/init/rw
udev 10240 84 10156 1% /dev
tmpfs 1817684 0 1817684 0% /dev/shm
/dev/mapper/st1500-bigdaddy
8589803488 2041498436 6548305052 24% /bigdaddy
cpq:/home/diskless/tftp/kernels/src
284388096 225657216 58730880 80% /usr/src
mfs:~# umount /bigdaddy/
mfs:~# mdadm --stop /dev/md0
mdadm: fail to stop array /dev/md0: Device or resource busy
mfs:~# mdadm --grow /dev/md0 --raid-devices=9
mdadm: Need to backup 1536K of critical section..
mdadm: /dev/md0: failed to find device 6. Array might be degraded.
--grow aborted
(reverse-i-search)`for': vi /home/michael/.forward
mfs:~# for i in a b c d e f g h i; do echo sd$i;dd if=/dev/sd$i of=/dev/null bs=1M count=200;done
sda
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,46597 s, 85,0 MB/s
sdb
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66641 s, 126 MB/s
sdc
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,51205 s, 83,5 MB/s
sdd
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 2,33702 s, 89,7 MB/s
sde
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,62686 s, 129 MB/s
sdf
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66326 s, 126 MB/s
sdg
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,59947 s, 131 MB/s
sdh
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,66347 s, 126 MB/s
sdi
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 1,6167 s, 130 MB/s
mfs:/usr/share/doc/mdadm# mdadm --detail /dev/md0
/dev/md0:
Version : 00.91
Creation Time : Tue Jun 9 21:56:04 2009
Raid Level : raid5
Array Size : 8790830976 (8383.59 GiB 9001.81 GB)
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 9
Total Devices : 9
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Jun 11 00:26:26 2009
State : clean, recovering
Active Devices : 9
Working Devices : 9
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Reshape Status : 13% complete
Delta Devices : 2, (7->9)
UUID : 5f206395:a6c11495:9091a83d:f5070ca0 (local to host mfs)
Events : 0.139246
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 32 1 active sync /dev/sdc
2 8 64 2 active sync /dev/sde
3 8 80 3 active sync /dev/sdf
4 8 96 4 active sync /dev/sdg
5 8 112 5 active sync /dev/sdh
6 8 128 6 active sync /dev/sdi
7 8 48 7 active sync /dev/sdd
8 8 16 8 active sync /dev/sdb
reshaping (used --grow with two new disks and 9 total)
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[>....................] reshape = 0.4% (6599364/1465138496) finish=3009.7min speed=8074K/sec
unused devices: <none>
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[>....................] reshape = 0.4% (6611448/1465138496) finish=3057.7min speed=7947K/sec
unused devices: <none>
mfs:/usr/share/doc/mdadm# echo 20000 > /proc/sys/dev/raid/speed_limit_min
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[>....................] reshape = 0.6% (9161780/1465138496) finish=3611.7min speed=6717K/sec
unused devices: <none>
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[>....................] reshape = 2.8% (41790784/1465138496) finish=3786.7min speed=6261K/sec
unused devices: <none>
mfs:/usr/share/doc/mdadm# echo 25000 > /proc/sys/dev/raid/speed_limit_min
mfs:/usr/share/doc/mdadm# echo 400000 >/proc/sys/dev/raid/speed_limit_max
mfs:/usr/share/doc/mdadm# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1]
8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU]
[==>..................] reshape = 13.6% (200267648/1465138496) finish=3044.2min speed=6922K/sec
mfs:/usr/share/doc/mdadm# lsmod
Module Size Used by
ext3 106356 0
jbd 40164 1 ext3
mbcache 6872 1 ext3
ipv6 198960 26
xfs 435712 0
exportfs 3628 1 xfs
raid456 116508 1
async_xor 1936 1 raid456
async_memcpy 1408 1 raid456
async_tx 2396 3 raid456,async_xor,async_memcpy
xor 13936 2 raid456,async_xor
md_mod 72944 2 raid456
dm_mod 45188 4
sd_mod 20916 9
sata_sil 8180 3
sata_nv 19468 6
ehci_hcd 28944 0
ohci_hcd 19464 0
libata 148672 2 sata_sil,sata_nv
i2c_nforce2 5804 0
scsi_mod 91884 2 sd_mod,libata
usbcore 106508 2 ehci_hcd,ohci_hcd
i2c_core 20640 1 i2c_nforce2
mfs:/usr/share/doc/mdadm# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1980 648 ? Ss Jun10 0:00 init [2]
root 2 0.0 0.0 0 0 ? S< Jun10 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< Jun10 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S< Jun10 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Jun10 0:00 [migration/1]
root 6 0.0 0.0 0 0 ? S< Jun10 0:00 [ksoftirqd/1]
root 7 0.0 0.0 0 0 ? S< Jun10 0:00 [events/0]
root 8 0.0 0.0 0 0 ? S< Jun10 0:00 [events/1]
root 9 0.0 0.0 0 0 ? S< Jun10 0:00 [work_on_cpu/0]
root 10 0.0 0.0 0 0 ? S< Jun10 0:00 [work_on_cpu/1]
root 11 0.0 0.0 0 0 ? S< Jun10 0:00 [khelper]
root 76 0.0 0.0 0 0 ? S< Jun10 0:32 [kblockd/0]
root 77 0.0 0.0 0 0 ? S< Jun10 0:01 [kblockd/1]
root 78 0.0 0.0 0 0 ? S< Jun10 0:00 [kacpid]
root 79 0.0 0.0 0 0 ? S< Jun10 0:00 [kacpi_notify]
root 187 0.0 0.0 0 0 ? S< Jun10 0:00 [kseriod]
root 236 1.5 0.0 0 0 ? S< Jun10 22:28 [kswapd0]
root 278 0.0 0.0 0 0 ? S< Jun10 0:00 [aio/0]
root 279 0.0 0.0 0 0 ? S< Jun10 0:00 [aio/1]
root 283 0.0 0.0 0 0 ? S< Jun10 0:00 [nfsiod]
root 998 0.0 0.0 0 0 ? S< Jun10 0:01 [rpciod/0]
root 999 0.0 0.0 0 0 ? S< Jun10 0:00 [rpciod/1]
root 1087 0.0 0.0 2144 764 ? S<s Jun10 0:00 udevd --daemon
root 2041 0.0 0.0 0 0 ? S< Jun10 0:00 [ksuspend_usbd]
root 2045 0.0 0.0 0 0 ? S< Jun10 0:00 [khubd]
root 2091 0.0 0.0 0 0 ? S< Jun10 0:00 [ata/0]
root 2092 0.0 0.0 0 0 ? S< Jun10 0:00 [ata/1]
root 2093 0.0 0.0 0 0 ? S< Jun10 0:00 [ata_aux]
root 2106 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_0]
root 2107 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_1]
root 2108 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_2]
root 2109 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_3]
root 2116 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_4]
root 2117 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_5]
root 2190 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_6]
root 2191 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_7]
root 2194 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_8]
root 2195 0.0 0.0 0 0 ? S< Jun10 0:00 [scsi_eh_9]
root 2420 0.0 0.0 0 0 ? S< Jun10 0:00 [kstriped]
root 2460 18.6 0.0 0 0 ? S< Jun10 271:03 [md0_raid5]
root 2482 0.0 0.0 0 0 ? S< Jun10 0:00 [kdmflush]
root 2518 0.0 0.0 0 0 ? S< Jun10 0:00 [xfs_mru_cache]
root 2522 0.0 0.0 0 0 ? S< Jun10 0:03 [xfslogd/0]
root 2523 0.0 0.0 0 0 ? S< Jun10 0:00 [xfslogd/1]
root 2524 0.4 0.0 0 0 ? S< Jun10 6:45 [xfsdatad/0]
root 2525 0.0 0.0 0 0 ? S< Jun10 0:23 [xfsdatad/1]
daemon 2581 0.0 0.0 1764 444 ? Ss Jun10 0:00 /sbin/portmap
statd 2592 0.0 0.0 1824 624 ? Ss Jun10 0:00 /sbin/rpc.statd
root 2747 0.0 0.0 5272 916 ? Ss Jun10 0:00 /usr/sbin/sshd
root 3025 0.0 0.0 3080 600 ? S Jun10 0:00 /usr/sbin/smartd --pidfile /var/run/smartd.pid --interval=1800
ntp 3038 0.0 0.0 4132 1044 ? Ss Jun10 0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 105:105 -g
root 3048 0.0 0.0 2012 572 ? Ss Jun10 0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog
root 3060 0.0 0.0 0 0 ? S< Jun10 0:00 [lockd]
root 3064 0.0 0.0 1644 504 tty1 Ss+ Jun10 0:00 /sbin/getty 38400 tty1
root 3067 0.0 0.0 1644 496 tty2 Ss+ Jun10 0:00 /sbin/getty 38400 tty2
root 3074 0.0 0.0 7992 1336 ? Ss Jun10 0:00 sshd: michael [priv]
michael 3076 0.0 0.0 7992 1084 ? S Jun10 0:00 sshd: michael@pts/0
michael 3077 0.0 0.0 4620 1904 pts/0 Ss Jun10 0:00 -bash
root 3084 0.0 0.0 3636 972 pts/0 S Jun10 0:00 su
root 3085 0.0 0.0 4132 1740 pts/0 S+ Jun10 0:00 bash
root 3147 0.0 0.0 7992 1336 ? Ss Jun10 0:00 sshd: michael [priv]
michael 3149 0.0 0.0 8140 1068 ? S Jun10 0:06 sshd: michael@pts/1
michael 3150 0.0 0.0 4632 1924 pts/1 Ss Jun10 0:00 -bash
root 3283 0.0 0.0 3636 976 pts/1 S Jun10 0:00 su
root 3284 0.0 0.0 4160 1776 pts/1 S Jun10 0:00 bash
root 3474 0.0 0.0 7924 1316 ? Ss Jun10 0:00 /usr/sbin/nmbd -D
root 3476 0.0 0.0 13388 2116 ? Ss Jun10 0:00 /usr/sbin/smbd -D
root 3478 0.0 0.0 13388 892 ? S Jun10 0:00 /usr/sbin/smbd -D
106 4608 0.0 0.0 6132 924 ? Ss Jun10 0:00 /usr/sbin/exim4 -bd -q30m
root 4702 0.2 0.0 0 0 ? S Jun10 2:35 [pdflush]
root 4707 0.2 0.0 0 0 ? S Jun10 2:09 [pdflush]
root 4870 0.0 0.0 0 0 ? S< Jun10 0:00 [kdmflush]
root 4966 3.4 0.0 0 0 ? D< Jun10 16:12 [md0_reshape]
root 5338 0.0 0.0 3580 1012 pts/1 R+ 00:25 0:00 ps aux
mfs:/usr/share/doc/mdadm# w
00:25:16 up 1 day, 12 min, 2 users, load average: 1,38, 1,30, 1,27
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
michael pts/0 cpq Wed00 1:06m 0.17s 0.03s sshd: michael [priv]
michael pts/1 cpq Wed00 0.00s 0.40s 0.07s sshd: michael [priv]
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 835 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: slow mdadm reshape, normal? 2009-06-10 22:40 slow mdadm reshape, normal? Michael Ole Olsen @ 2009-06-10 23:04 ` John Robinson 0 siblings, 0 replies; 12+ messages in thread From: John Robinson @ 2009-06-10 23:04 UTC (permalink / raw) To: linux-raid On 10/06/2009 23:40, Michael Ole Olsen wrote: > is it normal that a raid5 reshape happens at only 7000kB/s on > newer disks? i created the array at 60MB/s stable over the whole array > (accordin to /proc/mdstat), sometimes up to 90MB/s > > chunk size is 64kB, disks are 9xst1500 (1.5tb seagate with upgraded > firmware CC1H /SD1A) > > the hardware is a pci 32bit 4port sata and onboard sata2 (6 port sata - non > ide host board) if that explains it? > > modules are sata_nv and sata_sil > > I have lvm2 and xfs+2TB data on top of md before the growing of two new > disks from 7 to 9. I wonder if this is the same issue as in another thread this evening[1]: too many drives competing for the very limited bandwidth available on the PCI bus? Your extra discs, are they attached to the PCI SATA card? How many drives are there on there? Cheers, John. [1] http://marc.info/?l=linux-raid&m=124466834109307&w=2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* FW: Detecting errors on the RAID disks.
@ 2009-05-27 15:28 Simon Jackson
2009-05-27 19:35 ` Richard Scobie
[not found] ` <20090610224055.GW30825@xxxxxxxxx>
0 siblings, 2 replies; 12+ messages in thread
From: Simon Jackson @ 2009-05-27 15:28 UTC (permalink / raw)
To: linux-raid
I have been looking for a way of detecting whether disks used in RAID sets have had errors that might indicate the drive is failing..
I have noticed the files "/sys/block/mdX/md/rY/errors". They appear to contain a count value.
What errors are reported in this file?
Does this provide a way of detecting a dying disk?
Are there any better ways to achieve this?
Thanks Simon.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: FW: Detecting errors on the RAID disks. 2009-05-27 15:28 FW: Detecting errors on the RAID disks Simon Jackson @ 2009-05-27 19:35 ` Richard Scobie 2009-05-28 8:19 ` Simon Jackson [not found] ` <20090610224055.GW30825@xxxxxxxxx> 1 sibling, 1 reply; 12+ messages in thread From: Richard Scobie @ 2009-05-27 19:35 UTC (permalink / raw) To: Simon Jackson; +Cc: linux-raid Simon Jackson wrote: > I have been looking for a way of detecting whether disks used in RAID sets have had errors that might indicate the drive is failing.. smartd, while not foolproof, offers some assistence here: http://smartmontools.sourceforge.net/ Regards, Richard ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: FW: Detecting errors on the RAID disks. 2009-05-27 19:35 ` Richard Scobie @ 2009-05-28 8:19 ` Simon Jackson 0 siblings, 0 replies; 12+ messages in thread From: Simon Jackson @ 2009-05-28 8:19 UTC (permalink / raw) To: Richard Scobie; +Cc: linux-raid Richard. Thanks for the pointer, I shall look at that. I am still interested to know what the errors file provides a count of. Simon. -----Original Message----- From: Richard Scobie [mailto:richard@sauce.co.nz] Sent: 27 May 2009 20:35 To: Simon Jackson Cc: linux-raid@vger.kernel.org Subject: Re: FW: Detecting errors on the RAID disks. Simon Jackson wrote: > I have been looking for a way of detecting whether disks used in RAID sets have had errors that might indicate the drive is failing.. smartd, while not foolproof, offers some assistence here: http://smartmontools.sourceforge.net/ Regards, Richard ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <20090610224055.GW30825@xxxxxxxxx>]
* Re: Re: slow mdadm reshape, normal? [not found] ` <20090610224055.GW30825@xxxxxxxxx> @ 2009-06-10 23:36 ` Michael Ole Olsen 2009-06-11 8:19 ` Robin Hill 0 siblings, 1 reply; 12+ messages in thread From: Michael Ole Olsen @ 2009-06-10 23:36 UTC (permalink / raw) To: linux-raid the slow disks (them with 85MB/s) in the posting are the one on the pci 4x sata controller the 32bit pci sata (sata_sil) controller has 3 disks connected the onboard sata2 6x sata connectors have 6 disks connected (p5n32-e sli motherboard), i believe all these are sata2 and not shared with an ide controller. so only the pci controller should have bus limitations, but i wonder why it is that slow on reshape compared to build. unfortunately I have not gotten the recent postings on the mailinglist since 28th last month perhaps my subscription got cleared so I didn't see any reshape postings, just signed up on new. here you can see the 3 slow disks pci32bit compared to onboard sata2 (sdb): sda 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,46597 s, 85,0 MB/s sdb 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 1,66641 s, 126 MB/s sdc 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,51205 s, 83,5 MB/s sdd 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,33702 s, 89,7 MB/s are there any good tools you can recommend to test bus congestion as the probable cause of this slowness? /Michael ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? 2009-06-10 23:36 ` Re: slow mdadm reshape, normal? Michael Ole Olsen @ 2009-06-11 8:19 ` Robin Hill 2009-06-11 16:07 ` Michael Ole Olsen 0 siblings, 1 reply; 12+ messages in thread From: Robin Hill @ 2009-06-11 8:19 UTC (permalink / raw) To: linux-raid [-- Attachment #1: Type: text/plain, Size: 1137 bytes --] On Thu Jun 11, 2009 at 01:36:59AM +0200, Michael Ole Olsen wrote: > the slow disks (them with 85MB/s) in the posting are the one on the pci 4x sata controller > > the 32bit pci sata (sata_sil) controller has 3 disks connected > > the onboard sata2 6x sata connectors have 6 disks connected > (p5n32-e sli motherboard), i believe all these are sata2 and not shared with > an ide controller. > > so only the pci controller should have bus limitations, but i wonder why it is > that slow on reshape compared to build. > The build was reading (sequentially) from all drives and writing (sequentially) only to one, whereas a reshape is reading from and writing to all the drives (including seeking between reads & writes). This means it is inevitably a lot slower. In addition, you're now using 3 rather than 2 drives on the PCI bus, which will increase the congestion. Cheers, Robin -- ___ ( ' } | Robin Hill <robin@robinhill.me.uk> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" | [-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? 2009-06-11 8:19 ` Robin Hill @ 2009-06-11 16:07 ` Michael Ole Olsen 2009-06-11 19:45 ` Michael Ole Olsen 0 siblings, 1 reply; 12+ messages in thread From: Michael Ole Olsen @ 2009-06-11 16:07 UTC (permalink / raw) To: linux-raid [-- Attachment #1: Type: text/plain, Size: 1751 bytes --] Thanks I was just hoping for something like 127MB/s divided by 3 = 42MB/s which is the practical limit of the pci bus assuming each disk gets its equal share. 6MB/s is only 14% of that, so I wonder where the lost bandwidth goes :) The reshape takes only 10-15% cpu or so on my 3ghz cpu so dont think that is the bottleneck. Might be interesting to test a pci network controller at the same time had I installed one before the build, that way i could see if the pci bus really got congested. On Thu, 11 Jun 2009, Robin Hill wrote: > On Thu Jun 11, 2009 at 01:36:59AM +0200, Michael Ole Olsen wrote: > > > the slow disks (them with 85MB/s) in the posting are the one on the pci 4x sata controller > > > > the 32bit pci sata (sata_sil) controller has 3 disks connected > > > > the onboard sata2 6x sata connectors have 6 disks connected > > (p5n32-e sli motherboard), i believe all these are sata2 and not shared with > > an ide controller. > > > > so only the pci controller should have bus limitations, but i wonder why it is > > that slow on reshape compared to build. > > > The build was reading (sequentially) from all drives and writing > (sequentially) only to one, whereas a reshape is reading from and > writing to all the drives (including seeking between reads & writes). > This means it is inevitably a lot slower. In addition, you're now using > 3 rather than 2 drives on the PCI bus, which will increase the > congestion. > > Cheers, > Robin > -- > ___ > ( ' } | Robin Hill <robin@robinhill.me.uk> | > / / ) | Little Jim says .... | > // !! | "He fallen in de water !!" | [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? 2009-06-11 16:07 ` Michael Ole Olsen @ 2009-06-11 19:45 ` Michael Ole Olsen 2009-06-11 20:50 ` John Robinson 2009-06-12 0:53 ` slow mdadm reshape, normal? Roger Heflin 0 siblings, 2 replies; 12+ messages in thread From: Michael Ole Olsen @ 2009-06-11 19:45 UTC (permalink / raw) To: linux-raid [-- Attachment #1: Type: text/plain, Size: 9272 bytes --] Here is some new info (iostat) about the 7MB/s slowness on my reshape with 3x sata disks on pci32bit controller and 6x sata on onboard sata2. sata_sil and sata_nv sda 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,46597 s, 85,0 MB/s sdb 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 1,66641 s, 126 MB/s sdc 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,51205 s, 83,5 MB/s sdd 200+0 records in 200+0 records out 209715200 bytes (210 MB) copied, 2,33702 s, 89,7 MB/s Here the 3 of them are the ones on pci32 controller, the other faster ones are on onboard non limited bus. mfs:/sys/block/md0/md# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1] 8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU] [==========>..........] reshape = 51.1% (749118592/1465138496) finish=1652.2min speed=7221K/sec unused devices: <none> iostat about 50% in the reshape process: mfs:/sys/block/md0/md# iostat -x Linux 2.6.29.3mfs_diskless (mfs) 2009-06-11 _i686_ avg-cpu: %user %nice %system %iowait %steal %idle 0,16 0,00 15,15 6,41 0,00 78,28 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 3704,06 1627,96 80,37 42,38 3992,51 13379,26 141,52 1,71 13,95 6,20 76,05 sdb 26,39 1082,62 30,03 62,18 13813,39 9170,85 249,26 0,21 2,23 1,78 16,43 sdc 3724,00 1631,34 70,22 39,28 4072,93 13382,07 159,40 2,28 20,79 7,90 86,48 sdd 23,03 1108,80 28,68 36,31 11215,20 9173,71 313,71 0,84 12,87 8,40 54,63 sde 3632,20 1616,12 166,13 55,42 4102,69 13387,29 78,94 0,75 3,39 1,29 28,48 sdf 3630,85 1615,61 166,97 55,42 4097,89 13383,03 78,60 0,71 3,18 1,24 27,60 sdg 3625,92 1614,66 168,60 56,45 4071,61 13383,51 77,56 0,65 2,88 1,17 26,36 sdh 3615,89 1613,62 168,65 57,71 3991,22 13385,10 76,77 0,63 2,76 1,15 26,09 sdi 1457,20 3823,78 66,63 90,17 12200,35 5030,01 109,88 0,74 4,69 2,16 33,89 md0 0,00 0,00 0,01 720,25 0,09 25050,75 34,78 0,00 0,00 0,00 0,00 dm-0 0,00 0,00 0,01 720,25 0,08 25050,75 34,78 15,42 21,41 0,25 17,96 dm-1 0,00 0,00 0,00 0,00 0,00 0,00 8,00 0,00 7,95 3,21 0,00 sda,sdc,sdd = addon pci sata_sil pci32 card the rest are onboard sata2 controller. seems there is a lot of wait for those disks, and that it is the pci controller which is the cause there seems to be only 2-4ms wait time for onboard sata disks, but 12-20ms on addon pci board. also the %util is almost 100% on each of the 3 disks on the pci controller but I still don't understand it, but somehow the whole system must be waiting for the pci bus :) it seems there is 21ms wait time for each request due to this pci slowness. (await is in miliseconds) might be the controller that isnt so fast to do simultaneous read and writes, perhaps because its NCQ support might be bad? so I don't really have more ideas, except to buy a new controller card from a better brand (Adaptec) :( Im planning another reshape and I dont want it to take 3 days again. Best Regards, Michael Ole Olsen On Thu, 11 Jun 2009, Michael Ole Olsen wrote: > Thanks > > I was just hoping for something like 127MB/s divided by 3 = 42MB/s which > is the practical limit of the pci bus assuming each disk gets its equal > share. > > 6MB/s is only 14% of that, so I wonder where the lost bandwidth goes :) > > The reshape takes only 10-15% cpu or so on my 3ghz cpu so dont think that is > the bottleneck. > > Might be interesting to test a pci network controller at the same time had I > installed one before the build, that way i could see if the pci bus really got > congested. > > > On Thu, 11 Jun 2009, Robin Hill wrote: > > > On Thu Jun 11, 2009 at 01:36:59AM +0200, Michael Ole Olsen wrote: > > > > > the slow disks (them with 85MB/s) in the posting are the one on the pci 4x sata controller > > > > > > the 32bit pci sata (sata_sil) controller has 3 disks connected > > > > > > the onboard sata2 6x sata connectors have 6 disks connected > > > (p5n32-e sli motherboard), i believe all these are sata2 and not shared with > > > an ide controller. > > > > > > so only the pci controller should have bus limitations, but i wonder why it is > > > that slow on reshape compared to build. > > > > > The build was reading (sequentially) from all drives and writing > > (sequentially) only to one, whereas a reshape is reading from and > > writing to all the drives (including seeking between reads & writes). > > This means it is inevitably a lot slower. In addition, you're now using > > 3 rather than 2 drives on the PCI bus, which will increase the > > congestion. > > > > Cheers, > > Robin > > -- > > ___ > > ( ' } | Robin Hill <robin@robinhill.me.uk> | > > / / ) | Little Jim says .... | > > // !! | "He fallen in de water !!" | > > [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? 2009-06-11 19:45 ` Michael Ole Olsen @ 2009-06-11 20:50 ` John Robinson 2009-06-11 21:11 ` slow mdadm reshape, normal? lspci/iostat info Michael Ole Olsen 2009-06-12 0:53 ` slow mdadm reshape, normal? Roger Heflin 1 sibling, 1 reply; 12+ messages in thread From: John Robinson @ 2009-06-11 20:50 UTC (permalink / raw) To: linux-raid On 11/06/2009 20:45, Michael Ole Olsen wrote: [...] > it seems there is 21ms wait time for each request due to this pci slowness. (await is in miliseconds) > > might be the controller that isnt so fast to do simultaneous read and writes, > perhaps because its NCQ support might be bad? > > so I don't really have more ideas, except to buy a new controller card from a better brand (Adaptec) :( I have no personal experience of SIL SATA cards, but they've been mentioned before as being behind slowness. It's also possible something else on your PCI bus is choking things, so before you go and blow lots of real money on a better brand, what else is on your PCI bus? Please tell me the output of `lspci -tv` and the contents of /proc/interrupts twice (before and after a short delay, e.g. `cat /proc/interrupts ; sleep 5 ; cat /proc/interrupts`). Cheers, John. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? lspci/iostat info 2009-06-11 20:50 ` John Robinson @ 2009-06-11 21:11 ` Michael Ole Olsen 0 siblings, 0 replies; 12+ messages in thread From: Michael Ole Olsen @ 2009-06-11 21:11 UTC (permalink / raw) To: linux-raid [-- Attachment #1: Type: text/plain, Size: 9252 bytes --] Here is some more info, I only have an old Matrox card installed on the PCI bus and the Silicon controller except for the reserved motherboard stuff. mfs:/sys/block/md0/md# lspci -tv -[0000:00]-+-00.0 nVidia Corporation C55 Host Bridge +-00.1 nVidia Corporation C55 Memory Controller +-00.2 nVidia Corporation C55 Memory Controller +-00.3 nVidia Corporation C55 Memory Controller +-00.4 nVidia Corporation C55 Memory Controller +-00.5 nVidia Corporation C55 Memory Controller +-00.6 nVidia Corporation C55 Memory Controller +-00.7 nVidia Corporation C55 Memory Controller +-01.0 nVidia Corporation C55 Memory Controller +-01.1 nVidia Corporation C55 Memory Controller +-01.2 nVidia Corporation C55 Memory Controller +-01.3 nVidia Corporation C55 Memory Controller +-01.4 nVidia Corporation C55 Memory Controller +-01.5 nVidia Corporation C55 Memory Controller +-01.6 nVidia Corporation C55 Memory Controller +-02.0 nVidia Corporation C55 Memory Controller +-02.1 nVidia Corporation C55 Memory Controller +-02.2 nVidia Corporation C55 Memory Controller +-09.0 nVidia Corporation MCP55 Memory Controller +-0a.0 nVidia Corporation MCP55 LPC Bridge +-0a.1 nVidia Corporation MCP55 SMBus +-0b.0 nVidia Corporation MCP55 USB Controller +-0b.1 nVidia Corporation MCP55 USB Controller +-0d.0 nVidia Corporation MCP55 IDE +-0e.0 nVidia Corporation MCP55 SATA Controller +-0e.1 nVidia Corporation MCP55 SATA Controller +-0e.2 nVidia Corporation MCP55 SATA Controller +-0f.0-[0000:01]--+-06.0 Matrox Graphics, Inc. MGA 1064SG [Mystique] | \-07.0 Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller +-11.0 nVidia Corporation MCP55 Ethernet +-12.0 nVidia Corporation MCP55 Ethernet \-13.0-[0000:02]-- mfs:/sys/block/md0/md# cat /proc/interrupts ; sleep 5 ; cat /proc/interrupts CPU0 CPU1 0: 1028 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 7: 1 0 IO-APIC-edge 9: 0 0 IO-APIC-fasteoi acpi 12: 3 0 IO-APIC-edge i8042 17: 51373228 0 IO-APIC-fasteoi sata_sil 20: 0 0 IO-APIC-fasteoi ehci_hcd:usb2 21: 186438698 0 IO-APIC-fasteoi ohci_hcd:usb1, sata_nv 22: 207200954 0 IO-APIC-fasteoi sata_nv 23: 169890292 0 IO-APIC-fasteoi eth0, sata_nv NMI: 0 0 Non-maskable interrupts LOC: 42237922 42340136 Local timer interrupts RES: 1091032 2105741 Rescheduling interrupts CAL: 15 252 Function call interrupts TLB: 729 1726 TLB shootdowns SPU: 0 0 Spurious interrupts ERR: 1 MIS: 0 CPU0 CPU1 0: 1028 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 7: 1 0 IO-APIC-edge 9: 0 0 IO-APIC-fasteoi acpi 12: 3 0 IO-APIC-edge i8042 17: 51374643 0 IO-APIC-fasteoi sata_sil 20: 0 0 IO-APIC-fasteoi ehci_hcd:usb2 21: 186443797 0 IO-APIC-fasteoi ohci_hcd:usb1, sata_nv 22: 207205898 0 IO-APIC-fasteoi sata_nv 23: 169894870 0 IO-APIC-fasteoi eth0, sata_nv NMI: 0 0 Non-maskable interrupts LOC: 42239176 42341394 Local timer interrupts RES: 1091032 2105741 Rescheduling interrupts CAL: 15 252 Function call interrupts TLB: 729 1731 TLB shootdowns SPU: 0 0 Spurious interrupts ERR: 1 MIS: 0 mfs:/sys/block/md0/md# cat /proc/interrupts ; sleep 5 ; cat /proc/interrupts CPU0 CPU1 0: 1028 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 7: 1 0 IO-APIC-edge 9: 0 0 IO-APIC-fasteoi acpi 12: 3 0 IO-APIC-edge i8042 17: 51375118 0 IO-APIC-fasteoi sata_sil 20: 0 0 IO-APIC-fasteoi ehci_hcd:usb2 21: 186445519 0 IO-APIC-fasteoi ohci_hcd:usb1, sata_nv 22: 207207539 0 IO-APIC-fasteoi sata_nv 23: 169896360 0 IO-APIC-fasteoi eth0, sata_nv NMI: 0 0 Non-maskable interrupts LOC: 42239623 42341843 Local timer interrupts RES: 1091032 2105741 Rescheduling interrupts CAL: 15 252 Function call interrupts TLB: 729 1733 TLB shootdowns SPU: 0 0 Spurious interrupts ERR: 1 MIS: 0 CPU0 CPU1 0: 1028 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 7: 1 0 IO-APIC-edge 9: 0 0 IO-APIC-fasteoi acpi 12: 3 0 IO-APIC-edge i8042 17: 51376507 0 IO-APIC-fasteoi sata_sil 20: 0 0 IO-APIC-fasteoi ehci_hcd:usb2 21: 186450537 0 IO-APIC-fasteoi ohci_hcd:usb1, sata_nv 22: 207212461 0 IO-APIC-fasteoi sata_nv 23: 169900833 0 IO-APIC-fasteoi eth0, sata_nv NMI: 0 0 Non-maskable interrupts LOC: 42240876 42343099 Local timer interrupts RES: 1091032 2105741 Rescheduling interrupts CAL: 15 252 Function call interrupts TLB: 729 1736 TLB shootdowns SPU: 0 0 Spurious interrupts ERR: 1 MIS: 0 mfs:/sys/block/md0/md# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1] 8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU] [==========>..........] reshape = 53.6% (786554368/1465138496) finish=1539.3min speed=7346K/sec unused devices: <none> mfs:/sys/block/md0/md# iostat -x Linux 2.6.29.3mfs_diskless (mfs) 2009-06-11 _i686_ avg-cpu: %user %nice %system %iowait %steal %idle 0,15 0,00 15,02 6,20 0,00 78,62 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 3658,88 1632,25 79,87 42,53 4495,42 13414,83 146,32 1,71 13,93 6,21 75,98 sdb 25,52 1103,10 29,03 63,48 13357,33 9345,37 245,39 0,21 2,22 1,77 16,38 sdc 3678,72 1635,67 69,49 39,37 4573,18 13417,55 165,26 2,28 20,88 7,95 86,55 sdd 22,27 1129,86 27,74 37,01 10844,92 9348,14 311,85 0,84 13,00 8,48 54,92 sde 3588,14 1620,34 164,06 55,60 4601,96 13422,60 82,06 0,75 3,40 1,30 28,57 sdf 3586,83 1619,85 164,88 55,59 4597,31 13418,48 81,72 0,71 3,20 1,26 27,76 sdg 3581,97 1618,89 166,54 56,64 4571,90 13418,94 80,61 0,65 2,90 1,19 26,50 sdh 3572,31 1617,86 166,54 57,87 4494,17 13420,49 79,83 0,62 2,78 1,17 26,26 sdi 1484,87 3755,18 67,92 89,13 12432,27 5341,24 113,17 0,74 4,73 2,17 34,13 md0 0,00 0,00 0,01 696,47 0,09 24223,68 34,78 0,00 0,00 0,00 0,00 dm-0 0,00 0,00 0,01 696,47 0,07 24223,68 34,78 14,91 21,41 0,25 17,37 dm-1 0,00 0,00 0,00 0,00 0,00 0,00 8,00 0,00 7,95 3,21 0,00 On Thu, 11 Jun 2009, John Robinson wrote: > On 11/06/2009 20:45, Michael Ole Olsen wrote: > [...] >> it seems there is 21ms wait time for each request due to this pci slowness. (await is in miliseconds) >> >> might be the controller that isnt so fast to do simultaneous read and >> writes, perhaps because its NCQ support might be bad? >> >> so I don't really have more ideas, except to buy a new controller card from a better brand (Adaptec) :( > > I have no personal experience of SIL SATA cards, but they've been > mentioned before as being behind slowness. It's also possible something > else on your PCI bus is choking things, so before you go and blow lots > of real money on a better brand, what else is on your PCI bus? Please > tell me the output of `lspci -tv` and the contents of /proc/interrupts > twice (before and after a short delay, e.g. `cat /proc/interrupts ; > sleep 5 ; cat /proc/interrupts`). > > Cheers, > > John. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 835 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: slow mdadm reshape, normal? 2009-06-11 19:45 ` Michael Ole Olsen 2009-06-11 20:50 ` John Robinson @ 2009-06-12 0:53 ` Roger Heflin 1 sibling, 0 replies; 12+ messages in thread From: Roger Heflin @ 2009-06-12 0:53 UTC (permalink / raw) To: linux-raid Michael Ole Olsen wrote: > Here is some new info (iostat) about the 7MB/s slowness on my reshape > with 3x sata disks on pci32bit controller and 6x sata on onboard sata2. > > sata_sil and sata_nv > > > sda > 200+0 records in > 200+0 records out > 209715200 bytes (210 MB) copied, 2,46597 s, 85,0 MB/s > sdb > 200+0 records in > 200+0 records out > 209715200 bytes (210 MB) copied, 1,66641 s, 126 MB/s > sdc > 200+0 records in > 200+0 records out > 209715200 bytes (210 MB) copied, 2,51205 s, 83,5 MB/s > sdd > 200+0 records in > 200+0 records out > 209715200 bytes (210 MB) copied, 2,33702 s, 89,7 MB/s > > Here the 3 of them are the ones on pci32 controller, > the other faster ones are on onboard non limited bus. > > > > mfs:/sys/block/md0/md# cat /proc/mdstat > Personalities : [raid6] [raid5] [raid4] > md0 : active raid5 sdd[7] sdb[8] sda[0] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdc[1] > 8790830976 blocks super 0.91 level 5, 64k chunk, algorithm 2 [9/9] [UUUUUUUUU] > [==========>..........] reshape = 51.1% (749118592/1465138496) finish=1652.2min speed=7221K/sec > > unused devices: <none> > > > > iostat about 50% in the reshape process: > > mfs:/sys/block/md0/md# iostat -x > Linux 2.6.29.3mfs_diskless (mfs) 2009-06-11 _i686_ > > avg-cpu: %user %nice %system %iowait %steal %idle > 0,16 0,00 15,15 6,41 0,00 78,28 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util > sda 3704,06 1627,96 80,37 42,38 3992,51 13379,26 141,52 1,71 13,95 6,20 76,05 > sdb 26,39 1082,62 30,03 62,18 13813,39 9170,85 249,26 0,21 2,23 1,78 16,43 > sdc 3724,00 1631,34 70,22 39,28 4072,93 13382,07 159,40 2,28 20,79 7,90 86,48 > sdd 23,03 1108,80 28,68 36,31 11215,20 9173,71 313,71 0,84 12,87 8,40 54,63 > sde 3632,20 1616,12 166,13 55,42 4102,69 13387,29 78,94 0,75 3,39 1,29 28,48 > sdf 3630,85 1615,61 166,97 55,42 4097,89 13383,03 78,60 0,71 3,18 1,24 27,60 > sdg 3625,92 1614,66 168,60 56,45 4071,61 13383,51 77,56 0,65 2,88 1,17 26,36 > sdh 3615,89 1613,62 168,65 57,71 3991,22 13385,10 76,77 0,63 2,76 1,15 26,09 > sdi 1457,20 3823,78 66,63 90,17 12200,35 5030,01 109,88 0,74 4,69 2,16 33,89 > md0 0,00 0,00 0,01 720,25 0,09 25050,75 34,78 0,00 0,00 0,00 0,00 > dm-0 0,00 0,00 0,01 720,25 0,08 25050,75 34,78 15,42 21,41 0,25 17,96 > dm-1 0,00 0,00 0,00 0,00 0,00 0,00 8,00 0,00 7,95 3,21 0,00 > > sda,sdc,sdd = addon pci sata_sil pci32 card > the rest are onboard sata2 controller. > > seems there is a lot of wait for those disks, and that it is the pci controller which is the cause > > there seems to be only 2-4ms wait time for onboard sata disks, but 12-20ms on addon pci board. > > also the %util is almost 100% on each of the 3 disks on the pci controller > > but I still don't understand it, but somehow the whole system must be waiting for the pci bus :) > > it seems there is 21ms wait time for each request due to this pci slowness. (await is in miliseconds) > > might be the controller that isnt so fast to do simultaneous read and writes, > perhaps because its NCQ support might be bad? > > so I don't really have more ideas, except to buy a new controller card from a better brand (Adaptec) :( > > Im planning another reshape and I dont want it to take 3 days again. > > Best Regards, > Michael Ole Olsen It does not really matter what kind of 32bit PCI card that yout put in a desktop pci bus. A standard pci-32 control has in theory about 133 mb/second, in reality it has around 90mb/second. Unless you are changing to a pci-x (500mb/second+ at 66mhz-but only on "server" boards) or a pcie-x1 (266mb/second) or pcie-x4 (1024mb/second) I would not expect a major improvement. And *ALL* of the pci bus bandwidth is shared between all of the slots so more cards won't help on a desktop pci board, on the server boards the pci-x slots only usually share bandwidth with at most one other slot. With pcie of almost any type they don't share bandwidth so multiple cards help when the slots are not sharing bandwidth, but may actually make things worse if you are sharing bandwidth. If you really want to open your eyes to how bad things are run 3 dd's at the same time on the 3 pci disk, and then do the same with 3 disks on the motherboard controller, and things will look much much worse, I would predict that the disks on the pci controller will each do about 25mb/second, where as the 3 on the mb controller will likely slow down very little over the single disk speed. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2009-06-12 0:53 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-06-10 22:40 slow mdadm reshape, normal? Michael Ole Olsen
2009-06-10 23:04 ` John Robinson
-- strict thread matches above, loose matches on Subject: below --
2009-05-27 15:28 FW: Detecting errors on the RAID disks Simon Jackson
2009-05-27 19:35 ` Richard Scobie
2009-05-28 8:19 ` Simon Jackson
[not found] ` <20090610224055.GW30825@xxxxxxxxx>
2009-06-10 23:36 ` Re: slow mdadm reshape, normal? Michael Ole Olsen
2009-06-11 8:19 ` Robin Hill
2009-06-11 16:07 ` Michael Ole Olsen
2009-06-11 19:45 ` Michael Ole Olsen
2009-06-11 20:50 ` John Robinson
2009-06-11 21:11 ` slow mdadm reshape, normal? lspci/iostat info Michael Ole Olsen
2009-06-12 0:53 ` slow mdadm reshape, normal? Roger Heflin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.