Setting up md-raid5: observations, errors, questions

linux-ide.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* Setting up md-raid5: observations, errors, questions
@ 2008-03-02 12:23 Christian Pernegger
  2008-03-02 12:41 ` Justin Piszcz
  2008-03-02 15:20 ` Michael Tokarev
  0 siblings, 2 replies; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 12:23 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

Hi all!

I'm not doing this for the first time but there were a few interesting
/ worrying points during the setup process and I'd rather clear those
up now.

Hardware:
Tyan Thunder K8W (S2885)
Dual Opteron 254, 2GB (2x2x512MB) RAM, 2x Promise SATA II TX4, Adaptec 29160
4x WD RE2-GP 1TB on the Promise (for raid5)
1x Maxtor Atlas 15K II on the Adaptec (system disk)

OS:
Debian testing-amd64
linux-2.6.22-3
mke2fs 1.40.6
mdadm

I did a badblocks -v -s -t random -w on all future RAID disks in
parallel to test / burn-in.
Result: no bad blocks, decent speed (vmstat: 148MB/s total read,
220MB/s total write), BUT the following error (once):

ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2
ata2.00: (port_status 0x20080000)
ata2.00: cmd 25/00:80:80:a1:55/00:00:3a:00:00/e0 tag 0 cdb 0x0 data 65536 in
         res 50/00:00:ff:a1:55/00:00:3a:00:00/e0 Emask 0x2 (HSM violation)
ata2: soft resetting port
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: configured for UDMA/133
ata2: EH complete
sd 2:0:0:0: [sdc] 1953525168 512-byte hardware sectors (1000205 MB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA

All four disks were beyond the first (random write) phase, but that's
all I can say as I had it running overnight. The "HSM violation" error
is all over Google but I couldn't find anything conlusive (= that I
could understand).

Ignoring the error I went on to create the array:

[exact command unavailable, see below. RAID5, 4 disks, 1024K chunk
size, internal bitmap, V1 superblock]

Went fine, only mdadm segfaulted:

mdadm[3295]: segfault at 0000000000000000 rip 0000000000412d2c rsp
00007fff9f31b5d0 error 4

This did only show up in dmesg so I'm not sure exactly when. Either
right after the create or after a first attempt at a create where I
had used -N instead of --name, which it didn't like (error message).
Recreated the array, just to be sure (same command as above).

Tried creating a filesystem:

mke2fs -E stride=256 -j -L tb-storage -m1 -T largfile4 /dev/md0

That was glacially slow, "writing inode tables" went up about 3-4/sec
(22357 total). Since I had forgotten the crypto layer anyway I
CTRL-Ced that attempt and added it:

[exact command unavailable, see below. Used 2048 (512byte sectors) for
LUKS payload alignment, which should land on it chunk boundaries]

OK. Back to the fs again, same command, different device. Still
glacially slow (and still running), only now the whole box is at a
standstill, too. cat /proc/cpuinfo takes about 3 minutes (!) to
complete, I'm still waiting for top to launch (15min and counting).
I'll leave mke2fs running for now ...

So, is all this normal? What did I do wrong, what can I do better?

Cheers,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 12:23 Setting up md-raid5: observations, errors, questions Christian Pernegger
@ 2008-03-02 12:41 ` Justin Piszcz
  2008-03-02 12:56   ` Christian Pernegger
  2008-03-02 15:20 ` Michael Tokarev
  1 sibling, 1 reply; 19+ messages in thread
From: Justin Piszcz @ 2008-03-02 12:41 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide



On Sun, 2 Mar 2008, Christian Pernegger wrote:

> Hi all!
>
> I'm not doing this for the first time but there were a few interesting
> / worrying points during the setup process and I'd rather clear those
> up now.
>
> Hardware:
> Tyan Thunder K8W (S2885)
> Dual Opteron 254, 2GB (2x2x512MB) RAM, 2x Promise SATA II TX4, Adaptec 29160
> 4x WD RE2-GP 1TB on the Promise (for raid5)
> 1x Maxtor Atlas 15K II on the Adaptec (system disk)
>
> OS:
> Debian testing-amd64
> linux-2.6.22-3
> mke2fs 1.40.6
> mdadm
>
> I did a badblocks -v -s -t random -w on all future RAID disks in
> parallel to test / burn-in.
> Result: no bad blocks, decent speed (vmstat: 148MB/s total read,
> 220MB/s total write), BUT the following error (once):
>
> ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2
> ata2.00: (port_status 0x20080000)
> ata2.00: cmd 25/00:80:80:a1:55/00:00:3a:00:00/e0 tag 0 cdb 0x0 data 65536 in
>         res 50/00:00:ff:a1:55/00:00:3a:00:00/e0 Emask 0x2 (HSM violation)
> ata2: soft resetting port
> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> ata2.00: configured for UDMA/133
> ata2: EH complete
> sd 2:0:0:0: [sdc] 1953525168 512-byte hardware sectors (1000205 MB)
> sd 2:0:0:0: [sdc] Write Protect is off
> sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't
> support DPO or FUA

Disable NCQ and your problem will go away.

echo 1 > /sys/block/$i/device/queue_depth

Justin.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 12:41 ` Justin Piszcz
@ 2008-03-02 12:56   ` Christian Pernegger
  2008-03-02 13:03     ` Justin Piszcz
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 12:56 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  Disable NCQ and your problem will go away.

Thank you. Just out of interest - which problem and why?

>  echo 1 > /sys/block/$i/device/queue_depth

I get access denied even as root. FWIW the value is at 1 for the 4
disks in the raid anyway. The SCSI disk has value 8, which is probably
irrelevant.

For completeness' sake:

mdadm version: mdadm - v2.6.4 - 19th October 2007
raid setup command: mdadm --create /dev/md0 --verbose --metadata=1.0
--homehost=jesus -n4 -c1024 -l5 --bitmap=internal --name tb-storage
-ayes /dev/sd[bcde]
cryptsetup command: cryptsetup luksFormat /dev/md0 --align-payload=2048

Any other suggestions on any of the issues?

Thanks,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 12:56   ` Christian Pernegger
@ 2008-03-02 13:03     ` Justin Piszcz
  2008-03-02 13:24       ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Justin Piszcz @ 2008-03-02 13:03 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide



On Sun, 2 Mar 2008, Christian Pernegger wrote:

>>  Disable NCQ and your problem will go away.
>
> Thank you. Just out of interest - which problem and why?
>
>>  echo 1 > /sys/block/$i/device/queue_depth
>
> I get access denied even as root. FWIW the value is at 1 for the 4
> disks in the raid anyway. The SCSI disk has value 8, which is probably
> irrelevant.
>
> For completeness' sake:
>
> mdadm version: mdadm - v2.6.4 - 19th October 2007
> raid setup command: mdadm --create /dev/md0 --verbose --metadata=1.0
> --homehost=jesus -n4 -c1024 -l5 --bitmap=internal --name tb-storage
> -ayes /dev/sd[bcde]
> cryptsetup command: cryptsetup luksFormat /dev/md0 --align-payload=2048
>
> Any other suggestions on any of the issues?
>
> Thanks,

You could try to change cables and such but you've already cc'd linux-ide, 
AFAIK it can/could be a chipset/related issue and the guys who work on NCQ 
etc are working on the problem is the last I heard..

You should try and narrow down the problem, is it always the same drive 
that has that problem every time?  Does it occur if you do a check on the 
RAID 5 array or only when building?  etc..

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 13:03     ` Justin Piszcz
@ 2008-03-02 13:24       ` Christian Pernegger
  2008-03-03 17:59         ` Bill Davidsen
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 13:24 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  You could try to change cables and such but you've already cc'd linux-ide,
>  AFAIK it can/could be a chipset/related issue and the guys who work on NCQ
>  etc are working on the problem is the last I heard..
>
>  You should try and narrow down the problem, is it always the same drive
>  that has that problem every time?  Does it occur if you do a check on the
>  RAID 5 array or only when building?  etc..

Assuming you're talking about the HSM violation error ... I got that
exactly once (yet) in the situation described (sometime after the
writing phase of badblocks). Nothing since and certainly not the spew
others are reporting.

My primary concern now is the second problem - seems like any access
to the raid array makes the machine unusable. I can type/edit a
command at the bash prompt normally but as soon as I hit enter it just
hangs there. If the command is trivial, say cat, I might get a result
a few minutes later.

bonnie++ results are in:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
1024k            4G           16282   5 13359   3           57373   7 149.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

Something is definitely not right ... :-(

As in, an old array of 300GB IDE Maxtors has 4x the seq writes, bus
capped (105MB/s, plain PCI) reads and 3x the IOs. And it doesn't block
the machine. Granted, there's the crypto but on my other (non-raid)
boxes the performance impact just isn't there.

Any help appreciated, as it is the box has expensive-paperweight status.

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 13:24       ` Christian Pernegger
@ 2008-03-03 17:59         ` Bill Davidsen
  2008-03-03 20:19           ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Bill Davidsen @ 2008-03-03 17:59 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide

Christian Pernegger wrote:
>>  You could try to change cables and such but you've already cc'd linux-ide,
>>  AFAIK it can/could be a chipset/related issue and the guys who work on NCQ
>>  etc are working on the problem is the last I heard..
>>
>>  You should try and narrow down the problem, is it always the same drive
>>  that has that problem every time?  Does it occur if you do a check on the
>>  RAID 5 array or only when building?  etc..
>>     
>
> Assuming you're talking about the HSM violation error ... I got that
> exactly once (yet) in the situation described (sometime after the
> writing phase of badblocks). Nothing since and certainly not the spew
> others are reporting.
>
>   
Are you by any chance running quota?

> My primary concern now is the second problem - seems like any access
> to the raid array makes the machine unusable. I can type/edit a
> command at the bash prompt normally but as soon as I hit enter it just
> hangs there. If the command is trivial, say cat, I might get a result
> a few minutes later.
>
> bonnie++ results are in:
>
> Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> 1024k            4G           16282   5 13359   3           57373   7 149.0   0
>                     ------Sequential Create------ --------Random Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
>
> Something is definitely not right ... :-(
>
> As in, an old array of 300GB IDE Maxtors has 4x the seq writes, bus
> capped (105MB/s, plain PCI) reads and 3x the IOs. And it doesn't block
> the machine. Granted, there's the crypto but on my other (non-raid)
> boxes the performance impact just isn't there.
>
> Any help appreciated, as it is the box has expensive-paperweight status.
>
> C.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   


-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-03 17:59         ` Bill Davidsen
@ 2008-03-03 20:19           ` Christian Pernegger
  0 siblings, 0 replies; 19+ messages in thread
From: Christian Pernegger @ 2008-03-03 20:19 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  > Assuming you're talking about the HSM violation error ... I got that
>  > exactly once (yet) in the situation described (sometime after the
>  > writing phase of badblocks). Nothing since and certainly not the spew
>  > others are reporting.
>  >
>  >
>  Are you by any chance running quota?

No, sorry ... next guess :-)

Ah, why?

Thanks,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 12:23 Setting up md-raid5: observations, errors, questions Christian Pernegger
  2008-03-02 12:41 ` Justin Piszcz
@ 2008-03-02 15:20 ` Michael Tokarev
  2008-03-02 16:32   ` Christian Pernegger
  1 sibling, 1 reply; 19+ messages in thread
From: Michael Tokarev @ 2008-03-02 15:20 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide

Christian Pernegger wrote:
> Hi all!

Hello.

> Hardware:
> Tyan Thunder K8W (S2885)
> Dual Opteron 254, 2GB (2x2x512MB) RAM, 2x Promise SATA II TX4, Adaptec 29160
> 4x WD RE2-GP 1TB on the Promise (for raid5)

[]
> [exact command unavailable, see below. RAID5, 4 disks, 1024K chunk
> size, internal bitmap, V1 superblock]

ok.

> Tried creating a filesystem:
> 
> mke2fs -E stride=256 -j -L tb-storage -m1 -T largfile4 /dev/md0
> 
> That was glacially slow, "writing inode tables" went up about 3-4/sec
> (22357 total). Since I had forgotten the crypto layer anyway I
> CTRL-Ced that attempt and added it:
> 
> [exact command unavailable, see below. Used 2048 (512byte sectors) for
> LUKS payload alignment, which should land on it chunk boundaries]
> 
> OK. Back to the fs again, same command, different device. Still
> glacially slow (and still running), only now the whole box is at a
> standstill, too. cat /proc/cpuinfo takes about 3 minutes (!) to
> complete, I'm still waiting for top to launch (15min and counting).
> I'll leave mke2fs running for now ...

What's the state of your array at this point - is it resyncing?

> So, is all this normal? What did I do wrong, what can I do better?

In order to debug, try the following:

  o how about making filesystem(s) on individual disks first, to see
    how that will work out?  Maybe on each of them in parallel? :)

  o try --assume-clean when creating the array - this will omit resync
    and the array will be "clean" from the beginning (which is not bad
    when you will run mkfs on it anyway).  And try creating the filesystem
    on a clean (non-resyncing) array.

  o don't use crypto layer yet, - it seems does not hurt, just additional
    complexity

  o look at your interrupts (/proc/interrupts, or just run vmstat with
    some interval while the system working hard) - I bet your promises
    are taking all system time in irq context....

/mjt

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 15:20 ` Michael Tokarev
@ 2008-03-02 16:32   ` Christian Pernegger
  2008-03-02 18:33     ` Michael Tokarev
  2008-03-02 18:53     ` Christian Pernegger
  0 siblings, 2 replies; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 16:32 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  > OK. Back to the fs again, same command, different device. Still
>  > glacially slow (and still running), only now the whole box is at a
>  > standstill, too. cat /proc/cpuinfo takes about 3 minutes (!) to
>  > complete, I'm still waiting for top to launch (15min and counting).
>  > I'll leave mke2fs running for now ...
>
>  What's the state of your array at this point - is it resyncing?

Yes. Didn't think it would matter (much). Never did before.

>   o how about making filesystem(s) on individual disks first, to see
>     how that will work out?  Maybe on each of them in parallel? :)

Running. System is perfectly responsive during 4x mke2fs -j -q on raw devices.
Done. Upper bound for duration is 8 minutes (probaby much lower,
forgot to let it beep on completion), which is much better than the 2
hours with the syncing RAID.


chris@jesus:~$ cat /proc/interrupts
           CPU0       CPU1
  0:       4939    1920632   IO-APIC-edge      timer
  1:        113        133   IO-APIC-edge      i8042
  6:          0          3   IO-APIC-edge      floppy
  7:          0          0   IO-APIC-edge      parport0
  8:          0          1   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          0          4   IO-APIC-edge      i8042
 14:          0        182   IO-APIC-edge      ide0
 19:         87         34   IO-APIC-fasteoi   ohci_hcd:usb1,
ohci_hcd:usb2, firewire_ohci
 24:      10142         57   IO-APIC-fasteoi   eth0
 26:    1041479        267   IO-APIC-fasteoi   sata_promise
 27:          0          0   IO-APIC-fasteoi   sata_promise
 28:       7141       2789   IO-APIC-fasteoi   aic7xxx
NMI:          0          0
LOC:    1925715    1925691
ERR:          0


chris@jesus:~$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  4      0  12716 1770264   9972    0    0  1034  4902  196  883  0  9 78 13
 0  4      0  11404 1771260   9972    0    0     0 150156  544  874  0 26 28 46
 0  4      0  11836 1771016  10024    0    0     0 147500  544  682  0 26 26 48
 0  4      0  12572 1770036  10108    0    0     0 131022  515  506  0 25 13 62
 0  4      0  12864 1769688  10000    0    0     0 146822  539  809  0 26 23 51
 0  4      0  12132 1769988   9956    0    0     0 145942  536  900  0 26 15 59
 0  4      0  12520 1770324   9976    0    0     0 144638  536  820  0 26 32 42


top - 17:08:55 up  2:12,  2 users,  load average: 4.37, 3.13, 1.49
Tasks:  78 total,   1 running,  77 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3%us,  8.1%sy,  0.0%ni, 41.6%id, 46.0%wa,  0.7%hi,  2.3%si,  0.0%st
Mem:   2063940k total,  2051316k used,    12624k free,  1746616k buffers


I hope you can interpret that :)

>   o try --assume-clean when creating the array

mke2fs (same command as in first post) now running on fresh
--assumed-clean array w/o crypto. System is only marginally less
responsive than under idle load, if at all.
But inode table writing speed is only about 8-10/second. For the
single disk case I couldn't read the numbers fast enough.

chris@jesus:~$ cat /proc/interrupts
           CPU0       CPU1
  0:       7485    2227196   IO-APIC-edge      timer
  1:        113        133   IO-APIC-edge      i8042
  6:          0          3   IO-APIC-edge      floppy
  7:          0          0   IO-APIC-edge      parport0
  8:          0          1   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          0          4   IO-APIC-edge      i8042
 14:          0        182   IO-APIC-edge      ide0
 19:        101         39   IO-APIC-fasteoi   ohci_hcd:usb1,
ohci_hcd:usb2, firewire_ohci
 24:      15656         57   IO-APIC-fasteoi   eth0
 26:    1211165        267   IO-APIC-fasteoi   sata_promise
 27:          0          0   IO-APIC-fasteoi   sata_promise
 28:       7892       2938   IO-APIC-fasteoi   aic7xxx
NMI:          0          0
LOC:    2234843    2234819
ERR:          0


chris@jesus:~$ vmstat 2
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1      0  12996 1811456  10860    0    0   912  4824  194  867  0  9 78 13
 0  0      0  11532 1812992  10832    0    0     0 12924  532 4992  0 11 61 28
 0  1      0  11092 1813376  10804    0    0     0 13316  535 5201  0  9 51 40
 0  0      0  12968 1811584  10832    0    0     0 12570  518 4890  0  9 58 32
 0  1      0  11724 1812736  10816    0    0     0 12818  508 5337  0 10 52 38
 0  0      0  12780 1811712  10804    0    0     0 13994  546 5055  0  9 52 40


top - 17:26:37 up  2:29,  2 users,  load average: 2.89, 2.12, 1.42
Tasks:  75 total,   2 running,  73 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us, 10.1%sy,  0.0%ni, 55.6%id, 33.7%wa,  0.2%hi,  0.3%si,  0.0%st
Mem:   2063940k total,  2052148k used,    11792k free,  1812288k buffers

>From vmstat I gather that total write throughput is an order of
magnitude slower than on the 4 raw disks in parallel. Naturally the
mke2fs on the raid isn't parallelized but it should still be
sequential enough to get the max for a single disk (~60-40MB/s),
right?

Thanks for helping.

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 16:32   ` Christian Pernegger
@ 2008-03-02 18:33     ` Michael Tokarev
  2008-03-02 21:19       ` Christian Pernegger
  2008-03-02 18:53     ` Christian Pernegger
  1 sibling, 1 reply; 19+ messages in thread
From: Michael Tokarev @ 2008-03-02 18:33 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide

Christian Pernegger wrote:
>>  > OK. Back to the fs again, same command, different device. Still
>>  > glacially slow (and still running), only now the whole box is at a
>>  > standstill, too. cat /proc/cpuinfo takes about 3 minutes (!) to
>>  > complete, I'm still waiting for top to launch (15min and counting).
>>  > I'll leave mke2fs running for now ...
>>
>>  What's the state of your array at this point - is it resyncing?
> 
> Yes. Didn't think it would matter (much). Never did before.

It does.  If everything works ok, it should not, but it's not your
case ;)

>>   o how about making filesystem(s) on individual disks first, to see
>>     how that will work out?  Maybe on each of them in parallel? :)
> 
> Running. System is perfectly responsive during 4x mke2fs -j -q on raw devices.
> Done. Upper bound for duration is 8 minutes (probaby much lower,
> forgot to let it beep on completion), which is much better than the 2
> hours with the syncing RAID.

Aha.  Excellent.

>  26:    1041479        267   IO-APIC-fasteoi   sata_promise
>  27:          0          0   IO-APIC-fasteoi   sata_promise

> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  0  4      0  12864 1769688  10000    0    0     0 146822  539  809  0 26 23 51

Ok.  146Mb/sec.

> Cpu(s):  1.3%us,  8.1%sy,  0.0%ni, 41.6%id, 46.0%wa,  0.7%hi,  2.3%si,  0.0%st

46.0% waiting

> I hope you can interpret that :)

Some ;)

>>   o try --assume-clean when creating the array
> 
> mke2fs (same command as in first post) now running on fresh
> --assumed-clean array w/o crypto. System is only marginally less
> responsive than under idle load, if at all.

So the responsibility problem is solved here, right?  I mean, if
there's no resync going on (the case with --assume-clean), the rest
of the system works as expected, right?

> But inode table writing speed is only about 8-10/second. For the
> single disk case I couldn't read the numbers fast enough.

Note that mkfs now has to do 3x more work, too - since the device
is 3x (for 4-drive raid5) larger.

> chris@jesus:~$ cat /proc/interrupts
>  26:    1211165        267   IO-APIC-fasteoi   sata_promise
>  27:          0          0   IO-APIC-fasteoi   sata_promise
> 
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  0  1      0  11092 1813376  10804    0    0     0 13316  535 5201  0  9 51 40

That's 10 times slower than in the case of 4 individual disks.

> Cpu(s):  0.0%us, 10.1%sy,  0.0%ni, 55.6%id, 33.7%wa,  0.2%hi,  0.3%si,  0.0%st

and only 33.7% waiting, which is probably due to the lack of
parallelism.

> From vmstat I gather that total write throughput is an order of
> magnitude slower than on the 4 raw disks in parallel. Naturally the
> mke2fs on the raid isn't parallelized but it should still be
> sequential enough to get the max for a single disk (~60-40MB/s),
> right?

Well, not really.  Mkfs is doing many small writes all over the
place, so each is seek+write.  And it's syncronous - no next write
gets submitted till the current one completes.

Ok.  For now I don't see a problem (over than that there IS a problem
somewhere - obviously).  Interrupts are ok.  System time (10.1%) in
second case doesn't look right, but it was 8.1% before...

Only 2 guesses left.  And I really mean "guesses", because I can't
say definitely what's going on anyway.

First, try to disable bitmaps on the raid array, and see if it makes
any difference.  For some reason I think it will... ;)

And second, the whole thing looks pretty much like a more general
problem discussed here and elsewhere last few days.  I mean handling
of parallel reads and writes - when single write may stall reads
for quite some time and vise versa.  I see it every day on disks
without NCQ/TCQ - system is mostly single-tasking, sorta like
ol'good MS-DOG :)  Good TCQ-enabled drives survives very high load
while the system is still more-or-less responsible (and I forgot when
I last saw "bad" TCQ-enabled drive - even 10 y/o 4Gb seagate has
excellent TCQ support ;).  And all modern SATA stuff works pretty
much like old IDE drives, which were designed "for personal use",
or "single-task only" -- even ones that CLAMS to support NCQ in
reality does not....  But that's a long story, and your disks
and/or controllers (or the combination) don't even support NCQ...

/mjt

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 18:33     ` Michael Tokarev
@ 2008-03-02 21:19       ` Christian Pernegger
  2008-03-02 21:56         ` Michael Tokarev
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 21:19 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  So the responsibility problem is solved here, right?

It is? I'm not sure yet.

>  I mean, if there's no resync going on (the case with --assume-clean), the rest of the system works as expected, right?

Yes, but the array itself is still dog slow and resync shouldn't have
that much impact on performance. What's more

mdadm --create /dev/md0 -l5 -n4 /dev/sd[bcde] -e 1.0

leaves room for tuning but is basically fine, whereas the original case

mdadm --create /dev/md0 --verbose --metadata=1.0 --homehost=jesus -n4
-c1024 -l5 --bitmap=internal --name tb-storage -ayes /dev/sd[bcde]

is all but unusable, which leaves two prime suspects

- the bitmap
- the chunk size

Could it be that some cache or other somewhere in the I/O stack
(probaby the controller itself) is too small for the 1MB chunks and
the disks are forced to work serially? The Promise has no RAM of
course but maybe it does have small send / receive buffers.
On the host side the I/O schedulers are set to cfq which is said to
play well with md-raid but I can experiment with that as well.

>  Note that mkfs now has to do 3x more work, too - since the device is 3x (for 4-drive raid5) larger.

Yes, but that just means there's more inode tables to write. It takes
longer, but the speed shouldn't change much.

>  Ok.  For now I don't see a problem (over than that there IS a problem
>  somewhere - obviously).  Interrupts are ok.  System time (10.1%) in
>  second case doesn't look right, but it was 8.1% before...

Too high? Too low?

>  Only 2 guesses left.

I'm fine with guesses, thank you :-) Of course a deus-ex-machina
solution (deus == Neil) is nice, too :)

>  First, try to disable bitmaps on the raid array

Maybe I did that by accident for the various vmstat data for different
RAID levels I posted previously. At least I forgot to explicitely
specify a bitmap for those tests (see above).

It's my understanding that the bitmap is a raid chunk level journal to
speed up recovery, correct? Doing that reduces the window during which
a second disk can die with catastrophic consequences -> bitmaps are a
good thing, especially on an array where a full rebuild takes hours.
Seeing as the primary purpose of the raid5 is fault tolerance I could
live with a performance penalty but why is it *that* slow?

If I put the bitmap on an external drive it will be a lot faster - but
what happens, when the bitmap "goes away" (because that disk fails,
isn't accessible, etc)?
Is it goodbye array or is the worst case a full resync? How well is
the external bitmap supported?
(That same consideration kept me from using external journals for ext3.)

>  And second, the whole thing looks pretty much like a more general
>  problem discussed here and elsewhere last few days.  I mean handling
>  of parallel reads and writes - when single write may stall reads
>  for quite some time and vise versa.

Any thread names to recommend?

>  I see it every day on disks without NCQ/TCQ [...] your disks
>  and/or controllers (or the combination) don't even support NCQ

The old IDE disks on mixed noname controllers array does well enough
and NCQ / ncq doesn't even show up in dmesg. Definitely something to
consider but probably not the root cause.

Back to testing ...

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 21:19       ` Christian Pernegger
@ 2008-03-02 21:56         ` Michael Tokarev
  2008-03-03  0:17           ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Michael Tokarev @ 2008-03-02 21:56 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide

Christian Pernegger wrote:
[]
>>  First, try to disable bitmaps on the raid array

It has been pointed out recently here on linux-raid that internal bitmap
doesn't work well:

Message-ID: <47C44DDB.3050201@free.fr>
Date:	Tue, 26 Feb 2008 18:35:23 +0100
From:	Hubert Verstraete <hubskml@free.fr>
To:	Neil Brown <neilb@suse.de>, linux-raid@vger.kernel.org
Subject: internal bitmap size

Hi Neil,

Neil Brown wrote:
 > For now, you will have to live with a smallish bitmap, which probably
 > isn't a real problem.  With 19078 bits, you will still get a
 > several-thousand-fold increase it resync speed after a crash
 > (i.e. hours become seconds) and to some extent, fewer bits are better
 > and you have to update them less.
 >
 > I've haven't made any measurements to see what size bitmap is
 > ideal... maybe someone should :-)

I've made some tries with a 4 250GB disks RAID-5 array and the write
speed is really ugly with the default internal bitmap size.
Setting a bigger bitmap chunk size (16 MB for example) creates a small
bitmap. The write speed is then almost the same as when there is no
bitmap, which is great. And as you said, the resync is a matter of
seconds (or minutes) instead of hours (without bitmap).
With such a setting, I've got both a nice write speed and a nice resync
speed. That's where I would look at to find MY ideal bitmap size.
....

> Maybe I did that by accident for the various vmstat data for different
> RAID levels I posted previously. At least I forgot to explicitely
> specify a bitmap for those tests (see above).
> 
> It's my understanding that the bitmap is a raid chunk level journal to
> speed up recovery, correct? Doing that reduces the window during which
> a second disk can die with catastrophic consequences -> bitmaps are a
> good thing, especially on an array where a full rebuild takes hours.
> Seeing as the primary purpose of the raid5 is fault tolerance I could
> live with a performance penalty but why is it *that* slow?

Umm..  You mixed it all ;)

Bitmap is a place (stored somewhere... ;) where each equally-sized
block of the array has a single bit of information - namely, if that
block has been written recently (which means it was dirty) or not.
So for each block (which is in no way related to chunk size etc!)
we've an on/off switch, telling us if the said block has to be
re-syncronized if we need to perform re-syncronisation of data -
for example, in case of power loss -- only those blocks marked
"dirty" in the bitmap needs to be recalculated and rewritten,
not the whole array.

This has nothing to do with window between first and second disk
failure.  Once first disk fails, bitmap is of no use anymore,
because you will need a replacement disk, which has to be
resyncronized in whole, because it's shiny new.  Bitmap only
helps for unclean shutdown, and only if there was no recent write
activity (which hasn't been "comitted" by md layer and the array
hasn't been re-marked as clean - it happens every 0.21 sec by
default - see /sys/block/mdN/md/safe_mode_delay).

> If I put the bitmap on an external drive it will be a lot faster - but
> what happens, when the bitmap "goes away" (because that disk fails,
> isn't accessible, etc)?
> Is it goodbye array or is the worst case a full resync? How well is
> the external bitmap supported?
> (That same consideration kept me from using external journals for ext3.)

If the bitmap is unaccessible, it's handled as there was no bitmap
at all - ie, if the array was dirty, it will be resynced as a whole;
if it was clean, nothing will be done.  Bitmap gives a set of blocks
to OMIT from resyncronisation, and if that information is unavailable...

Yes, external bitmaps are supported and working.  It doesn't mean
they're faster however - I tried placing a bitmap into a tmpfs (just
for testing) - and discovered about 95% drop in speed compared to the
case with internal bitmap (ie, only 5% speed when bitmap is on tmpfs -
bitmap size was the same).  It was long (more than a year) ago so things
may have changed already.

I highly doubt chunk size makes any difference.  Bitmap is the primary
suspect here.

/mjt

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 21:56         ` Michael Tokarev
@ 2008-03-03  0:17           ` Christian Pernegger
  2008-03-03  2:58             ` Michael Tokarev
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-03  0:17 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  I highly doubt chunk size makes any difference.  Bitmap is the primary
>  suspect here.

Some tests:

raid5, chunk size goes from 16k to 1m. arrays created with --assume-clean

dd-tests
========

read / write 4GB in 4-chunk blocks directly on the md device.

dd-read
-------
unaffected by bitmap as expected
gets MUCH better with inc. chunk size			 71 -> 219 MS/s
reads go up near theoretical bus maximum (266MB/s)
the maximum total reached via parallel single-disk reads is 220 MB/s

Conclusion: reads are fine

dd-write
--------
with bitmap: gets SLOWLY worse with inc. chunk size	 30 ->  27 MB/s
without bitmap: gets MUCH worse with inc chunk size	100 ->  59 MB/s

Conclusion: needs explanation / tuning

even omitting the bitmap the writes just touch 100 MB/s, more like 80
on any chunk size with nice reads.
why would it get worse? Anything tunable here?
the maximum total reached via parallel single-disk writes is 150 MB/s

mke2fs-tests
============

create ext3 fs with correct stride, get a 10-second vmstat average 10
seconds in and abort the mke2fs

with bitmap: goes down SLOWLY from 64k chunks		 17 ->  13 MB/s
without bitmap: gets MUCH worse with inc. chunk size	 80 ->  34 MB/s

Conclusion: needs explanation / tuning

the maximum total reached via parallel single-disk mke2fs is 150 MB/s.

Comments welcome.

Next step: smaller bitmap
When the performance seems normal I'll revisit the responsiveness-issue.

>  Umm..  You mixed it all ;)
>  Bitmap is a place (stored somewhere... ;) where each equally-sized
>  block of the array has a single bit of information - namely, if that
>  block has been written recently (which means it was dirty) or not.
>  So for each block (which is in no way related to chunk size etc!)

Aren't these blocks-represented-by-a-bit-in-the-bitmap called chunks,
too? Sorry for the confusion.

>  This has nothing to do with window between first and second disk
>  failure.  Once first disk fails, bitmap is of no use anymore,
>  because you will need a replacement disk, which has to be
>  resyncronized in whole,

Yes, that makes sense. Still sounds useful, since a lot of my
"failures" have been of the intermittent (SATA cables / connectors,
port resets, slow bad-sector remap) variety.

>  If the bitmap is unaccessible, it's handled as there was no bitmap
>  at all - ie, if the array was dirty, it will be resynced as a whole;
>  if it was clean, nothing will be done.

Ok, good to hear. In theory that's the sane mode of operation, in
practice it might just have been that the array refuses to assemble
without its bitmap.

>  Yes, external bitmaps are supported and working.  It doesn't mean
>  they're faster however - I tried placing a bitmap into a tmpfs (just
>  for testing) - and discovered about 95% drop in speed

Interesting ... what are external bitmaps good for, then?

Thank you, I appreciate your patience.

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-03  0:17           ` Christian Pernegger
@ 2008-03-03  2:58             ` Michael Tokarev
  2008-03-03  8:38               ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Michael Tokarev @ 2008-03-03  2:58 UTC (permalink / raw)
  To: Christian Pernegger; +Cc: linux-raid, linux-ide

Christian Pernegger wrote:
>>  I highly doubt chunk size makes any difference.  Bitmap is the primary
>>  suspect here.

I meant something else.  Sure thing chunk size will have quite
significant impact on write performance, see below.  What I meant
is that the PROBLEM you're facing is not due to chunk size but
due to bitmap.

> Some tests:
> 
> raid5, chunk size goes from 16k to 1m. arrays created with --assume-clean
> 
> dd-tests
> ========
[]
> dd-write
> --------
> with bitmap: gets SLOWLY worse with inc. chunk size	 30 ->  27 MB/s
> without bitmap: gets MUCH worse with inc chunk size	100 ->  59 MB/s

In other words, bitmap makes HUGE impact - when it's on, everything is
so damn slow that other factors aren't even noticeable.  When bitmap
is off, write speed drops when increasing chunk size.

> Conclusion: needs explanation / tuning

You know how raid5 process writes, right?  The read-modify-write or
similar technique, that involves READING as well as writing, reading
from other disks in order to calculate/update parity.  Unless you
write complete stripe (all chunks).

So the bigger your chunk size is, the less chances you have to perform
full-stripe write, just because "large, properly aligned" writes are
much less frequent than "smaller, unaligned" ones - at least for a
typical filesystem usage (special usage patterns exists for sure).

Here, both linux write-back cache (writes don't go directly to disk
but to kernel memory first, and kernel does some reordering/batching
here) and md stripe-cache makes huge difference, for obvious reasons.

> even omitting the bitmap the writes just touch 100 MB/s, more like 80
> on any chunk size with nice reads.
> why would it get worse? Anything tunable here?

Yes.  See your read-test with small chunk size.  Here, you've got
better "initial" results.  Why the reading with small chunk size
is so slow?  Because of the small request size to the underlying
device, that's why - each drive effectively is reading by 16Kb at
once, spending much time rotating and communicating with the controller...

But unlike for reads, increasing chunk size does not help writes -
because of the above reasons (writing "half" stripes more often and
hence requiring read-modify-write cycle more often etc).

You can get best results when writing WITHOUT a filesystem
(directly to /dev/mdFOO) with dd and blocksize equal to the
total strip size (chunk size * number of data disks, or
16k * 3 ..... 1M * 3, since in your raid 3 disks are data
in each strip), and trying direct write at the same time.
Yes it will be a bit worse than read speed, but it should be
pretty descent still.

I said "without a filesystem" because alignment is very important
too, -- to the same level as full-strip vs "half"-strip writes,
and with a filesystem in place, you can't be sure anymore how
your files are aligned on disk (xfs tries to align files correctly,
at least under some conditions).

> the maximum total reached via parallel single-disk writes is 150 MB/s
> 
> 
> mke2fs-tests
> ============
> 
> create ext3 fs with correct stride, get a 10-second vmstat average 10
> seconds in and abort the mke2fs
> 
> with bitmap: goes down SLOWLY from 64k chunks		 17 ->  13 MB/s
> without bitmap: gets MUCH worse with inc. chunk size	 80 ->  34 MB/s
> 
> Conclusion: needs explanation / tuning

The same thing.  Bitmap makes HUGE impact, and w/o bitmap, write speed
drops when increasing chunk size.

> Comments welcome.

I see one problem (the bitmap thing, finally discovered and confirmed),
and a need to tune your system beforehand -- it's the chunk size.  And
here, you're pretty much a wizard - it's your guess.  In any case,
unless your usage pattern will be special and optimized for such a
setup, don't try to choose large chunk size for raid456.  Choosing large
chunk size for raid0 and raid10 makes sense, but with raid456 it has
immediately bad sides.

> Next step: smaller bitmap
> When the performance seems normal I'll revisit the responsiveness-issue.
> 
>>  Umm..  You mixed it all ;)
>>  Bitmap is a place (stored somewhere... ;) where each equally-sized
>>  block of the array has a single bit of information - namely, if that
>>  block has been written recently (which means it was dirty) or not.
>>  So for each block (which is in no way related to chunk size etc!)
> 
> Aren't these blocks-represented-by-a-bit-in-the-bitmap called chunks,
> too? Sorry for the confusion.

Well yes, but I especially avoided using of *this* "chunk" in my sentence ;)
In any case, the chunk as in "usual-raid-chunk-size" and "chunk-represented-
by-a-bit-in-the-bitmap" are different chunks, the latter consists of one
or more the formers.  Oh well.

>>  This has nothing to do with window between first and second disk
>>  failure.  Once first disk fails, bitmap is of no use anymore,
>>  because you will need a replacement disk, which has to be
>>  resyncronized in whole,
> 
> Yes, that makes sense. Still sounds useful, since a lot of my
> "failures" have been of the intermittent (SATA cables / connectors,
> port resets, slow bad-sector remap) variety.

Yes, you're right.  Annoying stuff, and bitmaps definitely helps here.

>>  If the bitmap is unaccessible, it's handled as there was no bitmap
>>  at all - ie, if the array was dirty, it will be resynced as a whole;
>>  if it was clean, nothing will be done.
> 
> Ok, good to hear. In theory that's the sane mode of operation, in
> practice it might just have been that the array refuses to assemble
> without its bitmap.

I had the opposite here.  Due to various reasons, including operator
errors, bugs in mdadm and md kernel code, and probably phases of the
Moon, after creating a bitmap on a raid array and rebooting, I were
discovering that the bitmap isn't here anymore, -- it's gone.  It all
was down to the case when bitmap data (information about bitmap
presence and location) were not passed to the kernel correctly - either
because I forgot to specify it in a config file, or because mdadm didn't
pass that info in several cases or because with that superblock version
bitmaps were handled incorrectly....

>>  Yes, external bitmaps are supported and working.  It doesn't mean
>>  they're faster however - I tried placing a bitmap into a tmpfs (just
>>  for testing) - and discovered about 95% drop in speed
> 
> Interesting ... what are external bitmaps good for, then?

I had no time to investigate, and now I don't have a hardware to test
again.  In theory it should work, but I guess only a few people are
using them if at all - most are using internal bitmaps.

/mjt

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-03  2:58             ` Michael Tokarev
@ 2008-03-03  8:38               ` Christian Pernegger
  2008-03-04 16:54                 ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-03  8:38 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

>  You know how raid5 process writes, right?

Yes. What I'd forgotten, though, is that even with --assume-clean it's
still basically in degraded mode so testing writes at most serves as a
comparison between the different setups. This seems to be a common
late night error :)

>  So the bigger your chunk size is, the less chances you have to perform
>  full-stripe write, just because "large, properly aligned" writes are
>  much less frequent than "smaller, unaligned" ones

I had expected that the sequential dd-writes would be automatically
combined to write full stripes, yet they also go down massively with
increased chunk size.

>  You can get best results when writing WITHOUT a filesystem
>  (directly to /dev/mdFOO) with dd and blocksize equal to the
>  total strip size (chunk size * number of data disks, or
>  16k * 3 ..... 1M * 3, since in your raid 3 disks are data
>  in each strip)

I didn't use an fs for the dd tests, but 4*chunk size, which was
stupid. Still, I would have expected some combining to be going on
even at that level.

>  md stripe-cache makes huge difference, for obvious reasons.

Will experiment with that as well, thanks. Any other caches?

>  unless your usage pattern will be special and optimized for such a
>  setup, don't try to choose large chunk size for raid456.

I think I will be revising my test script to actually create a small
fs at the start of the device and run some tests in that. Note to
self: need a good load simulator ... ...

>  I had no time to investigate, and now I don't have a hardware to test
>  again.  In theory it should work, but I guess only a few people are
>  using [external bitmaps] if at all - most are using internal bitmaps.

I'd rather not use a feature where the probability that a bug will
bite me first is that high. :)

FWIW, using smaller bitmaps REALLY helps. Tests forthcoming.

Thanks,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-03  8:38               ` Christian Pernegger
@ 2008-03-04 16:54                 ` Christian Pernegger
  2008-03-05  6:38                   ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-04 16:54 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

>  FWIW, using smaller bitmaps REALLY helps. Tests forthcoming.

I accidentaly thrashed the original script output for the first test
so all I can offer is a spreadsheet (or not, 1.test should have most
of the data):

1. create array with given chunk size and bitmap chunk size and --assume-clean
2. dd write and read 4GB (2x RAM) directly on the device (no -oflag
direct, though)
3. create small ~13 GB fs at the start of the array
4. mount without options and run bonnie++
5. lather, rinse, repeat

Second test is the same as the first, only
1. create a smaller array with -z (4GB / disk so 12GB usable), let it
sync and set stripe_cache_size to 8192
2. ...
3. ...
4. mount with noatime, nodiratime
5. ...

Better. Much better. Now safely out of the realm of error and into
that of tuning.
Still a bottleneck somewhere ... If 50% in bonnie++ means 100% of one
CPU that could be it. Any comments on the CPU results (text version
only)?

Results are single-shot (not averaged) so the data is relatively low quality.

I didn't notice any responsiveness issues during the tests, but then
again I left the machine pretty much alone. Will tune first and tackle
that later. FWIW background resync alone isn't the culprit - that
doesn't even hurt benchmarks too badly. Maybe background sync + large
bitmap?

The HSM violation hasn't yet cropped up again. What does it mean exactly?

Also, aside from the fact that NCQ should apparently be turned off for
md raid anyway - why doesn't it work? The Promise SATA2 TX4 allegedly
does it, as do the disks.

Thanks,

C.

[-- Attachment #2: raid5-bench.ods --]
[-- Type: application/vnd.oasis.opendocument.spreadsheet, Size: 15346 bytes --]

[-- Attachment #3: 1.test --]
[-- Type: application/octet-stream, Size: 65731 bytes --]



Chunk size:  1024
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 72,7572 Sekunden, 59,0 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 19,702 Sekunden, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-512   4G           45594  32 37159  23           166326  45 356.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22935  95 +++++ +++ 26607 100 +++++ +++ +++++ +++ 28495  99
raid5-1024-512,4G,,,45594,32,37159,23,,,166326,45,356.9,1,16,22935,95,+++++,+++,26607,100,+++++,+++,+++++,+++,28495,99


Chunk size:  1024
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 73,9192 Sekunden, 58,1 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 19,6784 Sekunden, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-256   4G           45427  32 36280  22           158803  42 362.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 25353  99 25503  95 +++++ +++ 26475 100
raid5-1024-256,4G,,,45427,32,36280,22,,,158803,42,362.5,1,16,+++++,+++,+++++,+++,25353,99,25503,95,+++++,+++,26475,100


Chunk size:  1024
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 76,1866 Sekunden, 56,4 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,4955 Sekunden, 210 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-128   4G           43021  32 33947  23           151999  42 350.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19666  90 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-1024-128,4G,,,43021,32,33947,23,,,151999,42,350.6,0,16,19666,90,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  1024
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 79,3601 Sekunden, 54,1 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 21,0058 Sekunden, 204 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-64    4G           40781  30 33431  21           148294  40 354.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18983  94 +++++ +++ 30233 100 +++++ +++ +++++ +++ 26111  99
raid5-1024-64,4G,,,40781,30,33431,21,,,148294,40,354.6,0,16,18983,94,+++++,+++,30233,100,+++++,+++,+++++,+++,26111,99


Chunk size:  1024
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 81,6679 Sekunden, 52,6 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,5918 Sekunden, 209 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-32    4G           40169  30 33239  21           155120  42 338.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 32274  86 +++++ +++ +++++ +++ +++++ +++ +++++ +++ 26837  99
raid5-1024-32,4G,,,40169,30,33239,21,,,155120,42,338.7,0,16,32274,86,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26837,99


Chunk size:  1024
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 84,6962 Sekunden, 50,7 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,5898 Sekunden, 209 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-16    4G           39220  30 31604  20           150565  41 347.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 31511  94 +++++ +++ 24415 100 26566  95 +++++ +++ +++++ +++
raid5-1024-16,4G,,,39220,30,31604,20,,,150565,41,347.8,0,16,31511,94,+++++,+++,24415,100,26566,95,+++++,+++,+++++,+++


Chunk size:  1024
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930285568 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 91,1599 Sekunden, 47,1 MB/s

RAW read
1365+0 Datensätze ein
1365+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,7768 Sekunden, 207 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-8     4G           36976  26 30722  20           146899  41 339.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23504  98 +++++ +++ 27074  99 +++++ +++ +++++ +++ +++++ +++
raid5-1024-8,4G,,,36976,26,30722,20,,,146899,41,339.7,0,16,23504,98,+++++,+++,27074,99,+++++,+++,+++++,+++,+++++,+++


Chunk size:  512
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 64,9243 Sekunden, 66,1 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,9037 Sekunden, 205 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-512    4G           54883  37 41510  23           161461  39 356.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23127  97 +++++ +++ 27126 100 31788  88 +++++ +++ +++++ +++
raid5-512-512,4G,,,54883,37,41510,23,,,161461,39,356.0,1,16,23127,97,+++++,+++,27126,100,31788,88,+++++,+++,+++++,+++


Chunk size:  512
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 66,0985 Sekunden, 65,0 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,9707 Sekunden, 205 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-256    4G           55707  38 41697  23           162015  39 359.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18737  95 +++++ +++ 24738  99 +++++ +++ +++++ +++ 26717  99
raid5-512-256,4G,,,55707,38,41697,23,,,162015,39,359.3,1,16,18737,95,+++++,+++,24738,99,+++++,+++,+++++,+++,26717,99


Chunk size:  512
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 63,0405 Sekunden, 68,1 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,4682 Sekunden, 210 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-128    4G           56740  40 42806  24           166630  40 367.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18821  89 +++++ +++ 19445  80 20415  96 +++++ +++ 29868  99
raid5-512-128,4G,,,56740,40,42806,24,,,166630,40,367.4,0,16,18821,89,+++++,+++,19445,80,20415,96,+++++,+++,29868,99


Chunk size:  512
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 63,8452 Sekunden, 67,3 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 21,0159 Sekunden, 204 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-64     4G           56434  39 42049  24           158950  39 368.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20541  99 +++++ +++ 28462  93 +++++ +++ +++++ +++ +++++ +++
raid5-512-64,4G,,,56434,39,42049,24,,,158950,39,368.9,1,16,20541,99,+++++,+++,28462,93,+++++,+++,+++++,+++,+++++,+++


Chunk size:  512
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 69,3469 Sekunden, 61,9 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 21,104 Sekunden, 203 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-32     4G           52673  40 40229  23           152916  38 363.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18663  97 +++++ +++ 22457 100 23015  99 +++++ +++ 30797 100
raid5-512-32,4G,,,52673,40,40229,23,,,152916,38,363.1,1,16,18663,97,+++++,+++,22457,100,23015,99,+++++,+++,30797,100


Chunk size:  512
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 71,0779 Sekunden, 60,4 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 21,2219 Sekunden, 202 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-16     4G           50031  35 38853  21           156130  38 356.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22715  99 +++++ +++ 20617  86 +++++ +++ +++++ +++ 26704  99
raid5-512-16,4G,,,50031,35,38853,21,,,156130,38,356.4,1,16,22715,99,+++++,+++,20617,86,+++++,+++,+++++,+++,26704,99


Chunk size:  512
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 75,6919 Sekunden, 56,7 MB/s

RAW read
2730+0 Datensätze ein
2730+0 Datensätze aus
4293918720 Bytes (4,3 GB) kopiert, 20,7094 Sekunden, 207 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-8      4G           48642  34 38047  21           160889  39 359.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22720  98 +++++ +++ 24036  85 28420  85 +++++ +++ 32713  99
raid5-512-8,4G,,,48642,34,38047,21,,,160889,39,359.8,1,16,22720,98,+++++,+++,24036,85,28420,85,+++++,+++,32713,99


Chunk size:  256
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 58,231 Sekunden, 73,8 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,4732 Sekunden, 191 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-512    4G           63757  43 41872  21           161179  33 356.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18799  96 +++++ +++ 21537  84 +++++ +++ +++++ +++ +++++ +++
raid5-256-512,4G,,,63757,43,41872,21,,,161179,33,356.9,1,16,18799,96,+++++,+++,21537,84,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 58,7288 Sekunden, 73,1 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,1732 Sekunden, 194 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-256    4G           61955  41 42117  20           159276  32 354.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19279  99 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-256-256,4G,,,61955,41,42117,20,,,159276,32,354.7,0,16,19279,99,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 60,7463 Sekunden, 70,7 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,9683 Sekunden, 187 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-128    4G           59436  40 40913  20           158873  32 361.6   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22671  97 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-256-128,4G,,,59436,40,40913,20,,,158873,32,361.6,1,16,22671,97,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 62,0628 Sekunden, 69,2 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 23,3553 Sekunden, 184 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-64     4G           59342  43 40743  20           152804  31 358.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22073  95 +++++ +++ 29594  99 28599  99 +++++ +++ +++++ +++
raid5-256-64,4G,,,59342,43,40743,20,,,152804,31,358.4,0,16,22073,95,+++++,+++,29594,99,28599,99,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 61,0171 Sekunden, 70,4 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,4737 Sekunden, 191 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-32     4G           59409  42 41191  20           158059  31 369.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22923  98 +++++ +++ 24681  80 +++++ +++ +++++ +++ +++++ +++
raid5-256-32,4G,,,59409,42,41191,20,,,158059,31,369.0,0,16,22923,98,+++++,+++,24681,80,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 63,6786 Sekunden, 67,4 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,3311 Sekunden, 192 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-16     4G           55676  38 40154  19           155272  31 359.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22567  98 +++++ +++ 25491 100 +++++ +++ +++++ +++ +++++ +++
raid5-256-16,4G,,,55676,38,40154,19,,,155272,31,359.7,1,16,22567,98,+++++,+++,25491,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 72,476 Sekunden, 59,3 MB/s

RAW read
5461+0 Datensätze ein
5461+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 22,6458 Sekunden, 190 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-8      4G           51132  36 36603  18           147847  30 346.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23114  99 +++++ +++ 24802  99 31556  76 +++++ +++ 28139 100
raid5-256-8,4G,,,51132,36,36603,18,,,147847,30,346.0,0,16,23114,99,+++++,+++,24802,99,31556,76,+++++,+++,28139,100


Chunk size:  128
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 54,0318 Sekunden, 79,5 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 26,517 Sekunden, 162 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-512    4G           67024  44 38046  18           151708  28 368.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22467  95 +++++ +++ 26126 100 +++++ +++ +++++ +++ +++++ +++
raid5-128-512,4G,,,67024,44,38046,18,,,151708,28,368.1,1,16,22467,95,+++++,+++,26126,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 53,1742 Sekunden, 80,8 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 26,8148 Sekunden, 160 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-256    4G           67544  43 38478  17           156452  29 351.0   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19931  97 +++++ +++ 25386  99 +++++ +++ +++++ +++ +++++ +++
raid5-128-256,4G,,,67544,43,38478,17,,,156452,29,351.0,0,16,19931,97,+++++,+++,25386,99,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 55,1817 Sekunden, 77,8 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 27,5264 Sekunden, 156 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-128    4G           66143  42 38534  17           168539  31 360.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21179  91 +++++ +++ 30078  99 +++++ +++ +++++ +++ +++++ +++
raid5-128-128,4G,,,66143,42,38534,17,,,168539,31,360.4,1,16,21179,91,+++++,+++,30078,99,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 53,368 Sekunden, 80,5 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 25,4093 Sekunden, 169 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-64     4G           67181  42 38801  17           156906  29 362.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23030  97 +++++ +++ 24732  80 +++++ +++ +++++ +++ +++++ +++
raid5-128-64,4G,,,67181,42,38801,17,,,156906,29,362.7,1,16,23030,97,+++++,+++,24732,80,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 55,0117 Sekunden, 78,1 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 26,5023 Sekunden, 162 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-32     4G           62981  43 38609  18           151073  28 353.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21624  98 +++++ +++ 22879  99 21702  95 +++++ +++ 26161 100
raid5-128-32,4G,,,62981,43,38609,18,,,151073,28,353.7,0,16,21624,98,+++++,+++,22879,99,21702,95,+++++,+++,26161,100


Chunk size:  128
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 59,2844 Sekunden, 72,4 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 26,0047 Sekunden, 165 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-16     4G           59534  44 36088  17           155899  29 352.9   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 32588  96 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-128-16,4G,,,59534,44,36088,17,,,155899,29,352.9,0,16,32588,96,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287104 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 67,3536 Sekunden, 63,8 MB/s

RAW read
10922+0 Datensätze ein
10922+0 Datensätze aus
4294705152 Bytes (4,3 GB) kopiert, 27,2482 Sekunden, 158 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-8      4G           54037  37 34025  15           148177  27 354.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 32125  73 +++++ +++ +++++ +++
raid5-128-8,4G,,,54037,37,34025,15,,,148177,27,354.3,1,16,+++++,+++,+++++,+++,+++++,+++,32125,73,+++++,+++,+++++,+++


Chunk size:  64
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 51,0829 Sekunden, 84,1 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 34,6773 Sekunden, 124 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-512     4G           69866  42 35650  17           136541  24 364.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23372  96 +++++ +++ 21685  86 21490  93 +++++ +++ 28839  99
raid5-64-512,4G,,,69866,42,35650,17,,,136541,24,364.1,1,16,23372,96,+++++,+++,21685,86,21490,93,+++++,+++,28839,99


Chunk size:  64
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 51,816 Sekunden, 82,9 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 36,0709 Sekunden, 119 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-256     4G           68495  45 34508  16           139208  25 357.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19297  90 +++++ +++ 25628 100 +++++ +++ +++++ +++ 25352 100
raid5-64-256,4G,,,68495,45,34508,16,,,139208,25,357.1,1,16,19297,90,+++++,+++,25628,100,+++++,+++,+++++,+++,25352,100


Chunk size:  64
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 52,4412 Sekunden, 81,9 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 36,1506 Sekunden, 119 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-128     4G           69165  43 35540  16           133670  24 354.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20119  83 +++++ +++ 25692 100 24397  99 +++++ +++ 29648 100
raid5-64-128,4G,,,69165,43,35540,16,,,133670,24,354.2,0,16,20119,83,+++++,+++,25692,100,24397,99,+++++,+++,29648,100


Chunk size:  64
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 53,1809 Sekunden, 80,8 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 34,8616 Sekunden, 123 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-64      4G           67482  42 34286  16           136122  25 359.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18987  97 +++++ +++ 30550  99 25117  93 +++++ +++ +++++ +++
raid5-64-64,4G,,,67482,42,34286,16,,,136122,25,359.7,1,16,18987,97,+++++,+++,30550,99,25117,93,+++++,+++,+++++,+++


Chunk size:  64
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 54,3026 Sekunden, 79,1 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 36,2894 Sekunden, 118 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-32      4G           66273  44 33775  16           134606  25 354.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16313  98 +++++ +++ 23776  99 21796  94 +++++ +++ 25468 100
raid5-64-32,4G,,,66273,44,33775,16,,,134606,25,354.1,1,16,16313,98,+++++,+++,23776,99,21796,94,+++++,+++,25468,100


Chunk size:  64
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 58,2792 Sekunden, 73,7 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 34,9719 Sekunden, 123 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-16      4G           60404  37 34011  15           143417  26 332.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 25513  99 +++++ +++ 19331  76 21539  92 +++++ +++ 27364 100
raid5-64-16,4G,,,60404,37,34011,15,,,143417,26,332.9,1,16,25513,99,+++++,+++,19331,76,21539,92,+++++,+++,27364,100


Chunk size:  64
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 64,6569 Sekunden, 66,4 MB/s

RAW read
21845+0 Datensätze ein
21845+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 34,9528 Sekunden, 123 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-8       4G           55374  36 32321  15           139629  26 345.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22126  91 +++++ +++ 26532  99 31869  87 +++++ +++ +++++ +++
raid5-64-8,4G,,,55374,36,32321,15,,,139629,26,345.8,0,16,22126,91,+++++,+++,26532,99,31869,87,+++++,+++,+++++,+++


Chunk size:  32
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 50,1915 Sekunden, 85,6 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 53,1271 Sekunden, 80,8 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-512     4G           70760  44 30992  16           81190  17 347.9   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21865  96 +++++ +++ 24592  99 19328  99 +++++ +++ 29263 100
raid5-32-512,4G,,,70760,44,30992,16,,,81190,17,347.9,0,16,21865,96,+++++,+++,24592,99,19328,99,+++++,+++,29263,100


Chunk size:  32
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 49,4897 Sekunden, 86,8 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 54,5374 Sekunden, 78,8 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-256     4G           71793  45 31387  17           88111  19 352.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23237  94 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-32-256,4G,,,71793,45,31387,17,,,88111,19,352.8,1,16,23237,94,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  32
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 49,4241 Sekunden, 86,9 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 53,429 Sekunden, 80,4 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-128     4G           70404  40 31272  17           85745  18 339.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20471  97 +++++ +++ 29674  99 32724  95 +++++ +++ 32626 100
raid5-32-128,4G,,,70404,40,31272,17,,,85745,18,339.4,1,16,20471,97,+++++,+++,29674,99,32724,95,+++++,+++,32626,100


Chunk size:  32
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 51,9741 Sekunden, 82,6 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 53,7874 Sekunden, 79,8 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-64      4G           70446  46 31517  16           87237  18 357.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-32-64,4G,,,70446,46,31517,16,,,87237,18,357.4,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  32
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 50,7563 Sekunden, 84,6 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 52,3644 Sekunden, 82,0 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-32      4G           68135  42 30593  16           82078  17 346.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 31394  99
raid5-32-32,4G,,,68135,42,30593,16,,,82078,17,346.4,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,31394,99


Chunk size:  32
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 56,4179 Sekunden, 76,1 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 55,0037 Sekunden, 78,1 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-16      4G           62977  41 29604  15           82337  17 338.5   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23071  98 +++++ +++ 26835  99 26639  84 +++++ +++ +++++ +++
raid5-32-16,4G,,,62977,41,29604,15,,,82337,17,338.5,1,16,23071,98,+++++,+++,26835,99,26639,84,+++++,+++,+++++,+++


Chunk size:  32
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287296 blocks super 1.0 level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 61,8923 Sekunden, 69,4 MB/s

RAW read
43690+0 Datensätze ein
43690+0 Datensätze aus
4294901760 Bytes (4,3 GB) kopiert, 52,9937 Sekunden, 81,0 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-32-8       4G           59055  39 30465  16           90835  19 345.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22809  94 +++++ +++ 26384 100 +++++ +++ +++++ +++ +++++ +++
raid5-32-8,4G,,,59055,39,30465,16,,,90835,19,345.2,1,16,22809,94,+++++,+++,26384,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  16
BM chunk size:  524288

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 524288KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 46,6759 Sekunden, 92,0 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 60,7322 Sekunden, 70,7 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-512     4G           76616  50 30053  18           70686  17 331.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22769  97 +++++ +++ 26120 100 +++++ +++ +++++ +++ 28388  99
raid5-16-512,4G,,,76616,50,30053,18,,,70686,17,331.8,1,16,22769,97,+++++,+++,26120,100,+++++,+++,+++++,+++,28388,99


Chunk size:  16
BM chunk size:  262144

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 2/2 pages [8KB], 262144KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 47,9805 Sekunden, 89,5 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 61,9545 Sekunden, 69,3 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-256     4G           71308  46 28986  17           67698  17 331.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18085  96 +++++ +++ 21984 100 29970  95 +++++ +++ +++++ +++
raid5-16-256,4G,,,71308,46,28986,17,,,67698,17,331.2,1,16,18085,96,+++++,+++,21984,100,29970,95,+++++,+++,+++++,+++


Chunk size:  16
BM chunk size:  131072

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 4/4 pages [16KB], 131072KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 49,8801 Sekunden, 86,1 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 62,1272 Sekunden, 69,1 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-128     4G           71435  45 28882  17           68398  17 321.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22292  95 +++++ +++ 24351  99 +++++ +++ +++++ +++ 28510  73
raid5-16-128,4G,,,71435,45,28882,17,,,68398,17,321.4,0,16,22292,95,+++++,+++,24351,99,+++++,+++,+++++,+++,28510,73


Chunk size:  16
BM chunk size:  65536

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 49,5794 Sekunden, 86,6 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 63,6062 Sekunden, 67,5 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-64      4G           70578  43 28218  17           67001  17 311.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22457  97 +++++ +++ 24748  99 +++++ +++ +++++ +++ 27224  94
raid5-16-64,4G,,,70578,43,28218,17,,,67001,17,311.6,0,16,22457,97,+++++,+++,24748,99,+++++,+++,+++++,+++,27224,94


Chunk size:  16
BM chunk size:  32768

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 15/15 pages [60KB], 32768KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 52,1215 Sekunden, 82,4 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 63,324 Sekunden, 67,8 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-32      4G           68027  44 28480  17           68451  17 316.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24408  91 +++++ +++ +++++ +++ 23152  87 +++++ +++ 23775  87
raid5-16-32,4G,,,68027,44,28480,17,,,68451,17,316.1,1,16,24408,91,+++++,+++,+++++,+++,23152,87,+++++,+++,23775,87


Chunk size:  16
BM chunk size:  16384

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 30/30 pages [120KB], 16384KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 55,5958 Sekunden, 77,3 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 64,7869 Sekunden, 66,3 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-16      4G           63479  41 27328  16           64311  15 309.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 31516  99 +++++ +++ 25676 100 21536  99 +++++ +++ 24351  86
raid5-16-16,4G,,,63479,41,27328,16,,,64311,15,309.8,0,16,31516,99,+++++,+++,25676,100,21536,99,+++++,+++,24351,86


Chunk size:  16
BM chunk size:  8192

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active(auto-read-only) raid5 sde[3] sdd[2] sdc[1] sdb[0]
      2930287344 blocks super 1.0 level 5, 16k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 59/59 pages [236KB], 8192KB chunk

unused devices: <none>

RAW write
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 62,0137 Sekunden, 69,3 MB/s

RAW read
87381+0 Datensätze ein
87381+0 Datensätze aus
4294950912 Bytes (4,3 GB) kopiert, 66,3221 Sekunden, 64,8 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-16-8       4G           58136  40 26564  16           65783  16 307.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20129  96 +++++ +++ 26316  99 26425  92 +++++ +++ 25398  99
raid5-16-8,4G,,,58136,40,26564,16,,,65783,16,307.7,0,16,20129,96,+++++,+++,26316,99,26425,92,+++++,+++,25398,99

[-- Attachment #4: 2.test --]
[-- Type: application/octet-stream, Size: 39636 bytes --]



Chunk size:  1024 KB
BM chunk size:  512 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 524288KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 40.5786 seconds, 106 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.6135 seconds, 219 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-512   4G           95498  47 55099  31           181800  50 356.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 20313  81 29348  99 +++++ +++ 25667  99
raid5-1024-512,4G,,,95498,47,55099,31,,,181800,50,356.3,1,16,+++++,+++,+++++,+++,20313,81,29348,99,+++++,+++,25667,99


Chunk size:  1024 KB
BM chunk size:  256 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 262144KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 40.9307 seconds, 105 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.5909 seconds, 219 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-256   4G           92692  49 56209  32           182883  52 361.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 25816  99 +++++ +++ +++++ +++ +++++ +++
raid5-1024-256,4G,,,92692,49,56209,32,,,182883,52,361.2,1,16,+++++,+++,+++++,+++,25816,99,+++++,+++,+++++,+++,+++++,+++


Chunk size:  1024 KB
BM chunk size:  128 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 41.3454 seconds, 104 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.6736 seconds, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-128   4G           89952  43 55755  32           183070  52 356.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ 31179  99 +++++ +++ 27462  99
raid5-1024-128,4G,,,89952,43,55755,32,,,183070,52,356.1,0,16,+++++,+++,+++++,+++,+++++,+++,31179,99,+++++,+++,27462,99


Chunk size:  1024 KB
BM chunk size:  64 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 42.9143 seconds, 100 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.5113 seconds, 220 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-64    4G           87404  45 54147  32           180478  50 356.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 20382  80 19318  93 +++++ +++ 25119  99
raid5-1024-64,4G,,,87404,45,54147,32,,,180478,50,356.8,1,16,+++++,+++,+++++,+++,20382,80,19318,93,+++++,+++,25119,99


Chunk size:  1024 KB
BM chunk size:  32 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 45.4728 seconds, 94.4 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.9099 seconds, 216 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-32    4G           85629  46 52402  30           191734  52 356.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 17969  89 +++++ +++ 29295 100 29619  98 +++++ +++ +++++ +++
raid5-1024-32,4G,,,85629,46,52402,30,,,191734,52,356.8,0,16,17969,89,+++++,+++,29295,100,29619,98,+++++,+++,+++++,+++


Chunk size:  1024 KB
BM chunk size:  16 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 1024k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 16384KB chunk

unused devices: <none>

RAW write
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 47.0853 seconds, 91.2 MB/s

RAW read
1365+0 records in
1365+0 records out
4293918720 bytes (4.3 GB) copied, 19.5459 seconds, 220 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-1024-16    4G           73640  43 51536  29           182022  50 351.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
raid5-1024-16,4G,,,73640,43,51536,29,,,182022,50,351.7,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++


Chunk size:  512 KB
BM chunk size:  512 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 524288KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 39.5976 seconds, 108 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 19.7725 seconds, 217 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-512    4G           88372  45 53751  28           191042  46 373.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22820  98 +++++ +++ 24124  99 +++++ +++ +++++ +++ 28290  99
raid5-512-512,4G,,,88372,45,53751,28,,,191042,46,373.0,1,16,22820,98,+++++,+++,24124,99,+++++,+++,+++++,+++,28290,99


Chunk size:  512 KB
BM chunk size:  256 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 262144KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 40.5155 seconds, 106 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 19.6669 seconds, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-256    4G           90175  41 54175  27           185666  45 372.0   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19880  95 +++++ +++ 30023 100 28176  96 +++++ +++ 29616  72
raid5-512-256,4G,,,90175,41,54175,27,,,185666,45,372.0,1,16,19880,95,+++++,+++,30023,100,28176,96,+++++,+++,29616,72


Chunk size:  512 KB
BM chunk size:  128 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 40.9181 seconds, 105 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 19.742 seconds, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-128    4G           87302  40 53847  27           189503  45 377.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 25008 100 +++++ +++ 26976 100 31679  91 +++++ +++ +++++ +++
raid5-512-128,4G,,,87302,40,53847,27,,,189503,45,377.2,1,16,25008,100,+++++,+++,26976,100,31679,91,+++++,+++,+++++,+++


Chunk size:  512 KB
BM chunk size:  64 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 42.6449 seconds, 101 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 19.6633 seconds, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-64     4G           84251  43 53430  28           190712  48 376.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22908 100 +++++ +++ 30805 100 +++++ +++ +++++ +++ 21364  77
raid5-512-64,4G,,,84251,43,53430,28,,,190712,48,376.8,1,16,22908,100,+++++,+++,30805,100,+++++,+++,+++++,+++,21364,77


Chunk size:  512 KB
BM chunk size:  32 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 44.6483 seconds, 96.2 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 20.0377 seconds, 214 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-32     4G           82500  43 51850  27           186544  46 364.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24322  97 +++++ +++ 31309 100 +++++ +++ +++++ +++ +++++ +++
raid5-512-32,4G,,,82500,43,51850,27,,,186544,46,364.4,1,16,24322,97,+++++,+++,31309,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  512 KB
BM chunk size:  16 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 16384KB chunk

unused devices: <none>

RAW write
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 46.7281 seconds, 91.9 MB/s

RAW read
2730+0 records in
2730+0 records out
4293918720 bytes (4.3 GB) copied, 19.689 seconds, 218 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-512-16     4G           75349  40 50679  26           192831  47 373.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22475  96 +++++ +++ 32572 100 +++++ +++ +++++ +++ 30313  99
raid5-512-16,4G,,,75349,40,50679,26,,,192831,47,373.4,1,16,22475,96,+++++,+++,32572,100,+++++,+++,+++++,+++,30313,99


Chunk size:  256 KB
BM chunk size:  512 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 524288KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 39.8064 seconds, 108 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 20.5954 seconds, 209 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-512    4G           93154  43 51789  25           180362  36 373.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22590  98 +++++ +++ 25852 100 +++++ +++ +++++ +++ +++++ +++
raid5-256-512,4G,,,93154,43,51789,25,,,180362,36,373.1,1,16,22590,98,+++++,+++,25852,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256 KB
BM chunk size:  256 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 262144KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 40.0364 seconds, 107 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 20.6355 seconds, 208 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-256    4G           87925  45 50103  25           181072  37 368.9   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22094  97 +++++ +++ 25312 100 22751  94 +++++ +++ 25841  99
raid5-256-256,4G,,,87925,45,50103,25,,,181072,37,368.9,1,16,22094,97,+++++,+++,25312,100,22751,94,+++++,+++,25841,99


Chunk size:  256 KB
BM chunk size:  128 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 41.1222 seconds, 104 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 20.4957 seconds, 210 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-128    4G           85730  43 51644  26           184493  36 372.7   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 25045  99 +++++ +++ +++++ +++ 22390  71
raid5-256-128,4G,,,85730,43,51644,26,,,184493,36,372.7,1,16,+++++,+++,+++++,+++,25045,99,+++++,+++,+++++,+++,22390,71


Chunk size:  256 KB
BM chunk size:  64 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 42.5863 seconds, 101 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 21.175 seconds, 203 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-64     4G           89552  45 50419  25           183476  36 370.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23192  99 +++++ +++ 27721 100 +++++ +++ +++++ +++ +++++ +++
raid5-256-64,4G,,,89552,45,50419,25,,,183476,36,370.1,0,16,23192,99,+++++,+++,27721,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256 KB
BM chunk size:  32 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 44.3219 seconds, 96.9 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 20.589 seconds, 209 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-32     4G           81179  44 49705  24           179691  36 370.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23072 100 +++++ +++ 25838 100 +++++ +++ +++++ +++ +++++ +++
raid5-256-32,4G,,,81179,44,49705,24,,,179691,36,370.8,0,16,23072,100,+++++,+++,25838,100,+++++,+++,+++++,+++,+++++,+++


Chunk size:  256 KB
BM chunk size:  16 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 16384KB chunk

unused devices: <none>

RAW write
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 46.7222 seconds, 91.9 MB/s

RAW read
5461+0 records in
5461+0 records out
4294705152 bytes (4.3 GB) copied, 20.906 seconds, 205 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-256-16     4G           72610  42 48187  24           183348  35 364.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23412  99 +++++ +++ 24615  99 29517  84 +++++ +++ +++++ +++
raid5-256-16,4G,,,72610,42,48187,24,,,183348,35,364.1,1,16,23412,99,+++++,+++,24615,99,29517,84,+++++,+++,+++++,+++


Chunk size:  128 KB
BM chunk size:  512 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 524288KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 39.8155 seconds, 108 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 25.1119 seconds, 171 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-512    4G           97043  46 47814  24           172121  32 375.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23771  98 +++++ +++ 32280  99 +++++ +++ +++++ +++ +++++ +++
raid5-128-512,4G,,,97043,46,47814,24,,,172121,32,375.2,0,16,23771,98,+++++,+++,32280,99,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128 KB
BM chunk size:  256 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 262144KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 40.2179 seconds, 107 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 22.6821 seconds, 189 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-256    4G           93185  47 46657  23           173416  33 367.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18597  96 +++++ +++ 17321  79 20444  95 +++++ +++ 29672  99
raid5-128-256,4G,,,93185,47,46657,23,,,173416,33,367.4,1,16,18597,96,+++++,+++,17321,79,20444,95,+++++,+++,29672,99


Chunk size:  128 KB
BM chunk size:  128 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 40.9995 seconds, 105 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 24.2446 seconds, 177 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-128    4G           91055  45 46848  22           173601  32 369.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20715  99 +++++ +++ 22151  79 +++++ +++ +++++ +++ +++++ +++
raid5-128-128,4G,,,91055,45,46848,22,,,173601,32,369.4,1,16,20715,99,+++++,+++,22151,79,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128 KB
BM chunk size:  64 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 42.5521 seconds, 101 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 24.3978 seconds, 176 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-64     4G           88929  45 46437  22           175345  33 367.5   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20554  97 +++++ +++ 25292  79 +++++ +++ +++++ +++ +++++ +++
raid5-128-64,4G,,,88929,45,46437,22,,,175345,33,367.5,0,16,20554,97,+++++,+++,25292,79,+++++,+++,+++++,+++,+++++,+++


Chunk size:  128 KB
BM chunk size:  32 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 44.5575 seconds, 96.4 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 23.097 seconds, 186 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-32     4G           81481  43 46331  22           172024  32 366.2   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22892  97 +++++ +++ 20822  85 23391  99 +++++ +++ 26530  99
raid5-128-32,4G,,,81481,43,46331,22,,,172024,32,366.2,1,16,22892,97,+++++,+++,20822,85,23391,99,+++++,+++,26530,99


Chunk size:  128 KB
BM chunk size:  16 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 16384KB chunk

unused devices: <none>

RAW write
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 46.4805 seconds, 92.4 MB/s

RAW read
10922+0 records in
10922+0 records out
4294705152 bytes (4.3 GB) copied, 24.6012 seconds, 175 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-128-16     4G           79314  46 44287  21           171650  32 369.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23251  99 +++++ +++ 26461 100 26709  91 +++++ +++ +++++ +++
raid5-128-16,4G,,,79314,46,44287,21,,,171650,32,369.1,0,16,23251,99,+++++,+++,26461,100,26709,91,+++++,+++,+++++,+++


Chunk size:  64 KB
BM chunk size:  512 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 524288KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 39.7231 seconds, 108 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 32.1528 seconds, 134 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-512     4G           92985  46 42132  21           156068  29 371.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21908  96 +++++ +++ 30917 100 29271  97 +++++ +++ +++++ +++
raid5-64-512,4G,,,92985,46,42132,21,,,156068,29,371.4,1,16,21908,96,+++++,+++,30917,100,29271,97,+++++,+++,+++++,+++


Chunk size:  64 KB
BM chunk size:  256 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 262144KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 40.2671 seconds, 107 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 32.5262 seconds, 132 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-256     4G           98544  49 40969  21           155997  30 366.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21010  99 +++++ +++ 26723  99 17128  90 +++++ +++ 24930  99
raid5-64-256,4G,,,98544,49,40969,21,,,155997,30,366.8,1,16,21010,99,+++++,+++,26723,99,17128,90,+++++,+++,24930,99


Chunk size:  64 KB
BM chunk size:  128 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 131072KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 41.0159 seconds, 105 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 32.173 seconds, 133 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-128     4G           100053  48 39885  21           159066  29 361.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18001  94 +++++ +++ 24494 100 18526  95 +++++ +++ 24797  99
raid5-64-128,4G,,,100053,48,39885,21,,,159066,29,361.3,1,16,18001,94,+++++,+++,24494,100,18526,95,+++++,+++,24797,99


Chunk size:  64 KB
BM chunk size:  64 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 42.2257 seconds, 102 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 31.0066 seconds, 139 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-64      4G           87611  42 42815  20           153014  27 367.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 22569  91 +++++ +++ 27236  99 +++++ +++ +++++ +++ 28324  99
raid5-64-64,4G,,,87611,42,42815,20,,,153014,27,367.3,0,16,22569,91,+++++,+++,27236,99,+++++,+++,+++++,+++,28324,99


Chunk size:  64 KB
BM chunk size:  32 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 44.4222 seconds, 96.7 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 31.5176 seconds, 136 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-32      4G           86426  45 41501  20           156154  28 367.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 23451 100 +++++ +++ 26292 100 25721  90 +++++ +++ 28475 100
raid5-64-32,4G,,,86426,45,41501,20,,,156154,28,367.1,0,16,23451,100,+++++,+++,26292,100,25721,90,+++++,+++,28475,100


Chunk size:  64 KB
BM chunk size:  16 MB

Wait for sync ...
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
      12582912 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/1 pages [0KB], 16384KB chunk

unused devices: <none>

RAW write
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 46.6332 seconds, 92.1 MB/s

RAW read
21845+0 records in
21845+0 records out
4294901760 bytes (4.3 GB) copied, 31.9816 seconds, 134 MB/s

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raid5-64-16      4G           75749  45 39776  20           165845  30 359.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 21195  85
raid5-64-16,4G,,,75749,45,39776,20,,,165845,30,359.1,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,21195,85

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-04 16:54                 ` Christian Pernegger
@ 2008-03-05  6:38                   ` Christian Pernegger
  2008-03-10 14:03                     ` Christian Pernegger
  0 siblings, 1 reply; 19+ messages in thread
From: Christian Pernegger @ 2008-03-05  6:38 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

Added the dm-crypt layer back again, just to see what it would do:
- responsiveness problem is back
- performance is around the level of a single disk, if that

Played around some more and it seems that the system just reacts very
badly to some I/O loads. fs creation / bonnie on a resyncing RAID
apparently can trigger the same symptoms (even without dm-crypt),
though not all configurations.

Just for kicks I tried raid on dm-crypt, which doesn't have the
problem but is even slower.

Back to square one.

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-05  6:38                   ` Christian Pernegger
@ 2008-03-10 14:03                     ` Christian Pernegger
  0 siblings, 0 replies; 19+ messages in thread
From: Christian Pernegger @ 2008-03-10 14:03 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

Dropping linux-ide as the HSM violation error was not reproducible
even a second time and I have since switched controllers from Promise
SATA300 TX4s (sata_promise) to Dawicontrol 4320 RAIDs (sil_3124 /
sata_sil24).

BTW the latter can have it's fakeraid firmware disabled via jumper,
which also skips any long option ROM / init sequences during boot.
When doing streaming seqential reads | writes on all four disk block
devices in parallel I get close to 300MB/s in both cases, which is the
limit for both the disk and controller hardware. Great driver!

Before it starts sounding too much like spam - I've only been pounding
the setup for a week, so no idea if it's really stable. Also I haven't
tested for data corruption yet :)

Cheers,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: Setting up md-raid5: observations, errors, questions
  2008-03-02 16:32   ` Christian Pernegger
  2008-03-02 18:33     ` Michael Tokarev
@ 2008-03-02 18:53     ` Christian Pernegger
  1 sibling, 0 replies; 19+ messages in thread
From: Christian Pernegger @ 2008-03-02 18:53 UTC (permalink / raw)
  To: linux-raid; +Cc: linux-ide

Some more data, captured with "vmstat 2 10" during "mke2fs -j -E
stride=16" (to match default chunk size of 64k). The disks are still
WD RE2-GP 1TB.


single disk (no stride):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  2      0 1391888 546344  12784    0    0   527  3795  187  700  0  6 85  9
 0  3      0 1260596 673608  12860    0    0     0 64816  402  136  0  7  0 93
 0  3      0 1130200 800808  12836    0    0     0 56576  405  135  0  7  0 93
 0  3      0 1007040 919952  12796    0    0     0 67108  405  149  0  7  0 93
 0  3      0 892572 1030672  12852    0    0     0 54528  398  129  0  8  0 92
 0  3      0 753968 1165840  12792    0    0     0 61696  404  145  0 12  0 88
 0  3      0 631500 1284656  12788    0    0     0 61184  403  136  0 10  0 90
 0  3      0 500448 1411856  12868    0    0     0 65536  404  139  0 10  0 90
 0  3      0 382016 1526736  12860    0    0     0 59392  400  132  0  9  0 91
 0  3      0 251276 1653840  12792    0    0     0 58880  403  138  0 11  0 89


RAID1 (2 disks, no stride):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 1346104 586692  11716    0    0   524  3914  187  698  0  6 84 10
 0  0      0 1236228 697284  11776    0    0     0 41452  568 2932  0 14 86  0
 0  0      0 1130244 799684  11768    0    0     0 57518  670 2164  0 13 86  0
 0  0      0 1013020 914200  11752    0    0     0 51870  637 1572  0 14 86  0
 1  0      0 899232 1024972  11720    0    0     0 55504  632 2164  0 15 85  0
 1  0      0 788188 1132912  11728    0    0     0 52908  643 1839  0 16 83  0
 0  0      0 785120 1135564  11768    0    0     0 49980  660 2351  0 13 88  0
 2  0      0 667624 1250252  11768    0    0     0 50028  671 2304  0 17 83  0
 0  0      0 549556 1364940  11768    0    0     0 48186  651 2060  0 17 83  0
 0  0      0 427292 1483724  11768    0    0     0 48568  711 3367  0 18 82  0

[progress of "writing inode tables" pauses regularly then increases in a burst]


RAID0 (2 disks):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1      0 1333272 566348  10716    0    0   515  4452  188  708  0  6 84 10
 0  2      0 1084244 808332  10632    0    0     0 123904  547  420  0 17 49 33
 1  1      0 847580 1039004  10720    0    0     0 113228  539  498  0 18 50 32
 1  1      0 603576 1276012  10724    0    0     0 119416  549  505  0 20 50 30
 0  2      0 366996 1505836  10692    0    0     0 120636  544  499  0 19 50 31
 1  1      0 113540 1751948  10700    0    0     0 116764  549  516  0 21 50 29
 0  2      0  12820 1849320  10092    0    0     0 122852  544  637  0 21 50 29
 0  2      0  11544 1850664  10160    0    0     0 120832  549  760  0 22 49 29
 1  1      0  11892 1850312   9980    0    0     0 117996  539  732  0 22 48 30
 0  2      0  12312 1849960   9980    0    0     0 107284  520  700  0 20 48 32


RAID1 (4 disks, no stride)

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 1701472 240556  11600    0    0   512  4653  189  706  0  6 84 10
 0  0      0 1487172 453548  11580    0    0     0 26432  705 8308  0 15 85  0
 2  0      0 1487804 453548  11580    0    0     0 28214 1122 2917  0  9 91  0
 1  3      0 1309804 609292  11544    0    0     4 72986 1019 2111  0 21 78  1
 3  0      0 1279008 626284  11584    0    0     0 63262  551  236  0 13 38 49
 0  1      0 1294940 626284  11584    0    0     0     0  549 8816  0  8 49 43
 0  0      0 1098588 831088  11596    0    0     0  6752  586 14067  0 13 78  8
 0  0      0 1098672 831088  11584    0    0     0 33944  772 1183  0  9 91  0
 0  0      0 981492 945776  11584    0    0     0 32974  841 4643  0 15 85  0
 0  0      0 981436 945776  11584    0    0     0 30546 1120 2474  0 11 89  0

[extremely bursty, can't data be written to the disks in parallel?]


RAID0 (4 disks):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  2      0 945164 866516  11620    0    0   507  4675  190  707  0  6 84 10
 0  1      0 633716 1169620  11528    0    0     0 151552  623  734  0 23 50 27
 1  0      0 324452 1470016  11540    0    0     0 149504  622  717  0 24 50 26
 1  0      0  14644 1771024  11540    0    0     0 149522  622  689  0 25 50 25
 1  0      0  11948 1773160  11044    0    0     0 151552  621  992  0 28 48 24
 1  1      0  12788 1772420  11156    0    0     0 151552  623  985  0 28 48 23
 0  1      0  11952 1773060  11088    0    0     0 151552  622 1004  0 27 48 25
 1  0      0  12744 1772220  11172    0    0     0 149504  620 1000  0 27 48 25
 0  1      0  11888 1773192  11172    0    0     0 151552  622  967  0 28 47 25
 0  1      0  12860 1773000  10268    0    0     0 151560  624  994  0 29 48 23

[there seems to be a cap when writing @~150MB/s, 4 single disks in
parallel yield the same value. It's not the bus so it's probably the
controller. Anyway I can live with that.]


RAID5 (4 disks, syncing in b/g):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 1223568 636836  11340    0    0   501  4748  190  702  0  6 84 10
 0  0      0 1067636 788388  11372    0    0     0 63074 1639 19766  0 32 45 23
 3  0      0 945316 907172  11276    0    0     0 63294 1684 20441  0 31 47 22
 1  1      0 852584 997292  11340    0    0     0 57012 1651 15925  0 27 54 18
 2  1      0 717548 1128364  11340    0    0     0 61824 1659 20125  0 31 46 23
 0  0      0 586852 1255340  11340    0    0     0 60608 1643 14772  0 29 49 22
 2  1      0 447692 1390508  11368    0    0     0 61400 1703 18710  0 31 43 26
 3  0      0 333892 1501100  11340    0    0     0 64998 1769 20846  0 33 45 23
 3  0      0 190696 1640364  11336    0    0     0 60992 1683 18032  0 32 48 20
 0  1      0 110568 1718188  11340    0    0     0 59970 1651 13064  0 25 57 18

[burstier than RAID0 or the single disk but a lot smoother than RAID1.
Keep in mind that it is syncing in parallel. NO responsiveness
problem.]


RAID5 (4 disks, --assume-clean):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  2      0  11332 1814052  10536    0    0   472  4739  214  819  0  7 84  9
 2  1      0  12304 1812828  10540    0    0     0 73586 1562 23273  0 38 41 20
 0  0      0  13004 1812188  10584    0    0     0 69642 1649 19816  0 34 44 22
 0  1      0  12188 1813084  10580    0    0     0 72452 1675 20730  0 37 42 21
 2  0      0  11784 1813596  10540    0    0     0 74662 1776 20616  0 37 42 21
 0  0      0  12348 1812956  10548    0    0     0 69546 1578 19984  0 32 47 21
 2  1      0  11416 1813724  10608    0    0     0 71092 1712 20723  0 37 41 22
 1  1      0  12496 1812880  10624    0    0     0 71368 1608 22813  0 38 42 20
 2  0      0  11436 1813852  10628    0    0     0 74796 1727 22632  0 38 40 22
 0  1      0  12552 1812572  10564    0    0     0 70248 1656 12608  0 33 48 19



Aside from the fact that RAID1 writes are somewhat erratic these
values seem ok to me. I have no idea how fast a degraded 3/4 disk
RAID5 array should be but it's still faster than a single disk. No
responsiveness problems in any test. Is it possible that the Promise
doesn't like the requests generated by the 1M chunk size used for the
original setup? dmesg is silent so I guess I'll be doing chunksize
tests this evening.

Thanks,

C.

^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2008-03-10 14:03 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-03-02 12:23 Setting up md-raid5: observations, errors, questions Christian Pernegger
2008-03-02 12:41 ` Justin Piszcz
2008-03-02 12:56   ` Christian Pernegger
2008-03-02 13:03     ` Justin Piszcz
2008-03-02 13:24       ` Christian Pernegger
2008-03-03 17:59         ` Bill Davidsen
2008-03-03 20:19           ` Christian Pernegger
2008-03-02 15:20 ` Michael Tokarev
2008-03-02 16:32   ` Christian Pernegger
2008-03-02 18:33     ` Michael Tokarev
2008-03-02 21:19       ` Christian Pernegger
2008-03-02 21:56         ` Michael Tokarev
2008-03-03  0:17           ` Christian Pernegger
2008-03-03  2:58             ` Michael Tokarev
2008-03-03  8:38               ` Christian Pernegger
2008-03-04 16:54                 ` Christian Pernegger
2008-03-05  6:38                   ` Christian Pernegger
2008-03-10 14:03                     ` Christian Pernegger
2008-03-02 18:53     ` Christian Pernegger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).