One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Hi All,

I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives
attached.  When it's formated with mdraid6+ext4 I get about 1200MB/s
for multiple streaming random reads with iozone.  With btrfs in
3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a
time.

As soon as I add a second (or more), the speed will drop to about
750MB/s.  If I add more streams (10, 20, etc), the total throughput
stays at around 750MB/s.  I only see the full 1200MB/s in btrfs when
I'm running a single read at a time (e.g. sequential reads with dd,
random reads with iozone, etc).

This feel like a bug or mis-configuration on my system.   As if can
read at the full speed, but just only with one stream running at a
time.  The options I have tried varying are "-l 64k" with mkfs.btrfs,
and "-o thread_pool=16" when mounting.  But, neither of those options
seem to change the behaviour.



Anyone know any reasons why I would see the speed drop when going from
one to more then one stream at a time with btrfs raid6?  We would like
to use btrfs (mostly for snapshots), but we do need to get the full
1200MB/s streaming speeds too..





Thanks,
Matt



___
Here's some example output..



Single thread = ~1.1GB/s
_____
kura1 persist # sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile bs=640k count=20000
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 7.14139 s, 1.8 GB/s

kura1 persist # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 11.2666 s, 1.2 GB/s

kura1 persist # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 11.5005 s, 1.1 GB/s

____



1 thread = ~1000MB/s ...
___
kura1 scripts # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/var/data/persist/testfile_$j bs=640k ; done
vm.drop_caches = 1
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 6.52018 s, 1.0 GB/s
kura1 scripts # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/var/data/persist/testfile_$j bs=640k ; done
vm.drop_caches = 1
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 6.55731 s, 999 MB/s
___


2 threads = ~750MB/s combined...
___
# sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd of=/dev/null
if=/var/data/persist/testfile_$j bs=640k & done
vm.drop_caches = 1
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 17.5068 s, 374 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 17.7599 s, 369 MB/s
___



20 threads = ~750MB/s combined...
___
# sysctl vm.drop_caches=1 ; for j in {1..20} ; do dd of=/dev/null
if=/var/data/persist/testfile_$j bs=640k & done
vm.drop_caches = 1
kura1 scripts # 10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 168.223 s, 39.0 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 168.275 s, 38.9 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 169.466 s, 38.7 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 169.606 s, 38.6 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.503 s, 38.4 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.629 s, 38.4 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.633 s, 38.4 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.744 s, 38.4 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.844 s, 38.4 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 170.896 s, 38.3 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.027 s, 38.3 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.135 s, 38.3 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.389 s, 38.2 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.414 s, 38.2 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.674 s, 38.2 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.897 s, 38.1 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.956 s, 38.1 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 171.995 s, 38.1 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 172.044 s, 38.1 MB/s
10000+0 records in
10000+0 records out
6553600000 bytes (6.6 GB) copied, 172.08 s, 38.1 MB/s
____



### Similar results with random reads in iozone...

1 thread = ~1000MB/s
_____
kura1 scripts # for j in {1..1} ; do sysctl vm.drop_caches=1 ; iozone
-f /var/data/10GBfolders/folder$j/iozone.DUMMY.1 -c -M -r 5120k -s 2g
-i 1 -w -+A 1 | tail -n 5 & done
vm.drop_caches = 1
[1] 22298
kura1 scripts #
     random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                  1077376  7288014

iozone test complete.
____



2 threads = ~750 MB/s combined...
___
# for j in {1..2} ; do sysctl vm.drop_caches=1 ; iozone -f
/var/data/10GBfolders/folder$j/iozone.DUMMY.1 -c -M -r 5120k -s 2g -i
1 -w -+A 1 | tail -n 5 & done
vm.drop_caches = 1
[1] 22302
vm.drop_caches = 1
[2] 22305
kura1 scripts #
     random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                   368864  5090095

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                   366834  5105457

iozone test complete.


20 threads = ~750MB/s combined...
___
# for j in {1..20} ; do sysctl vm.drop_caches=1 ; iozone -f
/var/data/10GBfolders/folder$j/iozone.DUMMY.1 -c -M -r 5120k -s 2g -i
1 -w -+A 1 | tail -n 5 & done

                                                             random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    40424  6459500

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    39678  5749776

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    39548  5417189

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38988  5924904

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38484  1963969

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38556  1793398

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38610  1343518

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38346  1394609

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38367  1163930

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38375  1143491

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38647  1046416

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38180  1115287

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38086  1192537

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38356  1120244

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38293  1138119

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    37966  1273741

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    38059  1201688

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    37947  1243573

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    37965  1245834

iozone test complete.
                                                            random
random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read
write    read  rewrite     read   fwrite frewrite   fread  freread
         2097152    5120                    37840  1354806

iozone test complete.
___



### Typical dstat output during multi-thread read running and then
finish and go idle...
___
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
0  10  28  62   0   0| 716M    0 | 582B  870B|   0     0 |4398    16k
  0  12  28  59   0   0| 728M    0 | 454B  982B|   0     0 |4665    16k
  0  12  25  63   0   0| 761M    0 | 454B 1112B|   0     0 |4661    16k
  0  11  22  66   0   0| 719M    0 | 390B  742B|   0     0 |4519    16k
  0  13  21  65   0   0| 741M    0 | 524B 1036B|   0     0 |4706    16k
  0  17  19  63   0   0| 706M    0 |3302B 3558B|   0     0 |4638    15k
  0  16  17  67   0   0| 721M    0 |  16k   15k|   0     0 |5002    17k
  2  72   7  19   0   0| 514M    0 | 454B  486B|   0     0 |4174  8591
  3  97   0   0   0   0|   0     0 | 788B 2884B|   0     0 |1280   380
  1  38  61   0   0   0|   0     0 |1428B 7460B|   0     0 | 888   346
  0   0 100   0   0   0|   0     0 | 582B  678B|   0     0 |  92   106
  0   0 100   0   0   0|   0     0 |1606B 1766B|   0     0 |  66    59
  0   0 100   0   0   0|   0  4096B| 390B  742B|   0     0 |  90   112
  0   0 100   0   0   0|   0     0 | 454B  486B|   0     0 |  45    65
  0   0 100   0   0   0|   0     0 | 454B  614B|   0     0 |  56    77
___



### Some system info...
____
## Kernel = 3.9.0-rc4
# uname -a
Linux server 3.9.0-rc4 #4 SMP Fri Apr 5 00:58:28 UTC 2013 x86_64
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux
# grep MemTotal /proc/meminfo
MemTotal:       65975896 kB

___
## 12 2.3 GHz Xeon cores...
kura1 scripts # head -n 26 /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 6
microcode : 0x616
cpu MHz : 2301.000
cache size : 15360 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb
xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 4600.26
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
___
## Asus Z9PA-U8 MB
# dmidecode --type 1
# dmidecode 2.11
SMBIOS 2.7 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: Z9PA-U8 Series
Version: 1.0X
Serial Number: To be filled by O.E.M.
UUID: 598C1800-5BCB-11D9-8F58-3085A9A7CBC7
Wake-up Type: Power Switch
SKU Number: SKU
Family: To be filled by O.E.M.
____

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Sander]

Matt Pursley wrote (ao):
> I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives
> attached.  When it's formated with mdraid6+ext4 I get about 1200MB/s
> for multiple streaming random reads with iozone.  With btrfs in
> 3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a
> time.

Just curious, is that btrfs on top of mdraid6, or is this experimental
btrfs raid6 without md?

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

On Tue, Apr 16, 2013 at 11:55 PM, Sander <sander@humilis.net> wrote:
> Matt Pursley wrote (ao):
>> I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives
>> attached.  When it's formated with mdraid6+ext4 I get about 1200MB/s
>> for multiple streaming random reads with iozone.  With btrfs in
>> 3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a
>> time.
>
> Just curious, is that btrfs on top of mdraid6, or is this experimental
> btrfs raid6 without md?



This is the "experimental btrfs raid6 without md".

But, I did do a "mdraid6 with btrfs" test last night... and with that
setup I only get the ~750MB/s result.. even with just one
thread/stream...

I will flip the system back to "btrfsraid6+btrfs" today to verify that
I still get the full 1200MB/s with one stream/thread and ~750MB/s with
two or more streams/threads with that setup...


Thanks,
Matt



___
mdraid6+btrfs_64GBRam_80files # sysctl vm.drop_caches=1 ; dd
of=/dev/null if=/var/data/persist/testfile bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 18.2109 s, 720 MB/s
___

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Hey All,

Here are the results of making and reading back a 13GB file on
"mdraid6 + ext4",  "mdraid6 + btrfs", and "btrfsraid6 + btrfs".

Seems to show that:
1) "mdraid6 + ext4" can do ~1100 MB/s for these sequential reads with
either one or two files at once.
2) "btrfsraid6 + btrfs" can do ~1100 MB/s for sequential reads with
one file at a time, but only ~750 MB/s with two (or more).
3) "mdraid6 + btrfs" can only do ~750 MB/s for these sequential reads
with either one or two files at once.


So, seems like the speed drop is related more to the btrfs files
system, then the experimental raid.
Although it is interesting that btrfs can only do the full ~1100 MB/s
with a single file on the btrfsraid6, but not mdraid6.


Anyway, just some more info and reproducible results.

I have also opened a ticket in bugzilla.kernel.org for this issue here...
https://bugzilla.kernel.org/show_bug.cgi?id=56771




Thanks,
Matt






___ mdraid6 + ext4 ___

kura1 / # mount | grep -i /var/data
/dev/md0 on /var/data type ext4 (rw)

kura1 / # cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
[linear] [multipath]
md0 : active raid6 sdm[11] sdl[10] sdk[9] sdj[8] sdi[7] sdh[6] sdg[5]
sdf[4] sde[3] sdd[2] sdc[1] sdb[0]
      29302650880 blocks super 1.2 level 6, 512k chunk, algorithm 2
[12/12] [UUUUUUUUUUUU]
      [>....................]  resync =  0.0% (2731520/2930265088)
finish=47268.1min speed=1031K/sec

unused devices: <none>



## Create two 13GB testfiles...

kura1 / #  sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile1 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 47.27 s, 277 MB/s
kura1 / #  sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile2 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 47.0237 s, 279 MB/s


## Read back one testfile... ~1300 MB/s

kura1 / # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 10.3469 s, 1.3 GB/s
kura1 / # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 10.0073 s, 1.3 GB/s
kura1 / # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 10.69 s, 1.2 GB/s



## Read back the two testfiles at the same time.. ~1100MB/s

kura1 / # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.4988 s, 535 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.591 s, 533 MB/s

kura1 / # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.7013 s, 531 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.7016 s, 531 MB/s

kura1 / # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.5512 s, 534 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 24.8276 s, 528 MB/s

________________________________


___ mdraid6 + btrfs _______________


kura1 ~ # mount | grep -i /var/data
/dev/md0 on /var/data type btrfs (rw,noatime)

kura1 ~ # cat  /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
[linear] [multipath]
md0 : active raid6 sdm[11] sdl[10] sdk[9] sdj[8] sdi[7] sdh[6] sdg[5]
sdf[4] sde[3] sdd[2] sdc[1] sdb[0]
      29302650880 blocks super 1.2 level 6, 512k chunk, algorithm 2
[12/12] [UUUUUUUUUUUU]
      [>....................]  resync =  0.0% (1917184/2930265088)
finish=44415.7min speed=1098K/sec

unused devices: <none>

kura1 ~ # btrfs filesystem show
failed to open /dev/sr0: No medium found
Label: none  uuid: 5eb756b5-03a1-4d06-8e91-0f683a763a88
Total devices 1 FS bytes used 448.00KB
devid    1 size 27.29TB used 2.04GB path /dev/md0

Label: none  uuid: 4546715c-8948-42b3-b529-a1c9cd175c2e
Total devices 12 FS bytes used 80.74GB
devid   12 size 2.73TB used 9.35GB path /dev/sdm
devid   11 size 2.73TB used 9.35GB path /dev/sdl
devid   10 size 2.73TB used 9.35GB path /dev/sdk
devid    9 size 2.73TB used 9.35GB path /dev/sdj
devid    8 size 2.73TB used 9.35GB path /dev/sdi
devid    7 size 2.73TB used 9.35GB path /dev/sdh
devid    6 size 2.73TB used 9.35GB path /dev/sdg
devid    5 size 2.73TB used 9.35GB path /dev/sdf
devid    4 size 2.73TB used 9.35GB path /dev/sde
devid    3 size 2.73TB used 9.35GB path /dev/sdd
devid    2 size 2.73TB used 9.35GB path /dev/sdc
devid    1 size 2.73TB used 9.37GB path /dev/sdb

Btrfs v0.20-rc1-253-g7854c8b


## Create two 13GB testfiles...

kura1 ~ # sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile1 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 34.2789 s, 382 MB/s
kura1 ~ # sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile2 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 43.2937 s, 303 MB/s



## Read back one testfile... ~750 MB/s

kura1 ~ # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 16.7785 s, 781 MB/s
kura1 ~ # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 18.1361 s, 723 MB/s
kura1 ~ # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 19.1985 s, 683 MB/s


## Read back the two testfiles at the same time.. ~750MB/s

kura1 ~ # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 30.8396 s, 425 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 35.5478 s, 369 MB/s

kura1 ~ # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 34.6504 s, 378 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 35.7795 s, 366 MB/s

kura1 ~ # (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=//dev/null if=/var/data/persist/testfile2 bs=640k) & wait
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 36.9101 s, 355 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 37.7395 s, 347 MB/s


________________________



___ btrfsraid6 + btrfs ___

kura1 ~ # mount | grep -i /var/data
/dev/sdl on /var/data type btrfs (rw,noatime)

kura1 ~ # btrfs filesystem show
failed to open /dev/sr0: No medium found
Label: none  uuid: 4546715c-8948-42b3-b529-a1c9cd175c2e
Total devices 12 FS bytes used 80.74GB
devid   12 size 2.73TB used 9.35GB path /dev/sdm
devid   11 size 2.73TB used 9.35GB path /dev/sdl
devid   10 size 2.73TB used 9.35GB path /dev/sdk
devid    9 size 2.73TB used 9.35GB path /dev/sdj
devid    8 size 2.73TB used 9.35GB path /dev/sdi
devid    7 size 2.73TB used 9.35GB path /dev/sdh
devid    6 size 2.73TB used 9.35GB path /dev/sdg
devid    5 size 2.73TB used 9.35GB path /dev/sdf
devid    4 size 2.73TB used 9.35GB path /dev/sde
devid    3 size 2.73TB used 9.35GB path /dev/sdd
devid    2 size 2.73TB used 9.35GB path /dev/sdc
devid    1 size 2.73TB used 9.37GB path /dev/sdb

Btrfs v0.20-rc1-253-g7854c8b


## Create two 13GB testfiles...

kura1 data # sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile2 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 21.5018 s, 610 MB/s
kura1 data # sysctl vm.drop_caches=1 ; dd if=/dev/zero
of=/var/data/persist/testfile1 bs=640k count=20000 conv=fdatasync
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 21.3389 s, 614 MB/s



## Read back one testfile... ~1100 MB/s

kura1 data # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 11.8312 s, 1.1 GB/s
kura1 data # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 11.7888 s, 1.1 GB/s
kura1 data # sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k
vm.drop_caches = 1
20000+0 records in
20000+0 records out

20000+0 records out
13107200000 bytes (13 GB) copied, 41.4113 s, 317 MB/s

kura1 data #  (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=/dev/null if=/var/data/persist/testfile2 bs=640k) & wait
[1] 19482
[2] 19483
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 36.0124 s, 364 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 36.2298 s, 362 MB/s

kura1 data #  (sysctl vm.drop_caches=1 ; dd of=/dev/null
if=/var/data/persist/testfile1 bs=640k) & (sysctl vm.drop_caches=1 ;
dd of=/dev/null if=/var/data/persist/testfile2 bs=640k) & wait
[1] 19500
[2] 19501
vm.drop_caches = 1
vm.drop_caches = 1
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 35.4703 s, 370 MB/s
20000+0 records in
20000+0 records out
13107200000 bytes (13 GB) copied, 35.7789 s, 366 MB/s
[1]-  Done                    ( sysctl vm.drop_caches=1; dd
of=/dev/null if=/var/data/persist/testfile1 bs=640k )
[2]+  Done                    ( sysctl vm.drop_caches=1; dd
of=/dev/null if=/var/data/persist/testfile2 bs=640k )


_____



On Wed, Apr 17, 2013 at 10:30 AM, Matt Pursley <mpursley@gmail.com> wrote:
> On Tue, Apr 16, 2013 at 11:55 PM, Sander <sander@humilis.net> wrote:
>> Matt Pursley wrote (ao):
>>> I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives
>>> attached.  When it's formated with mdraid6+ext4 I get about 1200MB/s
>>> for multiple streaming random reads with iozone.  With btrfs in
>>> 3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a
>>> time.
>>
>> Just curious, is that btrfs on top of mdraid6, or is this experimental
>> btrfs raid6 without md?
>
>
>
> This is the "experimental btrfs raid6 without md".
>
> But, I did do a "mdraid6 with btrfs" test last night... and with that
> setup I only get the ~750MB/s result.. even with just one
> thread/stream...
>
> I will flip the system back to "btrfsraid6+btrfs" today to verify that
> I still get the full 1200MB/s with one stream/thread and ~750MB/s with
> two or more streams/threads with that setup...
>
>
> Thanks,
> Matt
>
>
>
> ___
> mdraid6+btrfs_64GBRam_80files # sysctl vm.drop_caches=1 ; dd
> of=/dev/null if=/var/data/persist/testfile bs=640k
> vm.drop_caches = 1
> 20000+0 records in
> 20000+0 records out
> 13107200000 bytes (13 GB) copied, 18.2109 s, 720 MB/s
> ___

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Josef Bacik]

On Wed, Apr 17, 2013 at 07:50:09PM -0600, Matt Pursley wrote:
> Hey All,
> 
> Here are the results of making and reading back a 13GB file on
> "mdraid6 + ext4",  "mdraid6 + btrfs", and "btrfsraid6 + btrfs".
> 
> Seems to show that:
> 1) "mdraid6 + ext4" can do ~1100 MB/s for these sequential reads with
> either one or two files at once.
> 2) "btrfsraid6 + btrfs" can do ~1100 MB/s for sequential reads with
> one file at a time, but only ~750 MB/s with two (or more).
> 3) "mdraid6 + btrfs" can only do ~750 MB/s for these sequential reads
> with either one or two files at once.
> 
> 
> So, seems like the speed drop is related more to the btrfs files
> system, then the experimental raid.
> Although it is interesting that btrfs can only do the full ~1100 MB/s
> with a single file on the btrfsraid6, but not mdraid6.
> 
> 
> Anyway, just some more info and reproducible results.
> 
> I have also opened a ticket in bugzilla.kernel.org for this issue here...
> https://bugzilla.kernel.org/show_bug.cgi?id=56771
>

This is strange, and I can't see any reason why this would happen.  I'll try and
reproduce next week when I'm back from LSF.  Thanks,

Josef 

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Hey Josef,

Were you able to look into this any further?
It's still pretty reproducible on my machine...


Thanks,
Matt





On Thu, Apr 18, 2013 at 2:58 PM, Josef Bacik <jbacik@fusionio.com> wrote:

> This is strange, and I can't see any reason why this would happen.  I'll try and
> reproduce next week when I'm back from LSF.  Thanks,
>
> Josef

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Josef Bacik]

On Thu, Apr 25, 2013 at 01:52:44PM -0600, Matt Pursley wrote:
> Hey Josef,
> 
> Were you able to look into this any further?
> It's still pretty reproducible on my machine...
> 

Nope I've been tracking down random problems, I'll try it now.  Thanks,

Josef

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Ok, awesome, let me know how it goes..  I don't have the raid
formatted to btrfs right now, but I could probably do that in about 30
minutes or so.


Thanks Josef,
Matt




On Thu, Apr 25, 2013 at 1:39 PM, Josef Bacik <jbacik@fusionio.com> wrote:
> On Thu, Apr 25, 2013 at 01:52:44PM -0600, Matt Pursley wrote:
>> Hey Josef,
>>
>> Were you able to look into this any further?
>> It's still pretty reproducible on my machine...
>>
>
> Nope I've been tracking down random problems, I'll try it now.  Thanks,
>
> Josef

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Josef Bacik]

On Thu, Apr 25, 2013 at 03:01:18PM -0600, Matt Pursley wrote:
> Ok, awesome, let me know how it goes..  I don't have the raid
> formatted to btrfs right now, but I could probably do that in about 30
> minutes or so.
> 

Huh so I'm getting the full bandwidth, 120 mb/s with one thread and 60 mb/s with
two threads.  These are just cheap sata drives tho, I'll try and dig up a box
with 3 fusion cards for something a little closer to the speeds you are seeing
and see if that makes a difference.  Thanks,

Josef

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Hey Josef,

Thanks for looking into this further!    That is about the same
results that I was seeing, though I didn't test it with just one
drive.. only with all 12 drives in my jbod.  I will do a test with
just one disk, and see if I also get the same results.

Let me know if you also see the same results with multiple drives in
your raid...


Thanks,
Matt





On Thu, Apr 25, 2013 at 2:10 PM, Josef Bacik <jbacik@fusionio.com> wrote:
> On Thu, Apr 25, 2013 at 03:01:18PM -0600, Matt Pursley wrote:
>> Ok, awesome, let me know how it goes..  I don't have the raid
>> formatted to btrfs right now, but I could probably do that in about 30
>> minutes or so.
>>
>
> Huh so I'm getting the full bandwidth, 120 mb/s with one thread and 60 mb/s with
> two threads.  These are just cheap sata drives tho, I'll try and dig up a box
> with 3 fusion cards for something a little closer to the speeds you are seeing
> and see if that makes a difference.  Thanks,
>
> Josef

Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?

[Matt Pursley]

Hey Josef,

Were you able to try this multi-thread test on any more drives?


I did a test with 12, 6, 3, and 1 drive.  And, it looks like I see the
multi-thread speed reduces, as the number of drives in the raid goes
up.

Like this:
- 50% speed reduction with 2 threads on 12 drives
- 25% speed reduction with 2 threads on 6 drives
- 10% speed reduction with 2 threads on 3 drives
- 5% speed reduction with 2 threads on 1 drive



I only have 12 slots on my HBA card, but I wonder if 24 drives would
reduce the speed to 25% with 2 threads?

Matt










make btrfs fs...
___

12 drives...
mkfs.btrfs -f -d raid6 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
/dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl

6 drives...
mkfs.btrfs -f -d raid6 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

3 drives...
mkfs.btrfs -f -d raid5 /dev/sda /dev/sdb /dev/sdc

1 drive...
mkfs.btrfs -f /dev/sda

mount /dev/sda /tmp/btrfs_test/

___


make zero files...
___
kura1 ~ # for j in {1..2} ; do dd if=/dev/zero
of=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M count=10000
conv=fdatasync & done
___


===================

btrfs raid6 on 12 drives with 2 threads = ~650MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
vm.drop_caches = 1
10485760000 bytes (10 GB) copied, 31.0431 s, 338 MB/s
10485760000 bytes (10 GB) copied, 31.2235 s, 336 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 29.869 s, 351 MB/s
10485760000 bytes (10 GB) copied, 30.5561 s, 343 MB/s

___


btrfs raid6 on 12 drives with 1 thread = ~1100MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 9.69881 s, 1.1 GB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 9.56475 s, 1.1 GB/s
___


==================

btrfs raid6 on 6 drives with 2 thread =  ~500MB/s
___
 kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 41.3899 s, 253 MB/s
10485760000 bytes (10 GB) copied, 41.6916 s, 252 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 40.3178 s, 260 MB/s
10485760000 bytes (10 GB) copied, 41.4087 s, 253 MB/s

___



btrfs raid6 on 6 drives with 1 thread =  ~600MB/s
___

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 17.5686 s, 597 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 17.5396 s, 598 MB/s
___


==================

btrfs raid5 on 3 drives with 2 thread = ~300MB/s
___

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 67.636 s, 155 MB/s
10485760000 bytes (10 GB) copied, 70.1783 s, 149 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 69.4945 s, 151 MB/s
10485760000 bytes (10 GB) copied, 70.8279 s, 148 MB/s

___



btrfs raid5 on 3 drives with 1 thread = ~319MB/s
___

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 32.8559 s, 319 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 32.8483 s, 319 MB/s

___


==================


btrfs (no raid) on 1 drive with 2 thread =  ~155MB/s
___
 kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 134.982 s, 77.7 MB/s
10485760000 bytes (10 GB) copied, 135.237 s, 77.5 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 134.549 s, 77.9 MB/s
10485760000 bytes (10 GB) copied, 135.293 s, 77.5 MB/s


___


btrfs (no raid) on 1 drive with 1 thread =  ~162MB/s
___

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 64.5931 s, 162 MB/s

kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M &
done
10485760000 bytes (10 GB) copied, 64.6299 s, 162 MB/s

___



==================







On Fri, Apr 26, 2013 at 4:21 PM, Matt Pursley <mpursley@gmail.com> wrote:
> Hey Josef,
>
> Thanks for looking into this further!    That is about the same
> results that I was seeing, though I didn't test it with just one
> drive.. only with all 12 drives in my jbod.  I will do a test with
> just one disk, and see if I also get the same results.
>
> Let me know if you also see the same results with multiple drives in
> your raid...
>
>
> Thanks,
> Matt
>
>
>
>
>
> On Thu, Apr 25, 2013 at 2:10 PM, Josef Bacik <jbacik@fusionio.com> wrote:
>> On Thu, Apr 25, 2013 at 03:01:18PM -0600, Matt Pursley wrote:
>>> Ok, awesome, let me know how it goes..  I don't have the raid
>>> formatted to btrfs right now, but I could probably do that in about 30
>>> minutes or so.
>>>
>>
>> Huh so I'm getting the full bandwidth, 120 mb/s with one thread and 60 mb/s with
>> two threads.  These are just cheap sata drives tho, I'll try and dig up a box
>> with 3 fusion cards for something a little closer to the speeds you are seeing
>> and see if that makes a difference.  Thanks,
>>
>> Josef