Re: poor OSD performance using kernel 3.4

[Mark Nelson <mark.nelson@inktank.com>]

On 05/29/2012 04:08 PM, Stefan Priebe wrote:
> Am 29.05.2012 19:50, schrieb Mark Nelson:
>> I did some quick tests on a couple of nodes I had laying around this
>> morning.
>
> I just noticed that i get a constant rate of 40MB/s while using 1 
> thread. When i use two thread or more i get drop to 0MB/s and crazy 
> jumping values.
>
> ~# rados -p rbd bench 90 write -t 1
> Maintaining 1 concurrent writes of 4194304 bytes for at least 90 seconds.
>   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>     0       0         0         0         0         0         -         0
>     1       1        10         9    35.994        36  0.100147  0.101133
>     2       1        20        19   37.9931        40  0.096893  0.100719
>     3       1        31        30   39.9921        44   0.09784 0.0999607
>     4       1        41        40   39.9929        40  0.099156 0.0999003
>     5       1        51        50   39.9932        40  0.098239 0.0996518
>     6       1        61        60   39.9932        40  0.098682 0.0994851
>     7       1        71        70   39.9933        40  0.094397  0.099184
>     8       1        81        80   39.9931        40  0.099823 0.0993327
>     9       1        91        90   39.9931        40  0.101013 0.0992236
>    10       1       101       100    39.993        40  0.098277  0.099237
>
>

When you are using 1 thread, you are hitting a ~40MB/s limit (probably 
networking related) before the data gets to the journal.  Because (in 
this case) the filestore data disk can handle that throughput, 
everything looks nice and consistent.

>
> # rados -p rbd bench 90 write -t 2
> Maintaining 2 concurrent writes of 4194304 bytes for at least 90 seconds.
>   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>     0       0         0         0         0         0         -         0
>     1       2        15        13   51.9888        52    0.0956  0.115315
>     2       2        22        20   39.9928        28  0.120065  0.193125
>     3       2        41        39   51.9917        76   0.09557   0.15246
>     4       2        58        56   55.9912        68   0.09875  0.137688
>     5       2        67        65    51.992        36  0.111211  0.139465
>     6       2        85        83   55.3251        72  0.136967  0.143079
>     7       2       101        99   56.5625        64  0.098664  0.136263
>     8       2       101        99   49.4919         0         -  0.136263
>     9       2       112       110   48.8808        22  0.099479  0.160563
>

In this case, that 40MB/s limit with 1 thread has increased.  Now more 
data is getting fed into the journal than the filestore can write out to 
disk.  Eventually writes stall while the data is being written out.
> Stefan