poor OSD performance using kernel 3.4

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Hi list,

today while testing btrfs i discovered a very poor osd performance using
kernel 3.4.

Underlying FS is XFS but it is the same with btrfs.

3.0.30:
~# rados -p data bench 10 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 10 seconds.
  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16        41        25   99.9767       100  0.586984  0.447293
    2      16        71        55   109.979       120  0.934388  0.488375
    3      16        99        83   110.647       112   1.15982  0.503111
    4      16       130       114   113.981       124   1.05952  0.516925
    5      16       159       143   114.382       116  0.149313  0.510734
    6      16       188       172   114.649       116  0.287166   0.52203
    7      16       215       199   113.697       108  0.151784  0.531461
    8      16       242       226   112.984       108  0.623478  0.539896
    9      16       265       249   110.651        92   0.50354  0.538504
   10      16       296       280   111.984       124  0.155048  0.542846
Total time run:        10.776153
Total writes made:     297
Write size:            4194304
Bandwidth (MB/sec):    110.243

Average Latency:       0.577534
Max latency:           1.85499
Min latency:           0.091473


3.4:
~# rados -p data bench 10 write -t 16
Maintaining 16 concurrent writes of 4194304 bytes for at least 10 seconds.
  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
    0       0         0         0         0         0         -         0
    1      16        40        24   95.9794        96  0.393196  0.455936
    2      16        68        52   103.983       112  0.835652  0.517297
    3      16        85        69   91.9849        68   1.00535  0.493058
    4      16        96        80   79.9869        44  0.096564  0.577948
    5      16       103        87   69.5879        28  0.092722  0.589147
    6      16       117       101   67.3216        56  0.222175  0.675334
    7      16       130       114   65.1321        52   0.15677  0.623806
    8      16       144       128   63.9896        56  0.089157   0.56746
    9      16       144       128   56.8794         0         -   0.56746
   10      16       144       128   51.1912         0         -   0.56746
   11      16       144       128   46.5373         0         -   0.56746
   12      16       144       128   42.6591         0         -   0.56746
   13      16       144       128   39.3776         0         -   0.56746
   14      16       144       128   36.5649         0         -   0.56746
   15      16       144       128   34.1272         0         -   0.56746
   16      16       145       129   32.2443       0.5   11.3422  0.650985
Total time run:        16.193871
Total writes made:     145
Write size:            4194304
Bandwidth (MB/sec):    35.816

Average Latency:       1.78467
Max latency:           14.4744
Min latency:           0.088753

Stefan