Hi,

on my 4-node system (SSD + 10GbE, see bench-config.txt for details)
I can observe a pretty nice rados bench performance 
(see bench-rados.txt for details):

Bandwidth (MB/sec):     961.710 
Max bandwidth (MB/sec): 1040
Min bandwidth (MB/sec): 772


Also the bandwidth performance generated with
  fio --filename=/dev/rbd1 --direct=1 --rw=$io --bs=$bs --size=2G --iodepth=$threads --ioengine=libaio --runtime=60 --group_reporting --name=file1 --output=fio_${io}_${bs}_${threads}

.... is acceptable, e.g.
fio_write_4m_16		795 MB/s
fio_randwrite_8m_128	717 MB/s
fio_randwrite_8m_16	714 MB/s
fio_randwrite_2m_32	692 MB/s


But, the write IOPS seems to be limited around 19k ...
RBD                     4M      64k (= optimal_io_size)
fio_randread_512_128    53286   55925
fio_randread_4k_128     51110   44382
fio_randread_8k_128     30854   29938
fio_randwrite_512_128   18888    2386
fio_randwrite_512_64    18844    2582
fio_randwrite_8k_64     17350    2445
(...)
fio_read_4k_128         10073   53151
fio_read_4k_64           9500   39757
fio_read_4k_32           9220   23650
(...)
fio_read_4k_16           9122   14322
fio_write_4k_128         2190   14306
fio_read_8k_32            706   13894
fio_write_4k_64          2197   12297
fio_write_8k_64          3563   11705
fio_write_8k_128         3444   11219


Any hints for tuning the IOPS (read and/or write) would be appreciated.

How can I set the variables when the Journal data have go to the OSD ? (after X seconds and/or when Y %-full)


Kind Regards,
-Dieter