[performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

["hzwulibin" <hzwulibin@gmail.com>]

Hi, 
same environment, after a test script, the io latency(get from sudo ceph --admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) 
increase from about 4ms to 7.3ms

qemu version: debian 2.1.2
kernel:3.10.45-openstack-amd64
system: debian 7.8
ceph: 0.94.5
VM CPU number: 4  (cpu MHz : 2599.998)
VM memory size: 16GB
9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.

Test scripts in VM:
# cat reproduce.sh
#!/bin/bash

times=20
for((i=1;i<=$times;i++))
do
    tmpdate=`date "+%F-%T"`
    echo "=======================$tmpdate($i/$times)======================="
    tmp=$((i%2))
    if [[ $tmp -eq 0 ]];then
        echo "############### fio /root/vdb.cfg ###############"
        fio /root/vdb.cfg
    else
        echo "############### fio /root/vdc.cfg ###############"
        fio /root/vdc.cfg
    fi
done


tmpdate=`date "+%F-%T"`
echo "############### [$tmpdate] fio /root/vde.cfg ###############"
fio /root/vde.cfg


# cat vdb.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdb


# cat vdc.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdc

# cat vdd.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdd

# cat vde.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vde 		

After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from 12k to 5k, the latency increase from 4ms to 7.3ms.

run steps:
1. create a VM
2. create four volumes and attatch to the VM
3. sh reproduce.sh
4. in the runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the performance
	
After reproduce.sh finished, performance down.


Anyone has the same problem or has some ideas about this?

Thanks!
--------------
hzwulibin
2015-11-02