Performance degrad on big cluster

Performance degrad on big cluster

[Ketor D]

Hi all,
      We meet a performance degrad in one of our cluster. Our
randwrite latency degraded from 1ms to 5ms(fio -ioengine=rbd
iodepth=1)
      The cluster has about 200 osds runing on Intel 3500 SSD, we run
both qemu and ceph-osd on the hosts. The network for ceph is 10GbE.
       While the cluster is smaller and not so many qemu processes,
the IO latency is about 1ms, but now the latency is 5ms.
       I use strace to  get the time for syscall, all syscall (writev,
io_submit, recvfrom,sendmsg,lseek,fgetxattr etc.)  use 300us to 600us.
The syscall time on a small and idle cluster is near to 0us.
       After checked serval clusters, I come to a conclusion:
       num_of_osds     num_of_threads_on_host    time_of_syscall(us)
       200                     10000                                   300-600
       100                       5000                                   200-500
        70                        2500                                   100-300
         9                           750
      20-60

       The threads on one of 200 osds cluster's host is like this:
       name           num_of_processes  num_of_threads
num_of_threads_per_process
       qemu-kvm    49                              9748
        198
       ceph-osd      6                                5707
           951

       We are running Firefly 0.80.7 and 0.80.9, and qemu version is 2.1.

       I think too many threads on the host lead to high latency of
ceph-osd process, and cause to high I/O latency from the client-side.
       Anyone's help is welcome.

Thanks!

Performance degrad on big cluster

[juncheng bai]

Hi all,
    We meet a performance degrad in one of our cluster. Our
randwrite latency degraded from 1ms to 5ms(fio -ioengine=rbd
iodepth=1)
    The cluster has about 200 osds runing on Intel 3500 SSD, we run
both qemu and ceph-osd on the hosts. The network for ceph is 10GbE.
    While the cluster is smaller and not so many qemu processes,
the IO latency is about 1ms, but now the latency is 5ms.
    I use strace to  get the time for syscall, all syscall (writev,
io_sub mit, recvfrom,sendmsg,lseek,fgetxattr etc.)  use 300us to 600us.
The syscall time on a small and idle cluster is near to 0us.
    After checked serval clusters, I come to a conclusion:
    num_of_osds     num_of_threads_on_host    time_of_syscall(us)
    200                       10000             300-600
    100                       5000              200-500
    70                        2500              100-300
    9                         750               20-60

   The threads on one of 200 osds cluster's host is like this:
   name num_of_processes  num_of_threads  num_of_threads_per_process
   qemu-kvm    49                9748            198
   ceph-osd    6                 5707            951

   I think too many threads on the host lead to high latency of ceph-osd 
process, and cause to high I/O latency from the client-side.

   Anyone's help is welcome.

Thanks!