rdma performance verification

["Liu, Changcheng" <changcheng.liu@intel.com>]

Hi all,
   I'm working on using rdma to improve message transaction performance
   on distributed storage system(Ceph) development.

   Does anyone know what's the right tool to compare RDMA vs TCP peformance?
   Such as bandwidth, latency. Especially the tool that could measure
   the time to transact the same data size.

   Previously, I use iperf & ib_send_bw to do test(send same data size).
   However, it shows that ib_send_bw use more time to send data than
   iperf.
      nstcc1@nstcloudcc1:~$ time ib_send_bw -c RC -d rocep4s0 -i 1 -p 18515 -q 1 -r 4096 -t 1024 -s 1073741824 --report_gbits -F 192.168.199.222
      real    3m53.858s
      user    3m48.456s
      sys     0m5.318s
      nstcc1@nstcloudcc1:~$ time iperf -c 192.168.199.222 -p 8976 -n 1073741824 -P 1
      real    0m1.688s
      user    0m0.020s
      sys     0m1.644s

   In Ceph, the result shows that rdma performance (RC transaction type,
   SEDN operation) is worse or not much better than TCP implemented performance.
   Test A:
      1 client thread send 20GB data to 1 server thread (marked as 1C:1S)
   Result:
      1) implementation based on RDMA
         Take 171.921294s to finish send 20GB data.
      2) implementation based on TCP
         Take 62.444163s to finish send 20GB data.

   Test B:
      16 client threads send 16x20GB data to 1 server thread (marked as 16C:1S)
   Result:
      1) implementation base on RDMA
         Take 261.285612s to finish send 16x20GB data.
      2) implementation based on TCP
         Take 318.949126 to finish send 16x20GB data.

B.R.
Changcheng