IB perf test questions

IB perf test questions

[Tom Ammon]

  Hi,

I have done some benchmarking on a QDR fabric, and I wonder if someone 
could help me with a few questions.

Using ib_read_bw between 2 QDR nodes with a single switch between them, 
I get the following results:

[root@taildrop ~]# ib_write_bw -m 4096 -a -n 10000 155.101.5.4 -q 4
------------------------------------------------------------------
                     RDMA_Write BW Test
Number of qp's running 4
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 0 bytes message
  local address: LID 0x04 QPN 0x14004e PSN 0x94bcad RKey 0x48042000 
VAddr 0x002b8105a27000
  local address: LID 0x04 QPN 0x14004f PSN 0x1b2d93 RKey 0x48042000 
VAddr 0x002b8105a27000
  local address: LID 0x04 QPN 0x140050 PSN 0x79fae1 RKey 0x48042000 
VAddr 0x002b8105a27000
  local address: LID 0x04 QPN 0x140051 PSN 0xe8971c RKey 0x48042000 
VAddr 0x002b8105a27000
  remote address: LID 0x03 QPN 0x54004e PSN 0x2baff9 RKey 0x50042000 
VAddr 0x002ac8321f7000
  remote address: LID 0x03 QPN 0x54004f PSN 0x7d8026 RKey 0x50042000 
VAddr 0x002ac8321f7000
  remote address: LID 0x03 QPN 0x540050 PSN 0x94c242 RKey 0x50042000 
VAddr 0x002ac8321f7000
  remote address: LID 0x03 QPN 0x540051 PSN 0x662e32 RKey 0x50042000 
VAddr 0x002ac8321f7000
Mtu : 4096
------------------------------------------------------------------
  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
       2        10000               7.24                  7.23
       4        10000              14.55                 14.54
       8        10000              29.10                 29.07
      16        10000              58.37                 57.89
      32        10000             114.54                114.40
      64        10000             231.57                231.39
     128        10000             443.32                435.70
     256        10000             805.18                788.10
     512        10000             797.66                789.05
    1024        10000             790.05                788.34
    2048        10000            2804.49               2802.98
    4096        10000            2863.22               2862.29
    8192        10000            2895.43               2895.26
   16384        10000            2917.64               2917.42
   32768        10000            2924.17               2924.13
   65536        10000            2927.74               2927.73
  131072        10000            2928.85               2928.84
  262144        10000            2929.42               2929.41
  524288        10000            2929.78               2929.77
1048576        10000            2929.95               2929.95
2097152        10000            2930.02               2930.01
4194304        10000            2929.87               2929.87
8388608        10000            2929.78               2929.78
------------------------------------------------------------------


2930 MB/s = ~23 Gb/s. This seems reasonably close to the QDR line speed 
of 32Gb/s (not counting encoding overhead). The nodes are pretty new 
Dell c6100s with 4 x 4 Intel X5560 (2.8GHz) and 12GB of RAM. Does this 
look like a normal number for this kind of machine? Or am I missing 
something obvious to make it perform better?

Also, whether I use ib_read_bw or ib_write_bw, the machine I initiate 
the test from (in this case "taildrop") shows one of its CPU cores 
pegged at 100% for the duration of the test, but I see no CPU 
utilization at all on the receiving node. Can someone explain to me 
what's going on under the hood, here? I would think that read_bw would 
load up the sending host but that write_bw would load up the receiving 
host (or maybe vice versa), so this seems counterintuitive to me. when I 
use the -b flag to do a bidirectional test, a single CPU core on both 
machines pegs at 100%.

Thanks,

Tom

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB perf test questions

[Jason Gunthorpe]

On Wed, Jul 14, 2010 at 02:28:15PM -0600, Tom Ammon wrote:

> Also, whether I use ib_read_bw or ib_write_bw, the machine I initiate 
> the test from (in this case "taildrop") shows one of its CPU cores 
> pegged at 100% for the duration of the test, but I see no CPU 
> utilization at all on the receiving node. Can someone explain to me 
> what's going on under the hood, here? I would think that read_bw would 
> load up the sending host but that write_bw would load up the receiving 
> host (or maybe vice versa), so this seems counterintuitive to me. when I 
> use the -b flag to do a bidirectional test, a single CPU core on both 
> machines pegs at 100%.

In all cases the master machine sits in a CPU bound loop waiting for
completions so it can issue more RDMA operations. The difference
between write and read is simply the RDMA op that is issued.

The slave side just sits there and the NIC does all the work.

Bi-directional mode runs a master operation on both sides..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: IB perf test questions

[Boris Shpolyansky]

Regarding the BW numbers: they look reasonable. You can try to improve the BW up to 3.2GB/s by increasing PCIe MTU from 128 to 256 byte. It usually requires BIOS configuration changes.

Boris Shpolyansky
Sr. Member of Technical Staff, Applications
 
Mellanox Technologies Inc.
350 Oakmead Parkway, Suite 100
Sunnyvale, CA 94085
Tel.: (408) 916 0014
Fax: (408) 585 0314
Cell: (408) 834 9365
www.mellanox.com
Mellanox on Twitter and Facebook

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Jason Gunthorpe
Sent: Wednesday, July 14, 2010 2:17 PM
To: Tom Ammon
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: IB perf test questions

On Wed, Jul 14, 2010 at 02:28:15PM -0600, Tom Ammon wrote:

> Also, whether I use ib_read_bw or ib_write_bw, the machine I initiate 
> the test from (in this case "taildrop") shows one of its CPU cores 
> pegged at 100% for the duration of the test, but I see no CPU 
> utilization at all on the receiving node. Can someone explain to me 
> what's going on under the hood, here? I would think that read_bw would 
> load up the sending host but that write_bw would load up the receiving 
> host (or maybe vice versa), so this seems counterintuitive to me. when I 
> use the -b flag to do a bidirectional test, a single CPU core on both 
> machines pegs at 100%.

In all cases the master machine sits in a CPU bound loop waiting for
completions so it can issue more RDMA operations. The difference
between write and read is simply the RDMA op that is issued.

The slave side just sits there and the NIC does all the work.

Bi-directional mode runs a master operation on both sides..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html