Re: rdma_lat whos

[Or Gerlitz <ogerlitz-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>]

Or Gerlitz wrote:
> Now, Jack, using this patch, 
> I could get rdma_lat which doesn't use the rdma-cm, which means setting all the low-level QP params 
> "by the hand" to produce the SAME result of 3.9us as with the rdma-cm, as you can see its one liner 
> patch which uses higher MTU of 2048 vs the hard coded MTU of 256 used in the code. This is quite counter 
> intuitive, for packets whose size is > 256, correct? is there any known issue that can 
> explain that?! The SA is convinced that 2048 (0x84) is the best MTU for that path, both nodes 
> have ConnectX DDR with firmware 2.7.0 

running with mtu=256 vs mtu=2048 on various msg size, you can see this phenomena in action
2.70us for 1k msg with LOWER mtu vs 3.54us with mtu=2048

> # ib_send_lat -a -m 256 celery
> ------------------------------------------------------------------
>                     Send Latency Test
> Inline data is used up to 400 bytes message
> Connection type : RC
>    local address: LID 0x04 QPN 0x4c004b PSN 0xf91b6e
>   remote address: LID 0x04 QPN 0x4c004a PSN 0xb9e19a
> Mtu : 256
> ------------------------------------------------------------------
>  #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
>       2        1000           1.04          10.33             1.06
>       4        1000           1.03           9.80             1.07
>       8        1000           1.03           7.89             1.07
>      16        1000           1.05           7.81             1.09
>      32        1000           1.09           7.77             1.11
>      64        1000           1.18           7.99             1.21
>     128        1000           1.48           8.05             1.51
>     256        1000           2.06           8.67             2.19
>     512        1000           2.31          14.38             2.38
>    1024        1000           2.62          14.54             2.70
>    2048        1000           3.26          14.90             3.36
>    4096        1000           4.54          28.84             4.64
>    8192        1000           7.19          18.48             7.30
>   16384        1000          12.61          24.15            12.75
>   32768        1000          23.46          34.77            24.30
>   65536        1000          45.19          56.48            45.75
>  131072        1000          88.64         102.99            89.22
>  262144        1000         175.78         186.59           175.97
>  524288        1000         349.16         359.92           349.35
> 1048576        1000         695.80         706.90           696.22
> 2097152        1000        1389.41        1400.87          1389.95
> 4194304        1000        2777.21        2799.41          2778.08
> 8388608        1000        5691.77        5752.48          5737.45
> ------------------------------------------------------------------
> All resources were Released successfully

> # ib_send_lat -a -m 2048 celery
> ------------------------------------------------------------------
>                     Send Latency Test
> Inline data is used up to 400 bytes message
> Connection type : RC
>    local address: LID 0x04 QPN 0x54004b PSN 0x59384f
>   remote address: LID 0x04 QPN 0x54004a PSN 0xcac60e
> Mtu : 2048
> ------------------------------------------------------------------
>  #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
>       2        1000           1.02          10.66             1.06
>       4        1000           1.03          54.53             1.06
>       8        1000           1.04          36.84             1.06
>      16        1000           1.05          34.58             1.08
>      32        1000           1.08          40.70             1.11
>      64        1000           1.18          66.86             1.21
>     128        1000           1.48          38.76             1.51
>     256        1000           2.07          39.15             2.18
>     512        1000           2.58          45.90             2.67
>    1024        1000           3.46          38.07             3.54
>    2048        1000           5.22          40.55             5.31
>    4096        1000           6.53          39.17             6.61
>    8192        1000           9.16          38.85             9.29
>   16384        1000          14.52          50.14            14.99
>   32768        1000          25.54          51.20            25.64
>   65536        1000          47.18          59.76            47.30
>  131072        1000          90.87          96.92            90.99
>  262144        1000         178.22         183.27           178.37
>  524288        1000         352.93         358.01           353.13
> 1048576        1000         702.37         707.69           702.66
> 2097152        1000        1401.30        1411.63          1401.66
> 4194304        1000        2799.16        2845.59          2801.92
> 8388608        1000        5833.30        5911.70          5886.55
> ------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html