Re[2]: xio messenger prelim benchmark

Re[2]: xio messenger prelim benchmark

[Vu Pham]

>Hi,
>
>I'm going to use same mellanox switchs in production next month,
>
>Isn't them ethernet switchs ?
>How xio messenger is working, if they are not infiband ?
>
>Maybe with RoCE ?
>

Yes, xio messenger which is implemented over Accelio can run over rdma 
transport (Infiniband, RoCE) and TCP. Please note that we have not 
enabled xio messenger / Accelio-tcp yet.

>
>I'll try to benchmark xio messenger too.
>
Great. Let us know.

>
>
>BTW, do you have tried to bench with fio + krbd instead fio-librbd ?
>I think you should have better results with numjobs=1.
>
>
xio messenger is currently working with user mode clients. We only 
validated/tested user mode rbd client.
Sandisk are working on krbd over kAccelio implementation. As of last 
week, SanDisk have basic I/Os working.
Hopefully krbd/kAccelio will be available soon.

@Mark, I'll try out disabling auth, in-memory debugging & RHEL.

-vu

Re: Re[2]: xio messenger prelim benchmark

[Alexandre DERUMIER]

>>Yes, xio messenger which is implemented over Accelio can run over rdma 
>>transport (Infiniband, RoCE) and TCP. Please note that we have not 
>>enabled xio messenger / Accelio-tcp yet. 

Oh, ok, Great !


>>xio messenger is currently working with user mode clients. We only 
>>validated/tested user mode rbd client. 
>>Sandisk are working on krbd over kAccelio implementation. As of last 
>>week, SanDisk have basic I/Os working. 
>>Hopefully krbd/kAccelio will be available soon. 

Great, so we can expect even better results :)


BTW, do you have some client side cpu usage benchmark ?



----- Mail original -----
De: "Vu Pham" <vuhuong@mellanox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Jeudi 5 Février 2015 19:15:21
Objet: Re[2]: xio messenger prelim benchmark

>Hi, 
> 
>I'm going to use same mellanox switchs in production next month, 
> 
>Isn't them ethernet switchs ? 
>How xio messenger is working, if they are not infiband ? 
> 
>Maybe with RoCE ? 
> 

Yes, xio messenger which is implemented over Accelio can run over rdma 
transport (Infiniband, RoCE) and TCP. Please note that we have not 
enabled xio messenger / Accelio-tcp yet. 

> 
>I'll try to benchmark xio messenger too. 
> 
Great. Let us know. 

> 
> 
>BTW, do you have tried to bench with fio + krbd instead fio-librbd ? 
>I think you should have better results with numjobs=1. 
> 
> 
xio messenger is currently working with user mode clients. We only 
validated/tested user mode rbd client. 
Sandisk are working on krbd over kAccelio implementation. As of last 
week, SanDisk have basic I/Os working. 
Hopefully krbd/kAccelio will be available soon. 

@Mark, I'll try out disabling auth, in-memory debugging & RHEL. 

-vu 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: xio messenger prelim benchmark

[Vu Pham]

>>>Yes, xio messenger which is implemented over Accelio can run over 
>>>rdma
>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>enabled xio messenger / Accelio-tcp yet.
>
>Oh, ok, Great !
>
>
>>>xio messenger is currently working with user mode clients. We only
>>>validated/tested user mode rbd client.
>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>week, SanDisk have basic I/Os working.
>>>Hopefully krbd/kAccelio will be available soon.
>
>Great, so we can expect even better results :)
>
>
>BTW, do you have some client side cpu usage benchmark ?
>
>
Sorry, I did not collect the cpu usage @client

BTW since you will use Mellanox sx1012 EN switch for your benchmark, you 
can take advantage of 56Gbe/s over 40Gbe/s with a license upgrade.

With my minimum setup of single node cluster (single OSD, 2 
pools/images) + 1 client node with 16 fio_rbd client streams, we already 
see the benefit of big pipe.

             ib_send_bw     64k read (ceph)      256k read(ceph)
---------------------------------------------------------------------------------------
40Gbe: 4350 MB/s       4300 MB/s             4350 MB/s
56Gbe: 5100 MB/s       4850 MB/s             5000 MB/s

-vu

Re: xio messenger prelim benchmark

[Vu Pham]

>
>>>>Yes, xio messenger which is implemented over Accelio can run over
>>>>rdma
>>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>>enabled xio messenger / Accelio-tcp yet.
>>
>>Oh, ok, Great !
>>
>>
>>>>xio messenger is currently working with user mode clients. We only
>>>>validated/tested user mode rbd client.
>>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>>week, SanDisk have basic I/Os working.
>>>>Hopefully krbd/kAccelio will be available soon.
>>
>>Great, so we can expect even better results :)
>>
>>
>>BTW, do you have some client side cpu usage benchmark ?
>>
>>
>Sorry, I did not collect the cpu usage @client
>

I just rerun the test with 2 OSDs, 4 fio_clients on 4 separated 
pools/images, each client uses numjobs=1 for random write test and 
numjobs=8 for random read test. I also measured the cpu usage@client 
this time

xio messenger:
--------------------
. 4K random write: ~26k iops (10 cores used at OSD, 2 cores used at 
client)
. 4K random read: ~332k iops (34 cores used at OSD, 4 cores used at 
client)
. 256K random write: ~1464 MB/s (6 cores used at OSD, 2 cores used at 
client)
. 256K random read: ~4300 MB/s (6 cores used at OSD, 5 cores used at 
client)

simple:
----------
. 4K random write: ~25k iops (10 cores used at OSD, 3 cores used at 
client)
. 4K random read: ~230k iops (38 cores used at OSD, 30 cores used at 
client)
. 256K random write: ~1475 MB/s (6 cores used at OSD, 2 cores used at 
client)
. 256K random read: ~4300 MB/s (7 cores used at OSD, 7 cores used at 
client)

-vu

Re: xio messenger prelim benchmark

[Vu Pham]

On 2/6/2015 10:41:15 AM, "Vu Pham" <vuhuong@mellanox.com> wrote:

>
>>>>Yes, xio messenger which is implemented over Accelio can run over
>>>>rdma
>>>>transport (Infiniband, RoCE) and TCP. Please note that we have not
>>>>enabled xio messenger / Accelio-tcp yet.
>>
>>Oh, ok, Great !
>>
>>
>>>>xio messenger is currently working with user mode clients. We only
>>>>validated/tested user mode rbd client.
>>>>Sandisk are working on krbd over kAccelio implementation. As of last
>>>>week, SanDisk have basic I/Os working.
>>>>Hopefully krbd/kAccelio will be available soon.
>>
>>Great, so we can expect even better results :)
>>
>>
>>BTW, do you have some client side cpu usage benchmark ?
>>
>>
>Sorry, I did not collect the cpu usage @client
>
>BTW since you will use Mellanox sx1012 EN switch for your benchmark, 
>you
>can take advantage of 56Gbe/s over 40Gbe/s with a license upgrade.
>
>With my minimum setup of single node cluster (single OSD, 2
>pools/images) + 1 client node with 16 fio_rbd client streams, we 
>already
>see the benefit of big pipe.
>
>              ib_send_bw 64k read (ceph) 256k read(ceph)
>---------------------------------------------------------------------------------------
>40Gbe: 4350 MB/s 4300 MB/s 4350 MB/s
>56Gbe: 5100 MB/s 4850 MB/s 5000 MB/s
>

On raw data (ib_send_bw), I get only 5100 MB/s on 56Gbe/s link because 
I'm using default mtu=1500.
There is RoCE header overhead per frame and which RoCE's mtu is 
selected/used (RoCE's mtu is 512-bytes in this case)

With jumbo frames (RoCE's mtu can be 2K or 4K), the raw bw can get to 
~50Gb/s

-vu