Re: Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky <andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org>]

[-- Attachment #1.1: Type: text/plain, Size: 4393 bytes --]

Somnath, 

Sounds very promising! I can't wait to try it on my cluster as I am currently using IPOIB instread of the native rdma. 

Cheers 

Andrei 

----- Original Message -----

> From: "Somnath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Andrei Mikhailovsky" <andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org>, "Andrey Korolyov"
> <andrey-5vqebrSIFTo@public.gmane.org>
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel"
> <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, 8 April, 2015 5:23:23 PM
> Subject: RE: [ceph-users] Preliminary RDMA vs TCP numbers

> Andrei,
> Yes, I see it has lot of potential and I believe fixing the
> performance bottlenecks inside XIO messenger it should go further.
> We are working on it and will keep community posted..

> Thanks & Regards
> Somnath

> From: Andrei Mikhailovsky [mailto:andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org]
> Sent: Wednesday, April 08, 2015 2:22 AM
> To: Andrey Korolyov
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org; ceph-devel; Somnath Roy
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> Hi,

> Am I the only person noticing disappointing results from the
> preliminary RDMA testing, or am I reading the numbers wrong?

> Yes, it's true that on a very small cluster you do see a great
> improvement in rdma, but in real life rdma is used in large
> infrastructure projects, not on a few servers with a handful of
> osds. In fact, from what i've seen from the slides, the rdma
> implementation scales horribly to the point that it becomes slower
> the more osds you through at it.

> From my limited knowledge, i have expected a much higher performance
> gains with rdma, taking into account that you should have much lower
> latency and overhead and lower cpu utilisation when using this
> transport in comparison with tcp.

> Are we likely to see a great deal of improvement with ceph and rdma
> in a near future? Is there a roadmap for having a stable and
> reliable rdma protocol support?

> Thanks

> Andrei
> ----- Original Message -----

> > From: "Andrey Korolyov" < andrey-5vqebrSIFTo@public.gmane.org >
> 
> > To: "Somnath Roy" < Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org >
> 
> > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org , "ceph-devel" <
> > ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >
> 
> > Sent: Wednesday, 8 April, 2015 9:28:12 AM
> 
> > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <
> > Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org > wrote:
> 
> > >
> 
> > > Hi,
> 
> > > Please find the preliminary performance numbers of TCP Vs RDMA
> > > (XIO) implementation (on top of SSDs) in the following link.
> 
> > >
> 
> > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> 
> > >
> 
> > > The attachment didn't go through it seems, so, I had to use
> > > slideshare.
> 
> > >
> 
> > > Mark,
> 
> > > If we have time, I can present it in tomorrow's performance
> > > meeting.
> 
> > >
> 
> > > Thanks & Regards
> 
> > > Somnath
> 
> > >
> 

> > Those numbers are really impressive (for small numbers at least)!
> > What
> 
> > are TCP settings you using?For example, difference can be lowered
> > on
> 
> > scale due to less intensive per-connection acceleration on CUBIC on
> > a
> 
> > larger number of nodes, though I do not believe that it was a main
> 
> > reason for an observed TCP catchup on a relatively flat workload
> > such
> 
> > as fio generates.
> 
> > _______________________________________________
> 
> > ceph-users mailing list
> 
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this
> message in error and that any review, dissemination, distribution,
> or copying of this message is strictly prohibited. If you have
> received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any and
> all copies of this message in your possession (whether hard copies
> or electronically stored copies).

[-- Attachment #1.2: Type: text/html, Size: 9660 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com