From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrei Mikhailovsky Subject: Re: Preliminary RDMA vs TCP numbers Date: Wed, 8 Apr 2015 10:22:05 +0100 (BST) Message-ID: <6442513.754.1428484840531.JavaMail.andrei@tuchka> References: <755F6B91B3BE364F9BCA11EA3F9E0C6F2CD75D78@SACMBXIP01.sdcorp.global.sandisk.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0495740272==" Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: "ceph-users" To: Andrey Korolyov Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, ceph-devel List-Id: ceph-devel.vger.kernel.org --===============0495740272== Content-Type: multipart/alternative; boundary="----=_Part_753_14911911.1428484840530" ------=_Part_753_14911911.1428484840530 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Am I the only person noticing disappointing results from the preliminary RDMA testing, or am I reading the numbers wrong? Yes, it's true that on a very small cluster you do see a great improvement in rdma, but in real life rdma is used in large infrastructure projects, not on a few servers with a handful of osds. In fact, from what i've seen from the slides, the rdma implementation scales horribly to the point that it becomes slower the more osds you through at it. >From my limited knowledge, i have expected a much higher performance gains with rdma, taking into account that you should have much lower latency and overhead and lower cpu utilisation when using this transport in comparison with tcp. Are we likely to see a great deal of improvement with ceph and rdma in a near future? Is there a roadmap for having a stable and reliable rdma protocol support? Thanks Andrei ----- Original Message ----- > From: "Andrey Korolyov" > To: "Somnath Roy" > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel" > > Sent: Wednesday, 8 April, 2015 9:28:12 AM > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy > wrote: > > > > Hi, > > Please find the preliminary performance numbers of TCP Vs RDMA > > (XIO) implementation (on top of SSDs) in the following link. > > > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma > > > > The attachment didn't go through it seems, so, I had to use > > slideshare. > > > > Mark, > > If we have time, I can present it in tomorrow's performance > > meeting. > > > > Thanks & Regards > > Somnath > > > Those numbers are really impressive (for small numbers at least)! > What > are TCP settings you using?For example, difference can be lowered on > scale due to less intensive per-connection acceleration on CUBIC on a > larger number of nodes, though I do not believe that it was a main > reason for an observed TCP catchup on a relatively flat workload such > as fio generates. > _______________________________________________ > ceph-users mailing list > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ------=_Part_753_14911911.1428484840530 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <= div style=3D'font-family: arial,helvetica,sans-serif; font-size: 10pt; colo= r: #000000'>Hi,

Am I the only person noticing disappointing results = from the preliminary RDMA testing, or am I reading the numbers wrong?
<= br>Yes, it's true that on a very small cluster you do see a great improveme= nt in rdma, but in real life rdma is used in large infrastructure projects,= not on a few servers with a handful of osds. In fact, from what i've seen = from the slides, the rdma implementation scales horribly to the point that = it becomes slower the more osds you through at it.

From my limited k= nowledge, i have expected a much higher performance gains with rdma, taking= into account that you should have much lower latency and overhead and lowe= r cpu utilisation when using this transport in comparison with tcp.

= Are we likely to see a great deal of improvement with ceph and rdma in a ne= ar future? Is there a roadmap for having a stable and reliable rdma protoco= l support?

Thanks

Andrei
To: "Somn= ath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: ceph-users-idqoXFIVOFJ4Eiagz67IpQ@public.gmane.org= h.com, "ceph-devel" <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Sent: Wedn= esday, 8 April, 2015 9:28:12 AM
Subject: Re: [ceph-users] Prelimi= nary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Ro= y <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
>
> Hi,
> Please= find the preliminary performance numbers of TCP Vs RDMA (XIO) implementati= on (on top of SSDs) in the following link.
>
> http://www.slide= share.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't= go through it seems, so, I had to use slideshare.
>
> Mark,> If we have time, I can present it in tomorrow's performance meeting.<= br>>
> Thanks & Regards
> Somnath
>

Those n= umbers are really impressive (for small numbers at least)! What
are TCP = settings you using?For example, difference can be lowered on
scale due t= o less intensive per-connection acceleration on CUBIC on a
larger number= of nodes, though I do not believe that it was a main
reason for an obse= rved TCP catchup on a relatively flat workload such
as fio generates._______________________________________________
ceph-users mailing list=
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-us= ers-ceph.com

------=_Part_753_14911911.1428484840530-- --===============0495740272== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============0495740272==--