* XioMessenger (RDMA) Performance results
@ 2014-03-18 20:05 Yaron Haviv
2014-03-18 20:17 ` Gregory Farnum
2014-03-18 20:32 ` Mark Nelson
0 siblings, 2 replies; 5+ messages in thread
From: Yaron Haviv @ 2014-03-18 20:05 UTC (permalink / raw)
To: ceph-devel (ceph-devel@vger.kernel.org),
Matt W. Benjamin (matt@cohortfs.com), Eyal Salomon
Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library
Results look pretty encouraging, demonstrating a * 20x * performance boost
Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware
Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)
Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound
protocol CRC msg depth IO size Msg/sec bandwidth (MB/s) CPU % server CPU% client
eth crc 50 4K 16,262 64 100% 100%
eth no_crc 50 4K 15,637 61 100% 100%
eth crc 50 64K 5,960 373 93% 100%
eth no_crc 50 64K 7,678 480 93% 100%
ipoib no_crc 50 4K 16,003 63 100% 100%
ipoib no_crc 50 64K 7,375 461 93% 100%
IB no_crc 50 4K 334,088 1305 98% 98%
IB no_crc 50 64K 95,078 5942 98% 98%
roce no_crc 50 4K 332,388 1298 95% 100%
roce no_crc 50 64K 69,445 4340 91% 87%
roce crc 50 4K 172,756 675 97% 100%
roce crc 50 64K 19,657 1229 100% 48%
Regards, Yaron
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XioMessenger (RDMA) Performance results
2014-03-18 20:05 XioMessenger (RDMA) Performance results Yaron Haviv
@ 2014-03-18 20:17 ` Gregory Farnum
2014-03-19 0:30 ` Matt W. Benjamin
2014-03-18 20:32 ` Mark Nelson
1 sibling, 1 reply; 5+ messages in thread
From: Gregory Farnum @ 2014-03-18 20:17 UTC (permalink / raw)
To: Yaron Haviv
Cc: ceph-devel (ceph-devel@vger.kernel.org),
Matt W. Benjamin (matt@cohortfs.com), Eyal Salomon
On Tue, Mar 18, 2014 at 1:05 PM, Yaron Haviv <yaronh@mellanox.com> wrote:
> Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library
> Results look pretty encouraging, demonstrating a * 20x * performance boost
Very nice! I'm excited. :)
What did you use to generate these numbers?
> Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware
> Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)
>
> Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound
>
> protocol CRC msg depth IO size Msg/sec bandwidth (MB/s) CPU % server CPU% client
> eth crc 50 4K 16,262 64 100% 100%
> eth no_crc 50 4K 15,637 61 100% 100%
Are these numbers correct? no_crc is slower?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
> eth crc 50 64K 5,960 373 93% 100%
> eth no_crc 50 64K 7,678 480 93% 100%
> ipoib no_crc 50 4K 16,003 63 100% 100%
> ipoib no_crc 50 64K 7,375 461 93% 100%
>
> IB no_crc 50 4K 334,088 1305 98% 98%
> IB no_crc 50 64K 95,078 5942 98% 98%
>
> roce no_crc 50 4K 332,388 1298 95% 100%
> roce no_crc 50 64K 69,445 4340 91% 87%
>
> roce crc 50 4K 172,756 675 97% 100%
> roce crc 50 64K 19,657 1229 100% 48%
>
> Regards, Yaron
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XioMessenger (RDMA) Performance results
2014-03-18 20:17 ` Gregory Farnum
@ 2014-03-19 0:30 ` Matt W. Benjamin
0 siblings, 0 replies; 5+ messages in thread
From: Matt W. Benjamin @ 2014-03-19 0:30 UTC (permalink / raw)
To: Gregory Farnum
Cc: ceph-devel (ceph-devel@vger.kernel.org), Eyal Salomon,
Yaron Haviv
Hi Greg,
----- "Gregory Farnum" <greg@inktank.com> wrote:
> On Tue, Mar 18, 2014 at 1:05 PM, Yaron Haviv <yaronh@mellanox.com>
> wrote:
> > Im happy to share test results we run in the lab with Matt's latest
> XioMessenger code which implements Ceph messaging over Accelio RDMA
> library
> > Results look pretty encouraging, demonstrating a * 20x * performance
> boost
>
> Very nice! I'm excited. :)
> What did you use to generate these numbers?
The two workload sets are created using the simple_client/simple_server vs
xio_client/xio_server client/server pairs, on...one of the latest XioMessenger
branches. Eyal, was it xio-rados-exp or xio-rados-exp-noreg?
I -think- it would be xio-rados-exp. The new stuff from us (including a couple
of librados fixups by Marcus and myself) is on xio-rados-exp-noreg.
>
> > Below is a table comparing XioMessenger (RDMA) with SimpleMessanger
> (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> > Note that we tested with CRC on/off, in RDMA there is no need for
> software CRC, its done by the hardware
> > Tests below use a single communication thread, using more threads
> would produce higher performance (on 64KB IO the Link/PCIe is
> saturated with RDMA using a single thread)
> >
> > Matt has some more small IO optimizations in the pipe, and we hope
> to share perf results using librados soon, I assume they will be disk
> bound
> >
> > protocol CRC msg depth IO size Msg/sec bandwidth
> (MB/s) CPU % server CPU% client
> > eth crc 50 4K 16,262 64 100%
> 100%
> > eth no_crc 50 4K 15,637 61 100%
> 100%
>
> Are these numbers correct? no_crc is slower?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
These numbers were from Mellanox, so I defer to Eyal. I have seen
some inconsistency with crc vs no_crc in informal runs, myself.
>
> > eth crc 50 64K 5,960 373 93%
> 100%
> > eth no_crc 50 64K 7,678 480 93%
> 100%
> > ipoib no_crc 50 4K 16,003 63 100%
> 100%
> > ipoib no_crc 50 64K 7,375 461 93%
> 100%
> >
> > IB no_crc 50 4K 334,088 1305 98%
> 98%
> > IB no_crc 50 64K 95,078 5942 98%
> 98%
> >
> > roce no_crc 50 4K 332,388 1298 95%
> 100%
> > roce no_crc 50 64K 69,445 4340 91%
> 87%
> >
> > roce crc 50 4K 172,756 675 97%
> 100%
> > roce crc 50 64K 19,657 1229 100%
> 48%
> >
> > Regards, Yaron
> >
Regards,
Matt
--
Matt Benjamin
CohortFS, LLC.
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XioMessenger (RDMA) Performance results
2014-03-18 20:05 XioMessenger (RDMA) Performance results Yaron Haviv
2014-03-18 20:17 ` Gregory Farnum
@ 2014-03-18 20:32 ` Mark Nelson
2014-03-19 0:31 ` Matt W. Benjamin
1 sibling, 1 reply; 5+ messages in thread
From: Mark Nelson @ 2014-03-18 20:32 UTC (permalink / raw)
To: Yaron Haviv
Cc: ceph-devel (ceph-devel@vger.kernel.org),
Matt W. Benjamin (matt@cohortfs.com), Eyal Salomon
On 03/18/2014 03:05 PM, Yaron Haviv wrote:
> Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library
> Results look pretty encouraging, demonstrating a * 20x * performance boost
>
> Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware
> Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)
>
> Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound
>
> protocol CRC msg depth IO size Msg/sec bandwidth (MB/s) CPU % server CPU% client
> eth crc 50 4K 16,262 64 100% 100%
> eth no_crc 50 4K 15,637 61 100% 100%
> eth crc 50 64K 5,960 373 93% 100%
> eth no_crc 50 64K 7,678 480 93% 100%
> ipoib no_crc 50 4K 16,003 63 100% 100%
> ipoib no_crc 50 64K 7,375 461 93% 100%
>
> IB no_crc 50 4K 334,088 1305 98% 98%
> IB no_crc 50 64K 95,078 5942 98% 98%
>
> roce no_crc 50 4K 332,388 1298 95% 100%
> roce no_crc 50 64K 69,445 4340 91% 87%
>
> roce crc 50 4K 172,756 675 97% 100%
> roce crc 50 64K 19,657 1229 100% 48%
Very nice! were you just writing to memory on the backend?
Mark
>
> Regards, Yaron
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-03-19 0:32 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-03-18 20:05 XioMessenger (RDMA) Performance results Yaron Haviv
2014-03-18 20:17 ` Gregory Farnum
2014-03-19 0:30 ` Matt W. Benjamin
2014-03-18 20:32 ` Mark Nelson
2014-03-19 0:31 ` Matt W. Benjamin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.