XioMessenger (RDMA) Status Update

XioMessenger (RDMA) Status Update

[Matt W. Benjamin]

Hi Ceph Devs,

I've pushed two Ceph+Accelio branches, xio-firefly and xio-firefly-cmake to our
public ceph repository https://github.com/linuxbox2/linuxbox-ceph.git .

These branches are pulled up to the HEAD of ceph/firefly, and also have improvements
to XioMessenger which allow:

1. operation of (at least) a minimal Ceph Mon and OSD cluster over Xio messaging alone
2. initial support for Accelio in the MDS server and Client/libcephfs (not formally tested)

There are some new config options:

(global addr) "rdma local" sets a local rdma interface address
(global bool) "cluster rdma" selects Accelio for intra-cluster communication
(global bool) "client rdma" selecs Accelio for libcephfs communications

These changes haven't been strenuously tested, we'll expect to have additional information
and likely new, simple rados bench results against this code in the next several days,
at latest.

Thanks,

Matt

-- 
Matt Benjamin
CohortFS, LLC.
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XioMessenger (RDMA) Status Update

[Mark Nelson]

On 05/13/2014 04:08 PM, Matt W. Benjamin wrote:
> Hi Ceph Devs,
>
> I've pushed two Ceph+Accelio branches, xio-firefly and xio-firefly-cmake to our
> public ceph repository https://github.com/linuxbox2/linuxbox-ceph.git .
>
> These branches are pulled up to the HEAD of ceph/firefly, and also have improvements
> to XioMessenger which allow:
>
> 1. operation of (at least) a minimal Ceph Mon and OSD cluster over Xio messaging alone
> 2. initial support for Accelio in the MDS server and Client/libcephfs (not formally tested)
>
> There are some new config options:
>
> (global addr) "rdma local" sets a local rdma interface address
> (global bool) "cluster rdma" selects Accelio for intra-cluster communication
> (global bool) "client rdma" selecs Accelio for libcephfs communications
>
> These changes haven't been strenuously tested, we'll expect to have additional information
> and likely new, simple rados bench results against this code in the next several days,
> at latest.

Excellent new Matt.  Can't wait to see your results! :D

>
> Thanks,
>
> Matt
>

Re: XioMessenger (RDMA) Status Update

[Gregory Farnum]

On Tue, May 13, 2014 at 2:08 PM, Matt W. Benjamin <matt@cohortfs.com> wrote:
> Hi Ceph Devs,
>
> I've pushed two Ceph+Accelio branches, xio-firefly and xio-firefly-cmake to our
> public ceph repository https://github.com/linuxbox2/linuxbox-ceph.git .
>
> These branches are pulled up to the HEAD of ceph/firefly, and also have improvements
> to XioMessenger which allow:
>
> 1. operation of (at least) a minimal Ceph Mon and OSD cluster over Xio messaging alone
> 2. initial support for Accelio in the MDS server and Client/libcephfs (not formally tested)
>
> There are some new config options:
>
> (global addr) "rdma local" sets a local rdma interface address
> (global bool) "cluster rdma" selects Accelio for intra-cluster communication
> (global bool) "client rdma" selecs Accelio for libcephfs communications
>
> These changes haven't been strenuously tested, we'll expect to have additional information
> and likely new, simple rados bench results against this code in the next several days,
> at latest.

Can you talk about what structural changes you've made since the last
branch? :) Is there support for mark_down or some of the other things
we've discussed?
-Greg

Re: XioMessenger (RDMA) Status Update

[Matt W. Benjamin]

Hi Greg,

1. there isn't support for for mark_down() yet (!), but we're working
on it.  I've had good discussions with the Accelio team regarding exposing
explicit channel and flow control interfaces, which round out that topic
as we were exploring it.

2. the big changes in this push are major cleanups and bug fixes, plus
much wider integration with the user space code.  my #1 action item from
our last convo was to first extend Xio coverage everywhere, and in particular,
add support for intra-cluster messaging over Xio.

2.1 on the cleanups and bugs front, starting with biggest, I finished support
for peer identification, using a small api change to Accelio (it's now on the
official Accelio for_next branch).  a Connection refcnt underrun is fixed.  then
I added a mechanism to bypass Accelio for intra-server messages, using a special
singleton connection.  I cleaned up, but didn't deepen, feature support, as a
temporary measure.  There's lot's of cleanup and various fixes in ceph_osd, too.

2.2 then, I did first cut integration with the MDS, and while at it, the Client
module as well.  I've only started verifying, but so far it looks like the
intra-cluster messaging paths work, and a client workload may work--more fixes
likely forthcoming.

I have a plan to get at least the starter policy interfaces like mark_down and
mark_down_all shorty.

Matt
 
----- "Gregory Farnum" <greg@inktank.com> wrote:

> On Tue, May 13, 2014 at 2:08 PM, Matt W. Benjamin <matt@cohortfs.com>
> wrote:
> > Hi Ceph Devs,
> >
> > I've pushed two Ceph+Accelio branches, xio-firefly and
> xio-firefly-cmake to our
> > public ceph repository
> https://github.com/linuxbox2/linuxbox-ceph.git .
> >
> > These branches are pulled up to the HEAD of ceph/firefly, and also
> have improvements
> > to XioMessenger which allow:
> >
> > 1. operation of (at least) a minimal Ceph Mon and OSD cluster over
> Xio messaging alone
> > 2. initial support for Accelio in the MDS server and
> Client/libcephfs (not formally tested)
> >
> > There are some new config options:
> >
> > (global addr) "rdma local" sets a local rdma interface address
> > (global bool) "cluster rdma" selects Accelio for intra-cluster
> communication
> > (global bool) "client rdma" selecs Accelio for libcephfs
> communications
> >
> > These changes haven't been strenuously tested, we'll expect to have
> additional information
> > and likely new, simple rados bench results against this code in the
> next several days,
> > at latest.
> 
> Can you talk about what structural changes you've made since the last
> branch? :) Is there support for mark_down or some of the other things
> we've discussed?
> -Greg

-- 
Matt Benjamin
CohortFS, LLC.
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XioMessenger (RDMA) Status Update

[Gregory Farnum]

On Tue, May 13, 2014 at 6:30 PM, Matt W. Benjamin <matt@cohortfs.com> wrote:
> Hi Greg,
>
> 1. there isn't support for for mark_down() yet (!), but we're working
> on it.  I've had good discussions with the Accelio team regarding exposing
> explicit channel and flow control interfaces, which round out that topic
> as we were exploring it.
>
> 2. the big changes in this push are major cleanups and bug fixes, plus
> much wider integration with the user space code.  my #1 action item from
> our last convo was to first extend Xio coverage everywhere, and in particular,
> add support for intra-cluster messaging over Xio.
>
> 2.1 on the cleanups and bugs front, starting with biggest, I finished support
> for peer identification, using a small api change to Accelio (it's now on the
> official Accelio for_next branch).  a Connection refcnt underrun is fixed.  then
> I added a mechanism to bypass Accelio for intra-server messages, using a special
> singleton connection.  I cleaned up, but didn't deepen, feature support, as a
> temporary measure.  There's lot's of cleanup and various fixes in ceph_osd, too.
>
> 2.2 then, I did first cut integration with the MDS, and while at it, the Client
> module as well.  I've only started verifying, but so far it looks like the
> intra-cluster messaging paths work, and a client workload may work--more fixes
> likely forthcoming.
>
> I have a plan to get at least the starter policy interfaces like mark_down and
> mark_down_all shorty.

Cool. Let me know when those features are ready for a semantics review
or there's something else we need to talk about. :)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: XioMessenger (RDMA) Status Update

[Sage Weil]

Hi Matt,

I've rebased this branch on top of master and pushed it to wip-xio in 
ceph.git, and then opened a pull request to capture review:

	https://github.com/ceph/ceph/pull/1819

I would like to get some of the preliminary pieces into master sooner 
rather than later so we can start cutting down on the size of the branch.  
I've started by looking just first few patches that modify the Messenger 
and made a few clean ups:

 - use SimplePolicyMessage for SimpleMessenger, too, to avoid dup code
 - move PipeConnection into a separate header and tweak a few things to 
   remove it from the generic Messenger/Message/Connection interface
 - cleaned up a bit of cruft from the original patch
 - resolved a few merge conflicts between firefly and master

Before I get too far into this can you take a look?  I'd like to pull 
*all* of the non-xio stuff to the top of the branch and get it in good 
shape first.

I think the next step for me is to look at how you've instantiated the 
alternate xio messenger and clean up that interface.  This is probably 
also a good time for us to get the entity_addr_t and entity_name_t stuff 
sorted out.

Thanks!
sage


On Tue, 13 May 2014, Matt W. Benjamin wrote:

> Hi Ceph Devs,
> 
> I've pushed two Ceph+Accelio branches, xio-firefly and xio-firefly-cmake to our
> public ceph repository https://github.com/linuxbox2/linuxbox-ceph.git .
> 
> These branches are pulled up to the HEAD of ceph/firefly, and also have improvements
> to XioMessenger which allow:
> 
> 1. operation of (at least) a minimal Ceph Mon and OSD cluster over Xio messaging alone
> 2. initial support for Accelio in the MDS server and Client/libcephfs (not formally tested)
> 
> There are some new config options:
> 
> (global addr) "rdma local" sets a local rdma interface address
> (global bool) "cluster rdma" selects Accelio for intra-cluster communication
> (global bool) "client rdma" selecs Accelio for libcephfs communications
> 
> These changes haven't been strenuously tested, we'll expect to have additional information
> and likely new, simple rados bench results against this code in the next several days,
> at latest.
> 
> Thanks,
> 
> Matt
> 
> -- 
> Matt Benjamin
> CohortFS, LLC.
> 206 South Fifth Ave. Suite 150
> Ann Arbor, MI  48104
> 
> http://cohortfs.com
> 
> tel.  734-761-4689 
> fax.  734-769-8938 
> cel.  734-216-5309 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: XioMessenger (RDMA) Status Update

[Gregory Farnum]

On Fri, May 16, 2014 at 11:21 AM, Sage Weil <sage@inktank.com> wrote:
> Hi Matt,
>
> I've rebased this branch on top of master and pushed it to wip-xio in
> ceph.git, and then opened a pull request to capture review:
>
>         https://github.com/ceph/ceph/pull/1819
>
> I would like to get some of the preliminary pieces into master sooner
> rather than later so we can start cutting down on the size of the branch.
> I've started by looking just first few patches that modify the Messenger
> and made a few clean ups:
>
>  - use SimplePolicyMessage for SimpleMessenger, too, to avoid dup code
>  - move PipeConnection into a separate header and tweak a few things to
>    remove it from the generic Messenger/Message/Connection interface
>  - cleaned up a bit of cruft from the original patch
>  - resolved a few merge conflicts between firefly and master
>
> Before I get too far into this can you take a look?  I'd like to pull
> *all* of the non-xio stuff to the top of the branch and get it in good
> shape first.
>
> I think the next step for me is to look at how you've instantiated the
> alternate xio messenger and clean up that interface.  This is probably
> also a good time for us to get the entity_addr_t and entity_name_t stuff
> sorted out.

Unfortunately, I think we need to get farther along before pulling any
of this into master. In particular, unless it's been updated since I
looked at it last, the SimplePolicyMessenger doesn't do anything right
now and we're not sure of whether the interface is actually helpful.
We can perhaps do some of the multi-messenger support bits, but the
internal interface changes really need to wait until we've verified
that it's possible to implement a non-TCP messenger with them before
it's worth making the TCP stuff follow their rules.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: XioMessenger (RDMA) Status Update

[Sage Weil]

On Fri, 16 May 2014, Gregory Farnum wrote:
> On Fri, May 16, 2014 at 11:21 AM, Sage Weil <sage@inktank.com> wrote:
> > Hi Matt,
> >
> > I've rebased this branch on top of master and pushed it to wip-xio in
> > ceph.git, and then opened a pull request to capture review:
> >
> >         https://github.com/ceph/ceph/pull/1819
> >
> > I would like to get some of the preliminary pieces into master sooner
> > rather than later so we can start cutting down on the size of the branch.
> > I've started by looking just first few patches that modify the Messenger
> > and made a few clean ups:
> >
> >  - use SimplePolicyMessage for SimpleMessenger, too, to avoid dup code
> >  - move PipeConnection into a separate header and tweak a few things to
> >    remove it from the generic Messenger/Message/Connection interface
> >  - cleaned up a bit of cruft from the original patch
> >  - resolved a few merge conflicts between firefly and master
> >
> > Before I get too far into this can you take a look?  I'd like to pull
> > *all* of the non-xio stuff to the top of the branch and get it in good
> > shape first.
> >
> > I think the next step for me is to look at how you've instantiated the
> > alternate xio messenger and clean up that interface.  This is probably
> > also a good time for us to get the entity_addr_t and entity_name_t stuff
> > sorted out.
> 
> Unfortunately, I think we need to get farther along before pulling any
> of this into master.

I agree. I'm just trying to get this cleaned up and ordered so that it can 
be sanely reviewed.  :)

> In particular, unless it's been updated since I looked at it last, the 
> SimplePolicyMessenger doesn't do anything right now and we're not sure 
> of whether the interface is actually helpful. We can perhaps do some of 
> the multi-messenger support bits, but the internal interface changes 
> really need to wait until we've verified that it's possible to implement 
> a non-TCP messenger with them before it's worth making the TCP stuff 
> follow their rules.

The SimplePolicyMessenger piece just pull the get/set Policy stuff out of 
SimpleMessenger into a parent class; nothing more.  The multiplexer is 
something else (which I can't find in that branch at the moment!).

So far everything I've looked at has been a nice cleanup of the abstract 
interface.  The next thing I'm concerned about is something akin to a sane 
factory method so that (hopefully) the explicit SimpleMessenger references 
can be dropped from other parts of the code.  I'm not touching the xio 
bits yet, although it looks like that will quickly lead us to some 
other core pieces like buffer support for xio.

Don't worry--none of this will get merged until it is cleaned up, 
reviewed, and tested!

sage