From: Kasper Dieter <dieter.kasper@ts.fujitsu.com>
To: Matthew Anderson <manderson8787@gmail.com>
Cc: ceph-devel@vger.kernel.org,
Kasper Dieter <Dieter.Kasper@ts.fujitsu.com>,
Andreas Bluemle <andreas.bluemle@itxperts.de>,
Andrey Korolyov <andrey@xdel.ru>
Subject: Re: Help needed porting Ceph to RSockets
Date: Fri, 9 Aug 2013 19:08:20 +0200 [thread overview]
Message-ID: <20130809170820.GB4531@oder.mch.fsc.net> (raw)
In-Reply-To: <CAJA05Umgio7XftGZtQdRzyWq70pXBb3oEx2jrT3BxcBr6MLoVQ@mail.gmail.com>
Hi Matthew,
please have a look at:
http://www.spinics.net/lists/linux-rdma/msg16710.html
http://wiki.ceph.com/01Planning/02Blueprints/Emperor/msgr%3A_implement_infiniband_support_via_rsockets
Maybe you should switch this discussion from ceph-user to the ceph-devel ML.
Kind Regards,
-Dieter
On Fri, Aug 09, 2013 at 09:11:07AM +0200, Matthew Anderson wrote:
> So I've had a chance to re-visit this since Bécholey Alexandre was kind
> enough to let me know how to compile Ceph with the RDMACM library
> (thankyou again!).
>
> At this stage it compiles and runs but there appears to be a problem with
> calling rshutdown in Pipe as it seems to just wait forever for the pipe to
> close which causes commands like 'ceph osd tree' to hang indefinitely
> after they work successfully. Debug MS is here -
> [1]http://pastebin.com/WzMJNKZY
>
> I also tried RADOS bench but it appears to be doing something similar.
> Debug MS is here - [2]http://pastebin.com/3aXbjzqS
>
> It seems like it's very close to working... I must be missing something
> small that's causing some grief. You can see the OSD coming up in the ceph
> monitor and the PG's all become active+clean. When shutting down the
> monitor I get the below which show's it waiting for the pipes to close -
>
> 2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing
> 2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping
> 2013-08-09 15:08:31.382115 7f464bd397c0 20 -- [3]172.16.0.1:6789/0 wait:
> stopped accepter thread
> 2013-08-09 15:08:31.382127 7f464bd397c0 20 -- [4]172.16.0.1:6789/0 wait:
> stopping reaper thread
> 2013-08-09 15:08:31.382146 7f4645500700 10 -- [5]172.16.0.1:6789/0
> reaper_entry done
> 2013-08-09 15:08:31.382182 7f464bd397c0 20 -- [6]172.16.0.1:6789/0 wait:
> stopped reaper thread
> 2013-08-09 15:08:31.382194 7f464bd397c0 10 -- [7]172.16.0.1:6789/0 wait:
> closing pipes
> 2013-08-09 15:08:31.382200 7f464bd397c0 10 -- [8]172.16.0.1:6789/0 reaper
> 2013-08-09 15:08:31.382205 7f464bd397c0 10 -- [9]172.16.0.1:6789/0 reaper
> done
> 2013-08-09 15:08:31.382210 7f464bd397c0 10 -- [10]172.16.0.1:6789/0 wait:
> waiting for pipes 0x3014c80,0x3015180,0x3015400 to close
>
> The git repo has been updated if anyone has a few spare minutes to take a
> look - [11]https://github.com/funkBuild/ceph-rsockets
>
> Thanks again
> -Matt
>
> On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson
> <[12]manderson8787@gmail.com> wrote:
>
> Hi All,
> I've had a few conversations on IRC about getting RDMA support into Ceph
> and thought I would give it a quick attempt to hopefully spur some
> interest. What I would like to accomplish is an RSockets only
> implementation so I'm able to use Ceph, RBD and QEMU at full speed over
> an Infiniband fabric.
> What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by
> replacing the regular socket calls with the rsocket equivalent.
> Unfortunately it doesn't compile and I get an error of -
> CXXLD ceph-osd
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::stop()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243:
> undefined reference to `rshutdown'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251:
> undefined reference to `rclose'
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::entry()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213:
> undefined reference to `raccept'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230:
> undefined reference to `rclose'
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::bind(entity_addr_t const&, int, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61:
> undefined reference to `rsocket'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80:
> undefined reference to `rsetsockopt'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
> undefined reference to `rbind'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118:
> undefined reference to `rgetsockname'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128:
> undefined reference to `rlisten'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100:
> undefined reference to `rbind'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
> undefined reference to `rbind'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_write(char const*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined
> reference to `rsend'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined
> reference to `rshutdown'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::do_sendmsg(msghdr*, int, bool)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined
> reference to `rsendmsg'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_read_nonblocking(char*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined
> reference to `rrecv'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_read(char*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined
> reference to `rshutdown'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined
> reference to `rclose'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined
> reference to `rsocket'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined
> reference to `rconnect'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined
> reference to `rwrite'
> collect2: error: ld returned 1 exit status
> make[3]: *** [ceph-mon] Error 1
> From the looks of it I need to include the 'rdma/rsocket.h' library
> somewhere else or add librdmacm but I'm not sure where.
> Full disclaimer, I am terrible at C++. If anyone has a few spare minutes
> to have a look into the error messages and can point out where I've gone
> wrong it would be greatly appreciated.
> I've put the code up at - [13]https://github.com/funkBuild/ceph-rsockets
> Thanks again
> -Matt
>
> References
>
> Visible links
> 1. http://pastebin.com/WzMJNKZY
> 2. http://pastebin.com/3aXbjzqS
> 3. http://172.16.0.1:6789/0
> 4. http://172.16.0.1:6789/0
> 5. http://172.16.0.1:6789/0
> 6. http://172.16.0.1:6789/0
> 7. http://172.16.0.1:6789/0
> 8. http://172.16.0.1:6789/0
> 9. http://172.16.0.1:6789/0
> 10. http://172.16.0.1:6789/0
> 11. https://github.com/funkBuild/ceph-rsockets
> 12. mailto:manderson8787@gmail.com
> 13. https://github.com/funkBuild/ceph-rsockets
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next parent reply other threads:[~2013-08-09 17:08 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAJA05UmTDGze7S50_j8RgHmPFYonk2Z94Fi6CQtxND9QxgRr3g@mail.gmail.com>
[not found] ` <CAJA05Umgio7XftGZtQdRzyWq70pXBb3oEx2jrT3BxcBr6MLoVQ@mail.gmail.com>
2013-08-09 17:08 ` Kasper Dieter [this message]
[not found] ` <20130812075513.43c338e1@andylap>
[not found] ` <CAJA05U=i22jHy2xbvCUk+UVN0+hOFDgt-JMSHTWKmr1+DZu0Dg@mail.gmail.com>
[not found] ` <20130812180644.447ca089@andylap>
[not found] ` <CAJA05U=KqzYis14sYXusbpk-T-F=uqvNz1XPurutmdeRz6BAdg@mail.gmail.com>
2013-08-13 4:27 ` [ceph-users] Help needed porting Ceph to RSockets Matthew Anderson
2013-08-13 5:53 ` Andreas Bluemle
2013-08-13 14:06 ` Andreas Bluemle
2013-08-13 14:35 ` Atchley, Scott
[not found] ` <1978D1F9-C675-4A37-AA57-C7E1158B2F72-1Heg1YXhbW8@public.gmane.org>
2013-08-13 21:44 ` Hefty, Sean
2013-08-14 7:21 ` Andreas Bluemle
2013-08-14 13:05 ` Atchley, Scott
2013-08-14 17:04 ` Hefty, Sean
2013-08-17 0:07 ` Hefty, Sean
2013-08-19 17:10 ` Hefty, Sean
2013-08-20 7:21 ` Andreas Bluemle
2013-08-20 10:30 ` Andreas Bluemle
2013-08-20 15:04 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA6F25-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-21 11:44 ` Matthew Anderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130809170820.GB4531@oder.mch.fsc.net \
--to=dieter.kasper@ts.fujitsu.com \
--cc=andreas.bluemle@itxperts.de \
--cc=andrey@xdel.ru \
--cc=ceph-devel@vger.kernel.org \
--cc=manderson8787@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.