From: Kasper Dieter <dieter.kasper@ts.fujitsu.com>
To: Matthew Anderson <manderson8787@gmail.com>
Cc: ceph-devel@vger.kernel.org,
Kasper Dieter <Dieter.Kasper@ts.fujitsu.com>,
Andreas Bluemle <andreas.bluemle@itxperts.de>,
Andrey Korolyov <andrey@xdel.ru>
Subject: Re: Help needed porting Ceph to RSockets
Date: Fri, 9 Aug 2013 19:08:20 +0200 [thread overview]
Message-ID: <20130809170820.GB4531@oder.mch.fsc.net> (raw)
In-Reply-To: <CAJA05Umgio7XftGZtQdRzyWq70pXBb3oEx2jrT3BxcBr6MLoVQ@mail.gmail.com>
Hi Matthew,
please have a look at:
http://www.spinics.net/lists/linux-rdma/msg16710.html
http://wiki.ceph.com/01Planning/02Blueprints/Emperor/msgr%3A_implement_infiniband_support_via_rsockets
Maybe you should switch this discussion from ceph-user to the ceph-devel ML.
Kind Regards,
-Dieter
On Fri, Aug 09, 2013 at 09:11:07AM +0200, Matthew Anderson wrote:
> So I've had a chance to re-visit this since Bécholey Alexandre was kind
> enough to let me know how to compile Ceph with the RDMACM library
> (thankyou again!).
>
> At this stage it compiles and runs but there appears to be a problem with
> calling rshutdown in Pipe as it seems to just wait forever for the pipe to
> close which causes commands like 'ceph osd tree' to hang indefinitely
> after they work successfully. Debug MS is here -
> [1]http://pastebin.com/WzMJNKZY
>
> I also tried RADOS bench but it appears to be doing something similar.
> Debug MS is here - [2]http://pastebin.com/3aXbjzqS
>
> It seems like it's very close to working... I must be missing something
> small that's causing some grief. You can see the OSD coming up in the ceph
> monitor and the PG's all become active+clean. When shutting down the
> monitor I get the below which show's it waiting for the pipes to close -
>
> 2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing
> 2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping
> 2013-08-09 15:08:31.382115 7f464bd397c0 20 -- [3]172.16.0.1:6789/0 wait:
> stopped accepter thread
> 2013-08-09 15:08:31.382127 7f464bd397c0 20 -- [4]172.16.0.1:6789/0 wait:
> stopping reaper thread
> 2013-08-09 15:08:31.382146 7f4645500700 10 -- [5]172.16.0.1:6789/0
> reaper_entry done
> 2013-08-09 15:08:31.382182 7f464bd397c0 20 -- [6]172.16.0.1:6789/0 wait:
> stopped reaper thread
> 2013-08-09 15:08:31.382194 7f464bd397c0 10 -- [7]172.16.0.1:6789/0 wait:
> closing pipes
> 2013-08-09 15:08:31.382200 7f464bd397c0 10 -- [8]172.16.0.1:6789/0 reaper
> 2013-08-09 15:08:31.382205 7f464bd397c0 10 -- [9]172.16.0.1:6789/0 reaper
> done
> 2013-08-09 15:08:31.382210 7f464bd397c0 10 -- [10]172.16.0.1:6789/0 wait:
> waiting for pipes 0x3014c80,0x3015180,0x3015400 to close
>
> The git repo has been updated if anyone has a few spare minutes to take a
> look - [11]https://github.com/funkBuild/ceph-rsockets
>
> Thanks again
> -Matt
>
> On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson
> <[12]manderson8787@gmail.com> wrote:
>
> Hi All,
> I've had a few conversations on IRC about getting RDMA support into Ceph
> and thought I would give it a quick attempt to hopefully spur some
> interest. What I would like to accomplish is an RSockets only
> implementation so I'm able to use Ceph, RBD and QEMU at full speed over
> an Infiniband fabric.
> What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by
> replacing the regular socket calls with the rsocket equivalent.
> Unfortunately it doesn't compile and I get an error of -
> CXXLD ceph-osd
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::stop()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243:
> undefined reference to `rshutdown'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251:
> undefined reference to `rclose'
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::entry()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213:
> undefined reference to `raccept'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230:
> undefined reference to `rclose'
> ./.libs/libglobal.a(libcommon_la-Accepter.o): In function
> `Accepter::bind(entity_addr_t const&, int, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61:
> undefined reference to `rsocket'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80:
> undefined reference to `rsetsockopt'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
> undefined reference to `rbind'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118:
> undefined reference to `rgetsockname'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128:
> undefined reference to `rlisten'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100:
> undefined reference to `rbind'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:
> undefined reference to `rbind'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_write(char const*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined
> reference to `rsend'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined
> reference to `rshutdown'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::do_sendmsg(msghdr*, int, bool)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined
> reference to `rsendmsg'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_read_nonblocking(char*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined
> reference to `rrecv'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function
> `Pipe::tcp_read(char*, int)':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined
> reference to `rshutdown'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined
> reference to `rclose'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined
> reference to `rsocket'
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined
> reference to `rconnect'
> ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':
> /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined
> reference to `rwrite'
> collect2: error: ld returned 1 exit status
> make[3]: *** [ceph-mon] Error 1
> From the looks of it I need to include the 'rdma/rsocket.h' library
> somewhere else or add librdmacm but I'm not sure where.
> Full disclaimer, I am terrible at C++. If anyone has a few spare minutes
> to have a look into the error messages and can point out where I've gone
> wrong it would be greatly appreciated.
> I've put the code up at - [13]https://github.com/funkBuild/ceph-rsockets
> Thanks again
> -Matt
>
> References
>
> Visible links
> 1. http://pastebin.com/WzMJNKZY
> 2. http://pastebin.com/3aXbjzqS
> 3. http://172.16.0.1:6789/0
> 4. http://172.16.0.1:6789/0
> 5. http://172.16.0.1:6789/0
> 6. http://172.16.0.1:6789/0
> 7. http://172.16.0.1:6789/0
> 8. http://172.16.0.1:6789/0
> 9. http://172.16.0.1:6789/0
> 10. http://172.16.0.1:6789/0
> 11. https://github.com/funkBuild/ceph-rsockets
> 12. mailto:manderson8787@gmail.com
> 13. https://github.com/funkBuild/ceph-rsockets
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next parent reply other threads:[~2013-08-09 17:08 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAJA05UmTDGze7S50_j8RgHmPFYonk2Z94Fi6CQtxND9QxgRr3g@mail.gmail.com>
[not found] ` <CAJA05Umgio7XftGZtQdRzyWq70pXBb3oEx2jrT3BxcBr6MLoVQ@mail.gmail.com>
2013-08-09 17:08 ` Kasper Dieter [this message]
[not found] ` <20130812075513.43c338e1@andylap>
[not found] ` <CAJA05U=i22jHy2xbvCUk+UVN0+hOFDgt-JMSHTWKmr1+DZu0Dg@mail.gmail.com>
[not found] ` <20130812180644.447ca089@andylap>
[not found] ` <CAJA05U=KqzYis14sYXusbpk-T-F=uqvNz1XPurutmdeRz6BAdg@mail.gmail.com>
2013-08-13 4:27 ` [ceph-users] Help needed porting Ceph to RSockets Matthew Anderson
2013-08-13 5:53 ` Andreas Bluemle
2013-08-13 14:06 ` Andreas Bluemle
2013-08-13 14:35 ` Atchley, Scott
[not found] ` <1978D1F9-C675-4A37-AA57-C7E1158B2F72-1Heg1YXhbW8@public.gmane.org>
2013-08-13 21:44 ` Hefty, Sean
2013-08-14 7:21 ` Andreas Bluemle
2013-08-14 13:05 ` Atchley, Scott
2013-08-14 17:04 ` Hefty, Sean
2013-08-17 0:07 ` Hefty, Sean
2013-08-19 17:10 ` Hefty, Sean
2013-08-20 7:21 ` Andreas Bluemle
2013-08-20 10:30 ` Andreas Bluemle
2013-08-20 15:04 ` Hefty, Sean
[not found] ` <1828884A29C6694DAF28B7E6B8A8237388CA6F25-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-08-21 11:44 ` Matthew Anderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130809170820.GB4531@oder.mch.fsc.net \
--to=dieter.kasper@ts.fujitsu.com \
--cc=andreas.bluemle@itxperts.de \
--cc=andrey@xdel.ru \
--cc=ceph-devel@vger.kernel.org \
--cc=manderson8787@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox