Re: Help needed porting Ceph to RSockets

Re: Help needed porting Ceph to RSockets

[Kasper Dieter]

Hi Matthew,

please have a look at: 
http://www.spinics.net/lists/linux-rdma/msg16710.html
http://wiki.ceph.com/01Planning/02Blueprints/Emperor/msgr%3A_implement_infiniband_support_via_rsockets

Maybe you should switch this discussion from ceph-user to the ceph-devel ML.

Kind Regards,
-Dieter

On Fri, Aug 09, 2013 at 09:11:07AM +0200, Matthew Anderson wrote:
>    So I've had a chance to re-visit this since Bécholey Alexandre was kind      
>    enough to let me know how to compile Ceph with the RDMACM library            
>    (thankyou again!).                                                           
>                                                                                 
>    At this stage it compiles and runs but there appears to be a problem with    
>    calling rshutdown in Pipe as it seems to just wait forever for the pipe to   
>    close which causes commands like 'ceph osd tree' to hang indefinitely        
>    after they work successfully. Debug MS is here -                             
>    [1]http://pastebin.com/WzMJNKZY                                              
>                                                                                 
>    I also tried RADOS bench but it appears to be doing something similar.       
>    Debug MS is here - [2]http://pastebin.com/3aXbjzqS                           
>                                                                                 
>    It seems like it's very close to working... I must be missing something      
>    small that's causing some grief. You can see the OSD coming up in the ceph   
>    monitor and the PG's all become active+clean. When shutting down the         
>    monitor I get the below which show's it waiting for the pipes to close -     
>                                                                                 
>    2013-08-09 15:08:31.339394 7f4643cfd700 20 accepter.accepter closing         
>    2013-08-09 15:08:31.382075 7f4643cfd700 10 accepter.accepter stopping        
>    2013-08-09 15:08:31.382115 7f464bd397c0 20 -- [3]172.16.0.1:6789/0 wait:     
>    stopped accepter thread                                                      
>    2013-08-09 15:08:31.382127 7f464bd397c0 20 -- [4]172.16.0.1:6789/0 wait:     
>    stopping reaper thread                                                       
>    2013-08-09 15:08:31.382146 7f4645500700 10 -- [5]172.16.0.1:6789/0           
>    reaper_entry done                                                            
>    2013-08-09 15:08:31.382182 7f464bd397c0 20 -- [6]172.16.0.1:6789/0 wait:     
>    stopped reaper thread                                                        
>    2013-08-09 15:08:31.382194 7f464bd397c0 10 -- [7]172.16.0.1:6789/0 wait:     
>    closing pipes                                                                
>    2013-08-09 15:08:31.382200 7f464bd397c0 10 -- [8]172.16.0.1:6789/0 reaper    
>    2013-08-09 15:08:31.382205 7f464bd397c0 10 -- [9]172.16.0.1:6789/0 reaper    
>    done                                                                         
>    2013-08-09 15:08:31.382210 7f464bd397c0 10 -- [10]172.16.0.1:6789/0 wait:    
>    waiting for pipes 0x3014c80,0x3015180,0x3015400 to close                     
>                                                                                 
>    The git repo has been updated if anyone has a few spare minutes to take a    
>    look - [11]https://github.com/funkBuild/ceph-rsockets                        
>                                                                                 
>    Thanks again                                                                 
>    -Matt                                                                        
>                                                                                 
>    On Thu, Jun 20, 2013 at 5:09 PM, Matthew Anderson                            
>    <[12]manderson8787@gmail.com> wrote:                                         
>                                                                                 
>      Hi All,                                                                    
>      I've had a few conversations on IRC about getting RDMA support into Ceph   
>      and thought I would give it a quick attempt to hopefully spur some         
>      interest. What I would like to accomplish is an RSockets only              
>      implementation so I'm able to use Ceph, RBD and QEMU at full speed over    
>      an Infiniband fabric.                                                      
>      What I've tried to do is port Pipe.cc and Acceptor.cc to rsockets by       
>      replacing the regular socket calls with the rsocket equivalent.            
>      Unfortunately it doesn't compile and I get an error of -                   
>       CXXLD  ceph-osd                                                           
>      ./.libs/libglobal.a(libcommon_la-Accepter.o): In function                  
>      `Accepter::stop()':                                                        
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:243:           
>      undefined reference to `rshutdown'                                         
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:251:           
>      undefined reference to `rclose'                                            
>      ./.libs/libglobal.a(libcommon_la-Accepter.o): In function                  
>      `Accepter::entry()':                                                       
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:213:           
>      undefined reference to `raccept'                                           
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:230:           
>      undefined reference to `rclose'                                            
>      ./.libs/libglobal.a(libcommon_la-Accepter.o): In function                  
>      `Accepter::bind(entity_addr_t const&, int, int)':                          
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:61:            
>      undefined reference to `rsocket'                                           
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:80:            
>      undefined reference to `rsetsockopt'                                       
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:            
>      undefined reference to `rbind'                                             
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:118:           
>      undefined reference to `rgetsockname'                                      
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:128:           
>      undefined reference to `rlisten'                                           
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:100:           
>      undefined reference to `rbind'                                             
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Accepter.cc:87:            
>      undefined reference to `rbind'                                             
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function                      
>      `Pipe::tcp_write(char const*, int)':                                       
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2175: undefined    
>      reference to `rsend'                                                       
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2162: undefined    
>      reference to `rshutdown'                                                   
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function                      
>      `Pipe::do_sendmsg(msghdr*, int, bool)':                                    
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1867: undefined    
>      reference to `rsendmsg'                                                    
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function                      
>      `Pipe::tcp_read_nonblocking(char*, int)':                                  
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2129: undefined    
>      reference to `rrecv'                                                       
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function                      
>      `Pipe::tcp_read(char*, int)':                                              
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:2079: undefined    
>      reference to `rshutdown'                                                   
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::connect()':   
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:768: undefined     
>      reference to `rclose'                                                      
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:773: undefined     
>      reference to `rsocket'                                                     
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:781: undefined     
>      reference to `rconnect'                                                    
>      ./.libs/libglobal.a(libcommon_la-Pipe.o): In function `Pipe::writer()':    
>      /home/matt/Desktop/ceph-0.61.3-rsockets/src/msg/Pipe.cc:1471: undefined    
>      reference to `rwrite'                                                      
>      collect2: error: ld returned 1 exit status                                 
>      make[3]: *** [ceph-mon] Error 1                                            
>      From the looks of it I need to include the 'rdma/rsocket.h' library        
>      somewhere else or add librdmacm but I'm not sure where.                    
>      Full disclaimer, I am terrible at C++. If anyone has a few spare minutes   
>      to have a look into the error messages and can point out where I've gone   
>      wrong it would be greatly appreciated.                                     
>      I've put the code up at - [13]https://github.com/funkBuild/ceph-rsockets   
>      Thanks again                                                               
>      -Matt                                                                      
> 
> References
> 
>    Visible links
>    1. http://pastebin.com/WzMJNKZY
>    2. http://pastebin.com/3aXbjzqS
>    3. http://172.16.0.1:6789/0
>    4. http://172.16.0.1:6789/0
>    5. http://172.16.0.1:6789/0
>    6. http://172.16.0.1:6789/0
>    7. http://172.16.0.1:6789/0
>    8. http://172.16.0.1:6789/0
>    9. http://172.16.0.1:6789/0
>   10. http://172.16.0.1:6789/0
>   11. https://github.com/funkBuild/ceph-rsockets
>   12. mailto:manderson8787@gmail.com
>   13. https://github.com/funkBuild/ceph-rsockets

> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Help needed porting Ceph to RSockets

[Matthew Anderson]

Moving this conversation to ceph-devel where the dev's might be able
to shed some light on this.

I've added some additional debug to my code to narrow the issue down a
bit and the reader thread appears to be getting locked by
tcp_read_wait() because rpoll never returns an event when the socket
is shutdown. A hack way of proving this was to lower the timeout in
rpoll to 5 seconds. When command like 'ceph osd tree' completes you
can see it block for 5 seconds until rpoll times out and returns 0.
The reader thread is then able to join and the pipe can be reaped.

Ceph log is here - http://pastebin.com/rHK4vYLZ
Mon log is here - http://pastebin.com/WyAJEw0m

What's particularly weird is that the monitor receives a POLLHUP event
when the ceph command shuts down it's socket but the ceph command
never does. When using regular sockets both sides of the connection
receive a POLLIN | POLLHUP | POLRDHUP event when the sockets are shut
down. It would seem like there is a bug in rsockets that causes the
side that calls shutdown first not to receive the correct rpoll
events.

Can anyone comment on whether the above seems right?

Thanks all
-Matt


> On Tue, Aug 13, 2013 at 12:06 AM, Andreas Bluemle
> <andreas.bluemle@itxperts.de> wrote:
>>
>> Hi Matthew,
>>
>> I am not quite sure about the POLLRDHUP.
>> On the server side (ceph-mon), tcp_read_wait does see the
>> POLLHUP - which should be the indicator that the
>> the other side is shutting down.
>>
>> I have also taken a brief look at the client side (ceph mon stat).
>> It initiates a shutdown - but never finishes. See attached log file
>> from "ceph --log-file ceph-mon-stat.rsockets --debug-ms 30 mon stat".
>> I have also attached the corresponding log file for regualr TCP/IP
>> sockets.
>>
>> It looks to me that in the rsockets case, the reaper is able to cleanup
>> even though there is still sth. left to do - and hence the shutdown
>> never completes.
>>
>>
>> Best Regards
>>
>> Andreas Bluemle
>>
>>
>> On Mon, 12 Aug 2013 15:11:47 +0800
>> Matthew Anderson <manderson8787@gmail.com> wrote:
>>
>> > Hi Andreas,
>> >
>> > I think we're both working on the same thing, I've just changed the
>> > function calls over to rsockets in the source instead of using the
>> > pre-load library. It explains why we're having the exact same problem!
>> >
>> > From what I've been able to tell the entire problem revolves around
>> > rsockets not supporting POLLRDHUP. As far as I can tell the pipe will
>> > only be removed when tcp_read_wait returns -1. With rsockets it never
>> > receives the POLLRDHUP event after shutdown_socket() is called so the
>> > rpoll call blocks until timeout (900 seconds) and the pipe stays
>> > active.
>> >
>> > The question then would be how can we destroy a pipe without relying
>> > on POLLRDHUP? shutdown_socket() always gets called when the socket
>> > should be closed so could there might be a way to trick
>> > tcp_read_wait() into returning -1 by doing somethere in
>> > shutdown_socket() but I'm not sure how to go about it.
>> >
>> > Any ideas?
>> >

Re: [ceph-users] Help needed porting Ceph to RSockets

[Andreas Bluemle]

Hi Matthew,

I can confirm the beahviour whichi you describe.
I too believe that the problem is on the client side (ceph command).
My log files show the very same symptom, i.e. the client side
not being able to shutdown the pipes properly.

(Q: I had problems yesterday to send a mail to ceph-users list
with the log files attached to it because of the size of 
the attachments exceeding some limit; I hadnÄt been subscribed
to the list at that point. Is the uses of pastebin.com the better
way to provide such lengthy information in general?


Best Regards

Andreas Bluemle

On Tue, 13 Aug 2013 11:59:36 +0800
Matthew Anderson <manderson8787@gmail.com> wrote:

> Moving this conversation to ceph-devel where the dev's might be able
> to shed some light on this.
> 
> I've added some additional debug to my code to narrow the issue down
> a bit and the reader thread appears to be getting locked by
> tcp_read_wait() because rpoll never returns an event when the socket
> is shutdown. A hack way of proving this was to lower the timeout in
> rpoll to 5 seconds. When command like 'ceph osd tree' completes you
> can see it block for 5 seconds until rpoll times out and returns 0.
> The reader thread is then able to join and the pipe can be reaped.
> 
> Ceph log is here - http://pastebin.com/rHK4vYLZ
> Mon log is here - http://pastebin.com/WyAJEw0m
> 
> What's particularly weird is that the monitor receives a POLLHUP
> event when the ceph command shuts down it's socket but the ceph
> command never does. When using regular sockets both sides of the
> connection receive a POLLIN | POLLHUP | POLRDHUP event when the
> sockets are shut down. It would seem like there is a bug in rsockets
> that causes the side that calls shutdown first not to receive the
> correct rpoll events.
> 
> Can anyone comment on whether the above seems right?
> 
> Thanks all
> -Matt
> 
> 
> On Tue, Aug 13, 2013 at 12:06 AM, Andreas Bluemle <
> andreas.bluemle@itxperts.de> wrote:
> 
> > Hi Matthew,
> >
> > I am not quite sure about the POLLRDHUP.
> > On the server side (ceph-mon), tcp_read_wait does see the
> > POLLHUP - which should be the indicator that the
> > the other side is shutting down.
> >
> > I have also taken a brief look at the client side (ceph mon stat).
> > It initiates a shutdown - but never finishes. See attached log file
> > from "ceph --log-file ceph-mon-stat.rsockets --debug-ms 30 mon
> > stat". I have also attached the corresponding log file for regualr
> > TCP/IP sockets.
> >
> > It looks to me that in the rsockets case, the reaper is able to
> > cleanup even though there is still sth. left to do - and hence the
> > shutdown never completes.
> >
> >
> > Best Regards
> >
> > Andreas Bluemle
> >
> >
> > On Mon, 12 Aug 2013 15:11:47 +0800
> > Matthew Anderson <manderson8787@gmail.com> wrote:
> >
> > > Hi Andreas,
> > >
> > > I think we're both working on the same thing, I've just changed
> > > the function calls over to rsockets in the source instead of
> > > using the pre-load library. It explains why we're having the
> > > exact same problem!
> > >
> > > From what I've been able to tell the entire problem revolves
> > > around rsockets not supporting POLLRDHUP. As far as I can tell
> > > the pipe will only be removed when tcp_read_wait returns -1. With
> > > rsockets it never receives the POLLRDHUP event after
> > > shutdown_socket() is called so the rpoll call blocks until
> > > timeout (900 seconds) and the pipe stays active.
> > >
> > > The question then would be how can we destroy a pipe without
> > > relying on POLLRDHUP? shutdown_socket() always gets called when
> > > the socket should be closed so could there might be a way to trick
> > > tcp_read_wait() into returning -1 by doing somethere in
> > > shutdown_socket() but I'm not sure how to go about it.
> > >
> > > Any ideas?
> > >
> >



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
Heinrich Boell Strasse 88           Phone: (+49) 89 4317582
D-81829 Muenchen (Germany)          Mobil: (+49) 177 522 0151
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Help needed porting Ceph to RSockets

[Andreas Bluemle]

Hi Matthew,

I found a workaround for my (our) problem: in the librdmacm
code, rsocket.c, there is a global constant polling_time, which
is set to 10 microseconds at the moment.

I raise this to 10000 - and all of a sudden things work nicely.

I think we are looking at two issues here:
1. the thread structure of ceph messenger
   For a given socket connection, there are 3 threads of interest
   here: the main messenger thread, the Pipe::reader and the
   Pipe::writer.

   For a ceph client like the ceph admin command, I see the following
   sequence
     - the connection to the ceph monitor is created by the
       main messenger  thread, the Pipe::reader and
       Pipe::writer are instantiated.
     - the requested command is sent to the ceph monitor, the
       answer is read and printed
     - at this point the Pipe::reader already has called
       tcp_read_wait(), polling for more data or connection termination
     - after the response had been printed, the main loop calls the
       shutdown routines which in in turn shutdown() the socket

    There is some time between the last two steps - and this gap is
    long enough to open a race:

2. rpoll, ibv and poll
   the rpoll implementation in rsockets is split in 2 phases:
   - a busy loop which checks the state of the underlying ibv queue pair
   - the call to real poll() system call (i.e. the uverbs(?)
     implementation of poll() inside the kernel)

   The busy loop has a maximum duration of polling_time (10 microseconds
   by default) - and is able detect the local shutdown and returns a
   POLLHUP.

   The poll() system call (i.e. the uverbs implementation of poll() 
   in the kernel) does not detect the local shutdown - and only returns
   after the caller supplied timeout expires.

Increasing the rsockets polloing_time from 10 to 10000 microseconds
results in the rpoll to detect the local shutdown within the busy loop.

Decreasing the ceph "ms tcp read timeout" from the default of 900 to 5
seconds serves a similar purpose, but is much coarser.

From my understanding, the real issue is neither at the ceph nor at the
rsockets level: it is related to the uverbs kernel module.

An alternative way to address the current problem at the rsockets level
would be w re-write of the rpoll(): instead of the busy loop at the
beginning followed by the reall poll() call with the full user
specificed timeout value ("ms tcp read timeout" in our case), I would
embed the real poll()  into a loop, splitting the user specified timeout
into smaller portions and doing the rsockets specific rs_poll_check()
on every timeout of the real poll().


Best Regards

Andreas Bluemle
     

On Tue, 13 Aug 2013 07:53:12 +0200
Andreas Bluemle <andreas.bluemle@itxperts.de> wrote:

> Hi Matthew,
> 
> I can confirm the beahviour whichi you describe.
> I too believe that the problem is on the client side (ceph command).
> My log files show the very same symptom, i.e. the client side
> not being able to shutdown the pipes properly.
> 
> (Q: I had problems yesterday to send a mail to ceph-users list
> with the log files attached to it because of the size of 
> the attachments exceeding some limit; I hadnÄt been subscribed
> to the list at that point. Is the uses of pastebin.com the better
> way to provide such lengthy information in general?
> 
> 
> Best Regards
> 
> Andreas Bluemle
> 
> On Tue, 13 Aug 2013 11:59:36 +0800
> Matthew Anderson <manderson8787@gmail.com> wrote:
> 
> > Moving this conversation to ceph-devel where the dev's might be able
> > to shed some light on this.
> > 
> > I've added some additional debug to my code to narrow the issue down
> > a bit and the reader thread appears to be getting locked by
> > tcp_read_wait() because rpoll never returns an event when the socket
> > is shutdown. A hack way of proving this was to lower the timeout in
> > rpoll to 5 seconds. When command like 'ceph osd tree' completes you
> > can see it block for 5 seconds until rpoll times out and returns 0.
> > The reader thread is then able to join and the pipe can be reaped.
> > 
> > Ceph log is here - http://pastebin.com/rHK4vYLZ
> > Mon log is here - http://pastebin.com/WyAJEw0m
> > 
> > What's particularly weird is that the monitor receives a POLLHUP
> > event when the ceph command shuts down it's socket but the ceph
> > command never does. When using regular sockets both sides of the
> > connection receive a POLLIN | POLLHUP | POLRDHUP event when the
> > sockets are shut down. It would seem like there is a bug in rsockets
> > that causes the side that calls shutdown first not to receive the
> > correct rpoll events.
> > 
> > Can anyone comment on whether the above seems right?
> > 
> > Thanks all
> > -Matt
> > 
> > 
> > On Tue, Aug 13, 2013 at 12:06 AM, Andreas Bluemle <
> > andreas.bluemle@itxperts.de> wrote:
> > 
> > > Hi Matthew,
> > >
> > > I am not quite sure about the POLLRDHUP.
> > > On the server side (ceph-mon), tcp_read_wait does see the
> > > POLLHUP - which should be the indicator that the
> > > the other side is shutting down.
> > >
> > > I have also taken a brief look at the client side (ceph mon stat).
> > > It initiates a shutdown - but never finishes. See attached log
> > > file from "ceph --log-file ceph-mon-stat.rsockets --debug-ms 30
> > > mon stat". I have also attached the corresponding log file for
> > > regualr TCP/IP sockets.
> > >
> > > It looks to me that in the rsockets case, the reaper is able to
> > > cleanup even though there is still sth. left to do - and hence the
> > > shutdown never completes.
> > >
> > >
> > > Best Regards
> > >
> > > Andreas Bluemle
> > >
> > >
> > > On Mon, 12 Aug 2013 15:11:47 +0800
> > > Matthew Anderson <manderson8787@gmail.com> wrote:
> > >
> > > > Hi Andreas,
> > > >
> > > > I think we're both working on the same thing, I've just changed
> > > > the function calls over to rsockets in the source instead of
> > > > using the pre-load library. It explains why we're having the
> > > > exact same problem!
> > > >
> > > > From what I've been able to tell the entire problem revolves
> > > > around rsockets not supporting POLLRDHUP. As far as I can tell
> > > > the pipe will only be removed when tcp_read_wait returns -1.
> > > > With rsockets it never receives the POLLRDHUP event after
> > > > shutdown_socket() is called so the rpoll call blocks until
> > > > timeout (900 seconds) and the pipe stays active.
> > > >
> > > > The question then would be how can we destroy a pipe without
> > > > relying on POLLRDHUP? shutdown_socket() always gets called when
> > > > the socket should be closed so could there might be a way to
> > > > trick tcp_read_wait() into returning -1 by doing somethere in
> > > > shutdown_socket() but I'm not sure how to go about it.
> > > >
> > > > Any ideas?
> > > >
> > >
> 
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
Heinrich Boell Strasse 88           Phone: (+49) 89 4317582
D-81829 Muenchen (Germany)          Mobil: (+49) 177 522 0151
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Help needed porting Ceph to RSockets

[Atchley, Scott]

On Aug 13, 2013, at 10:06 AM, Andreas Bluemle <andreas.bluemle@itxperts.de> wrote:

> Hi Matthew,
> 
> I found a workaround for my (our) problem: in the librdmacm
> code, rsocket.c, there is a global constant polling_time, which
> is set to 10 microseconds at the moment.
> 
> I raise this to 10000 - and all of a sudden things work nicely.

I am adding the linux-rdma list to CC so Sean might see this.

If I understand what you are describing, the caller to rpoll() spins for up to 10 ms (10,000 us) before calling the real poll().

What is the purpose of the real poll() call? Is it simply a means to block the caller and avoid spinning? Or does it actually expect to detect an event?

> I think we are looking at two issues here:
> 1. the thread structure of ceph messenger
>   For a given socket connection, there are 3 threads of interest
>   here: the main messenger thread, the Pipe::reader and the
>   Pipe::writer.
> 
>   For a ceph client like the ceph admin command, I see the following
>   sequence
>     - the connection to the ceph monitor is created by the
>       main messenger  thread, the Pipe::reader and
>       Pipe::writer are instantiated.
>     - the requested command is sent to the ceph monitor, the
>       answer is read and printed
>     - at this point the Pipe::reader already has called
>       tcp_read_wait(), polling for more data or connection termination
>     - after the response had been printed, the main loop calls the
>       shutdown routines which in in turn shutdown() the socket
> 
>    There is some time between the last two steps - and this gap is
>    long enough to open a race:
> 
> 2. rpoll, ibv and poll
>   the rpoll implementation in rsockets is split in 2 phases:
>   - a busy loop which checks the state of the underlying ibv queue pair
>   - the call to real poll() system call (i.e. the uverbs(?)
>     implementation of poll() inside the kernel)
> 
>   The busy loop has a maximum duration of polling_time (10 microseconds
>   by default) - and is able detect the local shutdown and returns a
>   POLLHUP.
> 
>   The poll() system call (i.e. the uverbs implementation of poll() 
>   in the kernel) does not detect the local shutdown - and only returns
>   after the caller supplied timeout expires.
> 
> Increasing the rsockets polloing_time from 10 to 10000 microseconds
> results in the rpoll to detect the local shutdown within the busy loop.
> 
> Decreasing the ceph "ms tcp read timeout" from the default of 900 to 5
> seconds serves a similar purpose, but is much coarser.
> 
> From my understanding, the real issue is neither at the ceph nor at the
> rsockets level: it is related to the uverbs kernel module.
> 
> An alternative way to address the current problem at the rsockets level
> would be w re-write of the rpoll(): instead of the busy loop at the
> beginning followed by the reall poll() call with the full user
> specificed timeout value ("ms tcp read timeout" in our case), I would
> embed the real poll()  into a loop, splitting the user specified timeout
> into smaller portions and doing the rsockets specific rs_poll_check()
> on every timeout of the real poll().

I have not looked at the rsocket code, so take the following with a grain of salt. If the purpose of the real poll() is to simply block the user for a specified time, then you can simply make it a short duration (taking into consideration what granularity the OS provides) and then call ibv_poll_cq(). Keep in mind, polling will prevent your CPU from reducing power.

If the real poll() is actually checking for something (e.g. checking on the RDMA channel's fd or the IB channel's fd), then you may not want to spin too much.

Scott

> Best Regards
> 
> Andreas Bluemle
> 
> 
> On Tue, 13 Aug 2013 07:53:12 +0200
> Andreas Bluemle <andreas.bluemle-jy6uzZffzV0b1SvskN2V4Q@public.gmane.org> wrote:
> 
>> Hi Matthew,
>> 
>> I can confirm the beahviour whichi you describe.
>> I too believe that the problem is on the client side (ceph command).
>> My log files show the very same symptom, i.e. the client side
>> not being able to shutdown the pipes properly.
>> 
>> (Q: I had problems yesterday to send a mail to ceph-users list
>> with the log files attached to it because of the size of 
>> the attachments exceeding some limit; I hadnÄt been subscribed
>> to the list at that point. Is the uses of pastebin.com the better
>> way to provide such lengthy information in general?
>> 
>> 
>> Best Regards
>> 
>> Andreas Bluemle
>> 
>> On Tue, 13 Aug 2013 11:59:36 +0800
>> Matthew Anderson <manderson8787-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> 
>>> Moving this conversation to ceph-devel where the dev's might be able
>>> to shed some light on this.
>>> 
>>> I've added some additional debug to my code to narrow the issue down
>>> a bit and the reader thread appears to be getting locked by
>>> tcp_read_wait() because rpoll never returns an event when the socket
>>> is shutdown. A hack way of proving this was to lower the timeout in
>>> rpoll to 5 seconds. When command like 'ceph osd tree' completes you
>>> can see it block for 5 seconds until rpoll times out and returns 0.
>>> The reader thread is then able to join and the pipe can be reaped.
>>> 
>>> Ceph log is here - http://pastebin.com/rHK4vYLZ
>>> Mon log is here - http://pastebin.com/WyAJEw0m
>>> 
>>> What's particularly weird is that the monitor receives a POLLHUP
>>> event when the ceph command shuts down it's socket but the ceph
>>> command never does. When using regular sockets both sides of the
>>> connection receive a POLLIN | POLLHUP | POLRDHUP event when the
>>> sockets are shut down. It would seem like there is a bug in rsockets
>>> that causes the side that calls shutdown first not to receive the
>>> correct rpoll events.
>>> 
>>> Can anyone comment on whether the above seems right?
>>> 
>>> Thanks all
>>> -Matt
>>> 
>>> 
>>> On Tue, Aug 13, 2013 at 12:06 AM, Andreas Bluemle <
>>> andreas.bluemle-jy6uzZffzV0b1SvskN2V4Q@public.gmane.org> wrote:
>>> 
>>>> Hi Matthew,
>>>> 
>>>> I am not quite sure about the POLLRDHUP.
>>>> On the server side (ceph-mon), tcp_read_wait does see the
>>>> POLLHUP - which should be the indicator that the
>>>> the other side is shutting down.
>>>> 
>>>> I have also taken a brief look at the client side (ceph mon stat).
>>>> It initiates a shutdown - but never finishes. See attached log
>>>> file from "ceph --log-file ceph-mon-stat.rsockets --debug-ms 30
>>>> mon stat". I have also attached the corresponding log file for
>>>> regualr TCP/IP sockets.
>>>> 
>>>> It looks to me that in the rsockets case, the reaper is able to
>>>> cleanup even though there is still sth. left to do - and hence the
>>>> shutdown never completes.
>>>> 
>>>> 
>>>> Best Regards
>>>> 
>>>> Andreas Bluemle
>>>> 
>>>> 
>>>> On Mon, 12 Aug 2013 15:11:47 +0800
>>>> Matthew Anderson <manderson8787-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> 
>>>>> Hi Andreas,
>>>>> 
>>>>> I think we're both working on the same thing, I've just changed
>>>>> the function calls over to rsockets in the source instead of
>>>>> using the pre-load library. It explains why we're having the
>>>>> exact same problem!
>>>>> 
>>>>> From what I've been able to tell the entire problem revolves
>>>>> around rsockets not supporting POLLRDHUP. As far as I can tell
>>>>> the pipe will only be removed when tcp_read_wait returns -1.
>>>>> With rsockets it never receives the POLLRDHUP event after
>>>>> shutdown_socket() is called so the rpoll call blocks until
>>>>> timeout (900 seconds) and the pipe stays active.
>>>>> 
>>>>> The question then would be how can we destroy a pipe without
>>>>> relying on POLLRDHUP? shutdown_socket() always gets called when
>>>>> the socket should be closed so could there might be a way to
>>>>> trick tcp_read_wait() into returning -1 by doing somethere in
>>>>> shutdown_socket() but I'm not sure how to go about it.
>>>>> 
>>>>> Any ideas?
>>>>> 
>>>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
> Heinrich Boell Strasse 88           Phone: (+49) 89 4317582
> D-81829 Muenchen (Germany)          Mobil: (+49) 177 522 0151
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ceph-users] Help needed porting Ceph to RSockets

[Hefty, Sean]

> > I found a workaround for my (our) problem: in the librdmacm
> > code, rsocket.c, there is a global constant polling_time, which
> > is set to 10 microseconds at the moment.
> >
> > I raise this to 10000 - and all of a sudden things work nicely.
> 
> I am adding the linux-rdma list to CC so Sean might see this.
> 
> If I understand what you are describing, the caller to rpoll() spins for up to
> 10 ms (10,000 us) before calling the real poll().
> 
> What is the purpose of the real poll() call? Is it simply a means to block the
> caller and avoid spinning? Or does it actually expect to detect an event?

When the real poll() is called, an event is expected on an fd associated with the CQ's completion channel.
 
> > I think we are looking at two issues here:
> > 1. the thread structure of ceph messenger
> >   For a given socket connection, there are 3 threads of interest
> >   here: the main messenger thread, the Pipe::reader and the
> >   Pipe::writer.
> >
> >   For a ceph client like the ceph admin command, I see the following
> >   sequence
> >     - the connection to the ceph monitor is created by the
> >       main messenger  thread, the Pipe::reader and
> >       Pipe::writer are instantiated.
> >     - the requested command is sent to the ceph monitor, the
> >       answer is read and printed
> >     - at this point the Pipe::reader already has called
> >       tcp_read_wait(), polling for more data or connection termination
> >     - after the response had been printed, the main loop calls the
> >       shutdown routines which in in turn shutdown() the socket
> >
> >    There is some time between the last two steps - and this gap is
> >    long enough to open a race:
> >
> > 2. rpoll, ibv and poll
> >   the rpoll implementation in rsockets is split in 2 phases:
> >   - a busy loop which checks the state of the underlying ibv queue pair
> >   - the call to real poll() system call (i.e. the uverbs(?)
> >     implementation of poll() inside the kernel)
> >
> >   The busy loop has a maximum duration of polling_time (10 microseconds
> >   by default) - and is able detect the local shutdown and returns a
> >   POLLHUP.
> >
> >   The poll() system call (i.e. the uverbs implementation of poll()
> >   in the kernel) does not detect the local shutdown - and only returns
> >   after the caller supplied timeout expires.

It sounds like there's an issue here either with a message getting lost or a race.  Given that spinning longer works for you, it sounds like an event is getting lost, not being generated correctly, or not being configured to generate.

> > Increasing the rsockets polloing_time from 10 to 10000 microseconds
> > results in the rpoll to detect the local shutdown within the busy loop.
> >
> > Decreasing the ceph "ms tcp read timeout" from the default of 900 to 5
> > seconds serves a similar purpose, but is much coarser.
> >
> > From my understanding, the real issue is neither at the ceph nor at the
> > rsockets level: it is related to the uverbs kernel module.
> >
> > An alternative way to address the current problem at the rsockets level
> > would be w re-write of the rpoll(): instead of the busy loop at the
> > beginning followed by the reall poll() call with the full user
> > specificed timeout value ("ms tcp read timeout" in our case), I would
> > embed the real poll()  into a loop, splitting the user specified timeout
> > into smaller portions and doing the rsockets specific rs_poll_check()
> > on every timeout of the real poll().
> 
> I have not looked at the rsocket code, so take the following with a grain of
> salt. If the purpose of the real poll() is to simply block the user for a
> specified time, then you can simply make it a short duration (taking into
> consideration what granularity the OS provides) and then call ibv_poll_cq().
> Keep in mind, polling will prevent your CPU from reducing power.
> 
> If the real poll() is actually checking for something (e.g. checking on the
> RDMA channel's fd or the IB channel's fd), then you may not want to spin too
> much.

The real poll() call is intended to block the application until a timeout occurs or an event shows up.  Since increasing the spin time works for you, it makes me suspect that there is a bug in the CQ event handling in rsockets.
 
> >>> What's particularly weird is that the monitor receives a POLLHUP
> >>> event when the ceph command shuts down it's socket but the ceph
> >>> command never does. When using regular sockets both sides of the
> >>> connection receive a POLLIN | POLLHUP | POLRDHUP event when the
> >>> sockets are shut down. It would seem like there is a bug in rsockets
> >>> that causes the side that calls shutdown first not to receive the
> >>> correct rpoll events.

rsockets does not support POLLRDHUP.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Help needed porting Ceph to RSockets

[Andreas Bluemle]

Hi,

maybe some information about the environment I am
working in:

- CentOS 6.4 with custom kernel 3.8.13
- librdmacm / librspreload from git, tag 1.0.17
- application started with librspreload in LD_PRELOAD environment

Currently, I have increased the value of the spin time by setting the
default value for polling_time in the source code.

I guess that the correct way to do this is via
configuration in /etc/rdma/rsocket/polling_time?

Concerning the rpoll() itself, some more comments/questions
embedded below.

On Tue, 13 Aug 2013 21:44:42 +0000
"Hefty, Sean" <sean.hefty@intel.com> wrote:

> > > I found a workaround for my (our) problem: in the librdmacm
> > > code, rsocket.c, there is a global constant polling_time, which
> > > is set to 10 microseconds at the moment.
> > >
> > > I raise this to 10000 - and all of a sudden things work nicely.
> > 
> > I am adding the linux-rdma list to CC so Sean might see this.
> > 
> > If I understand what you are describing, the caller to rpoll()
> > spins for up to 10 ms (10,000 us) before calling the real poll().
> > 
> > What is the purpose of the real poll() call? Is it simply a means
> > to block the caller and avoid spinning? Or does it actually expect
> > to detect an event?
> 
> When the real poll() is called, an event is expected on an fd
> associated with the CQ's completion channel. 

The first question I would have is: why is the rpoll() split into
these two pieces? There must have been some reason to do a busy
loop on some local state information rather than just call the
real poll() directly.

> > > I think we are looking at two issues here:
> > > 1. the thread structure of ceph messenger
> > >   For a given socket connection, there are 3 threads of interest
> > >   here: the main messenger thread, the Pipe::reader and the
> > >   Pipe::writer.
> > >
> > >   For a ceph client like the ceph admin command, I see the
> > > following sequence
> > >     - the connection to the ceph monitor is created by the
> > >       main messenger  thread, the Pipe::reader and
> > >       Pipe::writer are instantiated.
> > >     - the requested command is sent to the ceph monitor, the
> > >       answer is read and printed
> > >     - at this point the Pipe::reader already has called
> > >       tcp_read_wait(), polling for more data or connection
> > > termination
> > >     - after the response had been printed, the main loop calls the
> > >       shutdown routines which in in turn shutdown() the socket
> > >
> > >    There is some time between the last two steps - and this gap is
> > >    long enough to open a race:
> > >
> > > 2. rpoll, ibv and poll
> > >   the rpoll implementation in rsockets is split in 2 phases:
> > >   - a busy loop which checks the state of the underlying ibv
> > > queue pair
> > >   - the call to real poll() system call (i.e. the uverbs(?)
> > >     implementation of poll() inside the kernel)
> > >
> > >   The busy loop has a maximum duration of polling_time (10
> > > microseconds by default) - and is able detect the local shutdown
> > > and returns a POLLHUP.
> > >
> > >   The poll() system call (i.e. the uverbs implementation of poll()
> > >   in the kernel) does not detect the local shutdown - and only
> > > returns after the caller supplied timeout expires.
> 
> It sounds like there's an issue here either with a message getting
> lost or a race.  Given that spinning longer works for you, it sounds
> like an event is getting lost, not being generated correctly, or not
> being configured to generate.
> 

I am looking at a multithreaded application here, and I believe that
the race is between thread A calling the rpoll() for POLLIN event and
thread B calling the shutdown(SHUT_RDWR) for reading and writing of
the (r)socket almost immediately afterwards.

I think that the shutdown itself does not cause a POLLHUP event to be
generated from the kernel to interupt the real poll().
(BTW: which kernel module implements the poll() for rsockets?
Is that ib_uverbs.ko with ib_uverbs_poll_cq()?)

> > > Increasing the rsockets polloing_time from 10 to 10000
> > > microseconds results in the rpoll to detect the local shutdown
> > > within the busy loop.
> > >
> > > Decreasing the ceph "ms tcp read timeout" from the default of 900
> > > to 5 seconds serves a similar purpose, but is much coarser.
> > >
> > > From my understanding, the real issue is neither at the ceph nor
> > > at the rsockets level: it is related to the uverbs kernel module.
> > >
> > > An alternative way to address the current problem at the rsockets
> > > level would be w re-write of the rpoll(): instead of the busy
> > > loop at the beginning followed by the reall poll() call with the
> > > full user specificed timeout value ("ms tcp read timeout" in our
> > > case), I would embed the real poll()  into a loop, splitting the
> > > user specified timeout into smaller portions and doing the
> > > rsockets specific rs_poll_check() on every timeout of the real
> > > poll().
> > 
> > I have not looked at the rsocket code, so take the following with a
> > grain of salt. If the purpose of the real poll() is to simply block
> > the user for a specified time, then you can simply make it a short
> > duration (taking into consideration what granularity the OS
> > provides) and then call ibv_poll_cq(). Keep in mind, polling will
> > prevent your CPU from reducing power.
> > 
> > If the real poll() is actually checking for something (e.g.
> > checking on the RDMA channel's fd or the IB channel's fd), then you
> > may not want to spin too much.
> 
> The real poll() call is intended to block the application until a
> timeout occurs or an event shows up.  Since increasing the spin time
> works for you, it makes me suspect that there is a bug in the CQ
> event handling in rsockets. 

> > >>> What's particularly weird is that the monitor receives a POLLHUP
> > >>> event when the ceph command shuts down it's socket but the ceph
> > >>> command never does. When using regular sockets both sides of the
> > >>> connection receive a POLLIN | POLLHUP | POLRDHUP event when the
> > >>> sockets are shut down. It would seem like there is a bug in
> > >>> rsockets that causes the side that calls shutdown first not to
> > >>> receive the correct rpoll events.
> 
> rsockets does not support POLLRDHUP.
> 

I don't think the issue is POLLRDHUP.
I think the issue is POLLHUP and/or POLLIN.
My impression is that on a local shutdown (r)socket, a poll
for POLLIN event should at least return a POLLIN event and
a subsequent read should return 0 bytes indicating EOF.
But the POLLIN is not generated from the layer below 
rsockets (ib_uverbs.ko?) as far as I can tell.

See also: http://www.greenend.org.uk/rjk/tech/poll.html

Best Regards

Andreas Bluemle

> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 




-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

Re: [ceph-users] Help needed porting Ceph to RSockets

[Atchley, Scott]

On Aug 14, 2013, at 3:21 AM, Andreas Bluemle <andreas.bluemle-jy6uzZffzV0b1SvskN2V4Q@public.gmane.org> wrote:

> Hi,
> 
> maybe some information about the environment I am
> working in:
> 
> - CentOS 6.4 with custom kernel 3.8.13
> - librdmacm / librspreload from git, tag 1.0.17
> - application started with librspreload in LD_PRELOAD environment
> 
> Currently, I have increased the value of the spin time by setting the
> default value for polling_time in the source code.
> 
> I guess that the correct way to do this is via
> configuration in /etc/rdma/rsocket/polling_time?
> 
> Concerning the rpoll() itself, some more comments/questions
> embedded below.
> 
> On Tue, 13 Aug 2013 21:44:42 +0000
> "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
>>>> I found a workaround for my (our) problem: in the librdmacm
>>>> code, rsocket.c, there is a global constant polling_time, which
>>>> is set to 10 microseconds at the moment.
>>>> 
>>>> I raise this to 10000 - and all of a sudden things work nicely.
>>> 
>>> I am adding the linux-rdma list to CC so Sean might see this.
>>> 
>>> If I understand what you are describing, the caller to rpoll()
>>> spins for up to 10 ms (10,000 us) before calling the real poll().
>>> 
>>> What is the purpose of the real poll() call? Is it simply a means
>>> to block the caller and avoid spinning? Or does it actually expect
>>> to detect an event?
>> 
>> When the real poll() is called, an event is expected on an fd
>> associated with the CQ's completion channel. 
> 
> The first question I would have is: why is the rpoll() split into
> these two pieces? There must have been some reason to do a busy
> loop on some local state information rather than just call the
> real poll() directly.

Sean can answer specifically, but this is a typical HPC technique. The worst thing you can do is handle an event and then block when the next event is available. This adds 1-3 us to latency which is unacceptable in HPC. In HPC, we poll. If we worry about power, we poll until we get no more events and then we poll a little more before blocking. Determining the "little more" is the fun part. ;-) 

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ceph-users] Help needed porting Ceph to RSockets

[Hefty, Sean]

> The first question I would have is: why is the rpoll() split into
> these two pieces? There must have been some reason to do a busy
> loop on some local state information rather than just call the
> real poll() directly.

As Scott mentioned in his email, this is done for performance reasons.  The cost of always dropping into the kernel is too high for HPC.

> I am looking at a multithreaded application here, and I believe that
> the race is between thread A calling the rpoll() for POLLIN event and
> thread B calling the shutdown(SHUT_RDWR) for reading and writing of
> the (r)socket almost immediately afterwards.

Ah - this is likely the issue.  I did not assume that rshutdown() would be called simultaneously with rpoll().  I need to think about how to solve this, so that rpoll() unblocks.
 
> I think that the shutdown itself does not cause a POLLHUP event to be
> generated from the kernel to interupt the real poll().
> (BTW: which kernel module implements the poll() for rsockets?
> Is that ib_uverbs.ko with ib_uverbs_poll_cq()?)

The POLLHUP event in rsockets is just software indicating that such an 'event' occurred - basically when a call to rpoll() detects that the rsocket state is disconnected.

I believe that the real poll() call traps into ib_uverbs_event_poll() in the kernel.  The fd associated with the poll call corresponds to a 'completion channel', which is used to report events which occur on a CQ.  Connection related events don't actually go to that fd - only completions for data transfers.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ceph-users] Help needed porting Ceph to RSockets

[Hefty, Sean]

> I am looking at a multithreaded application here, and I believe that
> the race is between thread A calling the rpoll() for POLLIN event and
> thread B calling the shutdown(SHUT_RDWR) for reading and writing of
> the (r)socket almost immediately afterwards.

I modified a test program, and I can reproduce the hang as you describe -- calling rpoll() then rshutdown() from another thread.

These calls end up calling rpoll->poll followed by rshutdown->read.  The read completes, which completes rshutdown, but the poll continues to wait for an event.  In the kernel, poll ends up in uverbs.c::ib_uverbs_event_poll, and read in uverbs.c::ib_uverbs_event_read().

The behavior of poll/read seems reasonable, so I don't think this is a kernel issue.  I'm still trying to figure out a simple solution to fixing this.

- Sean

RE: [ceph-users] Help needed porting Ceph to RSockets

[Hefty, Sean]

Can you see if the patch below fixes the hang?

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 src/rsocket.c |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/src/rsocket.c b/src/rsocket.c
index d544dd0..e45b26d 100644
--- a/src/rsocket.c
+++ b/src/rsocket.c
@@ -2948,10 +2948,12 @@ static int rs_poll_events(struct pollfd *rfds, struct pollfd *fds, nfds_t nfds)
 
 		rs = idm_lookup(&idm, fds[i].fd);
 		if (rs) {
+			fastlock_acquire(&rs->cq_wait_lock);
 			if (rs->type == SOCK_STREAM)
 				rs_get_cq_event(rs);
 			else
 				ds_get_cq_event(rs);
+			fastlock_release(&rs->cq_wait_lock);
 			fds[i].revents = rs_poll_rs(rs, fds[i].events, 1, rs_poll_all);
 		} else {
 			fds[i].revents = rfds[i].revents;
@@ -3098,7 +3100,8 @@ int rselect(int nfds, fd_set *readfds, fd_set *writefds,
 
 /*
  * For graceful disconnect, notify the remote side that we're
- * disconnecting and wait until all outstanding sends complete.
+ * disconnecting and wait until all outstanding sends complete, provided
+ * that the remote side has not sent a disconnect message.
  */
 int rshutdown(int socket, int how)
 {
@@ -3138,6 +3141,12 @@ int rshutdown(int socket, int how)
 	if (rs->state & rs_connected)
 		rs_process_cq(rs, 0, rs_conn_all_sends_done);
 
+	if (rs->state & rs_disconnected) {
+		/* Generate event by flushing receives to unblock rpoll */
+		ibv_req_notify_cq(rs->cm_id->recv_cq, 0);
+		rdma_disconnect(rs->cm_id);
+	}
+
 	if ((rs->fd_flags & O_NONBLOCK) && (rs->state & rs_connected))
 		rs_set_nonblocking(rs, rs->fd_flags);
 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Help needed porting Ceph to RSockets

[Andreas Bluemle]

Hi Sean,

I will re-check until the end of the week; there is
some test scheduling issue with our test system, which
affects my access times.

Thanks

Andreas


On Mon, 19 Aug 2013 17:10:11 +0000
"Hefty, Sean" <sean.hefty@intel.com> wrote:

> Can you see if the patch below fixes the hang?
> 
> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
> ---
>  src/rsocket.c |   11 ++++++++++-
>  1 files changed, 10 insertions(+), 1 deletions(-)
> 
> diff --git a/src/rsocket.c b/src/rsocket.c
> index d544dd0..e45b26d 100644
> --- a/src/rsocket.c
> +++ b/src/rsocket.c
> @@ -2948,10 +2948,12 @@ static int rs_poll_events(struct pollfd
> *rfds, struct pollfd *fds, nfds_t nfds) 
>  		rs = idm_lookup(&idm, fds[i].fd);
>  		if (rs) {
> +			fastlock_acquire(&rs->cq_wait_lock);
>  			if (rs->type == SOCK_STREAM)
>  				rs_get_cq_event(rs);
>  			else
>  				ds_get_cq_event(rs);
> +			fastlock_release(&rs->cq_wait_lock);
>  			fds[i].revents = rs_poll_rs(rs,
> fds[i].events, 1, rs_poll_all); } else {
>  			fds[i].revents = rfds[i].revents;
> @@ -3098,7 +3100,8 @@ int rselect(int nfds, fd_set *readfds, fd_set
> *writefds, 
>  /*
>   * For graceful disconnect, notify the remote side that we're
> - * disconnecting and wait until all outstanding sends complete.
> + * disconnecting and wait until all outstanding sends complete,
> provided
> + * that the remote side has not sent a disconnect message.
>   */
>  int rshutdown(int socket, int how)
>  {
> @@ -3138,6 +3141,12 @@ int rshutdown(int socket, int how)
>  	if (rs->state & rs_connected)
>  		rs_process_cq(rs, 0, rs_conn_all_sends_done);
>  
> +	if (rs->state & rs_disconnected) {
> +		/* Generate event by flushing receives to unblock
> rpoll */
> +		ibv_req_notify_cq(rs->cm_id->recv_cq, 0);
> +		rdma_disconnect(rs->cm_id);
> +	}
> +
>  	if ((rs->fd_flags & O_NONBLOCK) && (rs->state &
> rs_connected)) rs_set_nonblocking(rs, rs->fd_flags);
>  
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
Heinrich Boell Strasse 88           Phone: (+49) 89 4317582
D-81829 Muenchen (Germany)          Mobil: (+49) 177 522 0151

Re: [ceph-users] Help needed porting Ceph to RSockets

[Andreas Bluemle]

Hi,

I have added the patch and re-tested: I still encounter
hangs of my application. I am not quite sure whether the
I hit the same error on the shutdown because now I don't hit
the error always, but only every now and then.

WHen adding the patch to my code base (git tag v1.0.17) I notice
an offset of "-34 lines". Which code base are you using?


Best Regards

Andreas Bluemle

On Tue, 20 Aug 2013 09:21:13 +0200
Andreas Bluemle <andreas.bluemle@itxperts.de> wrote:

> Hi Sean,
> 
> I will re-check until the end of the week; there is
> some test scheduling issue with our test system, which
> affects my access times.
> 
> Thanks
> 
> Andreas
> 
> 
> On Mon, 19 Aug 2013 17:10:11 +0000
> "Hefty, Sean" <sean.hefty@intel.com> wrote:
> 
> > Can you see if the patch below fixes the hang?
> > 
> > Signed-off-by: Sean Hefty <sean.hefty@intel.com>
> > ---
> >  src/rsocket.c |   11 ++++++++++-
> >  1 files changed, 10 insertions(+), 1 deletions(-)
> > 
> > diff --git a/src/rsocket.c b/src/rsocket.c
> > index d544dd0..e45b26d 100644
> > --- a/src/rsocket.c
> > +++ b/src/rsocket.c
> > @@ -2948,10 +2948,12 @@ static int rs_poll_events(struct pollfd
> > *rfds, struct pollfd *fds, nfds_t nfds) 
> >  		rs = idm_lookup(&idm, fds[i].fd);
> >  		if (rs) {
> > +			fastlock_acquire(&rs->cq_wait_lock);
> >  			if (rs->type == SOCK_STREAM)
> >  				rs_get_cq_event(rs);
> >  			else
> >  				ds_get_cq_event(rs);
> > +			fastlock_release(&rs->cq_wait_lock);
> >  			fds[i].revents = rs_poll_rs(rs,
> > fds[i].events, 1, rs_poll_all); } else {
> >  			fds[i].revents = rfds[i].revents;
> > @@ -3098,7 +3100,8 @@ int rselect(int nfds, fd_set *readfds, fd_set
> > *writefds, 
> >  /*
> >   * For graceful disconnect, notify the remote side that we're
> > - * disconnecting and wait until all outstanding sends complete.
> > + * disconnecting and wait until all outstanding sends complete,
> > provided
> > + * that the remote side has not sent a disconnect message.
> >   */
> >  int rshutdown(int socket, int how)
> >  {
> > @@ -3138,6 +3141,12 @@ int rshutdown(int socket, int how)
> >  	if (rs->state & rs_connected)
> >  		rs_process_cq(rs, 0, rs_conn_all_sends_done);
> >  
> > +	if (rs->state & rs_disconnected) {
> > +		/* Generate event by flushing receives to unblock
> > rpoll */
> > +		ibv_req_notify_cq(rs->cm_id->recv_cq, 0);
> > +		rdma_disconnect(rs->cm_id);
> > +	}
> > +
> >  	if ((rs->fd_flags & O_NONBLOCK) && (rs->state &
> > rs_connected)) rs_set_nonblocking(rs, rs->fd_flags);
> >  
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-rdma" in the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> 
> 



-- 
Andreas Bluemle                     mailto:Andreas.Bluemle@itxperts.de
Heinrich Boell Strasse 88           Phone: (+49) 89 4317582
D-81829 Muenchen (Germany)          Mobil: (+49) 177 522 0151

RE: [ceph-users] Help needed porting Ceph to RSockets

[Hefty, Sean]

> I have added the patch and re-tested: I still encounter
> hangs of my application. I am not quite sure whether the
> I hit the same error on the shutdown because now I don't hit
> the error always, but only every now and then.

I guess this is at least some progress... :/
 
> WHen adding the patch to my code base (git tag v1.0.17) I notice
> an offset of "-34 lines". Which code base are you using?

This patch was generated against the tip of the git tree. 

Re: [ceph-users] Help needed porting Ceph to RSockets

[Matthew Anderson]

Hi Sean,

I tested out the patch and unfortunately had the same results as
Andreas. About 50% of the time the rpoll() thread in Ceph still hangs
when rshutdown() is called. I saw a similar behaviour when increasing
the poll time on the pre-patched version if that's of any relevance.

Thanks

On Tue, Aug 20, 2013 at 11:04 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> I have added the patch and re-tested: I still encounter
>> hangs of my application. I am not quite sure whether the
>> I hit the same error on the shutdown because now I don't hit
>> the error always, but only every now and then.
>
> I guess this is at least some progress... :/
>
>> WHen adding the patch to my code base (git tag v1.0.17) I notice
>> an offset of "-34 lines". Which code base are you using?
>
> This patch was generated against the tip of the git tree.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html