rsockets and standard socket based TCP benchmarks

rsockets and standard socket based TCP benchmarks

[Sridhar Samudrala]

We have started looking into rsockets and here are our initial 
experiences and test
results using socket based standard benchmarks over RDMA using rsockets.

netperf
- By default netserver forks a child process for each netperf client. As 
rsockets
   doesn't support fork() yet, this doesn't work.
- netserver has a -f option to disable forking a child and handle 1 
netperf client at a time
   Using this option, data transfer completed successfully, but the 
client blocked on recv()
   even after the connection is closed on the other side. Here is the 
stack trace.

#0  0x000000805caa77e4 in __read_nocancel () from /lib64/libc.so.6
#1  0x00000fffa4543d48 in .read () from 
/home/sridhar/librdmacm/src/preload.so
#2  0x000000805ccfe68c in .ibv_get_cq_event () from 
/usr/lib64/libibverbs.so.1
#3  0x00000fffa456a534 in rs_get_cq_event (rs=0x100203105f0)
     at src/rsocket.c:825
#4  0x00000fffa456bb48 in rs_process_cq (rs=0x100203105f0,
     nonblock=<value optimized out>,
     test=@0xfffa457e7d8: 0xfffa45697b0 <rs_have_rdata>) at 
src/rsocket.c:894
#5  0x00000fffa456d4e0 in rrecv (socket=<value optimized out>,
     buf=0xffff97e15f8, len=1, flags=<value optimized out>)
     at src/rsocket.c:1007
#6  0x00000fffa4543a08 in .recv () from 
/home/sridhar/librdmacm/src/preload.so
#7  0x00000000100497f4 in .disconnect_data_socket ()
#8  0x000000001004c528 in .send_omni_inner ()
#9  0x00000000100502e0 in .send_tcp_stream ()
#10 0x00000000100028a4 in .main ()

iperf
- Successfully got iperf working using rsockets via preload library.
   With default 128K message sizes, on P6 systems with Mellanox ConnectX 
cards, here are
   some test results for comparision.
         IPoIB  : 1.5Gb/s
         IPoIB Connected Mode: 5.5Gb/s
         rsockets using RDMA : 8.9Gb/s
         ib_write_bw(native RDMA using ibverbs): 12Gb/s

Thanks
Sridhar

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Hefty, Sean]

> netperf
> - By default netserver forks a child process for each netperf client. As
> rsockets
>    doesn't support fork() yet, this doesn't work.

Btw, fork() is unlikely to work anytime soon, if ever...

> - netserver has a -f option to disable forking a child and handle 1
> netperf client at a time
>    Using this option, data transfer completed successfully, but the
> client blocked on recv()
>    even after the connection is closed on the other side. Here is the
> stack trace.
> 
> #0  0x000000805caa77e4 in __read_nocancel () from /lib64/libc.so.6
> #1  0x00000fffa4543d48 in .read () from
> /home/sridhar/librdmacm/src/preload.so
> #2  0x000000805ccfe68c in .ibv_get_cq_event () from
> /usr/lib64/libibverbs.so.1
> #3  0x00000fffa456a534 in rs_get_cq_event (rs=0x100203105f0)
>      at src/rsocket.c:825
> #4  0x00000fffa456bb48 in rs_process_cq (rs=0x100203105f0,
>      nonblock=<value optimized out>,
>      test=@0xfffa457e7d8: 0xfffa45697b0 <rs_have_rdata>) at
> src/rsocket.c:894
> #5  0x00000fffa456d4e0 in rrecv (socket=<value optimized out>,
>      buf=0xffff97e15f8, len=1, flags=<value optimized out>)
>      at src/rsocket.c:1007
> #6  0x00000fffa4543a08 in .recv () from
> /home/sridhar/librdmacm/src/preload.so
> #7  0x00000000100497f4 in .disconnect_data_socket ()
> #8  0x000000001004c528 in .send_omni_inner ()
> #9  0x00000000100502e0 in .send_tcp_stream ()
> #10 0x00000000100028a4 in .main ()

Thanks - I'll look into this.  Do you know offhand if the server calls shutdown/close?

I do know that abrupt shutdown is an area that needs work, along with real support for KEEPALIVE.

> iperf
> - Successfully got iperf working using rsockets via preload library.
>    With default 128K message sizes, on P6 systems with Mellanox ConnectX
> cards, here are
>    some test results for comparision.
>          IPoIB  : 1.5Gb/s
>          IPoIB Connected Mode: 5.5Gb/s
>          rsockets using RDMA : 8.9Gb/s
>          ib_write_bw(native RDMA using ibverbs): 12Gb/s

Note that I've recently added additional controls to rsockets to adjust the size of the QP and inline data.  I haven't posted these patches yet (still testing), but I've been able to use them to increase bandwidth that I see on my systems from 24 Gbps to 26 Gbps.  I'll add both netperf and iperf to my list of apps to test.  From the above trace, it looks like netperf uses blocking calls.  I'm not sure about iperf, but the use of blocking calls impacts rsocket performance considerably.  I'm actively working on trying to reduce that impact.  rstream uses nonblocking calls and should be able to come close to ib_write_bw() performance.

Thanks for the feedback!

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rsockets and standard socket based TCP benchmarks

[Sridhar Samudrala]

On 5/23/2012 4:16 PM, Hefty, Sean wrote:
>> - netserver has a -f option to disable forking a child and handle 1
>> netperf client at a time
>>     Using this option, data transfer completed successfully, but the
>> client blocked on recv()
>>     even after the connection is closed on the other side. Here is the
>> stack trace.
>>
>> #0  0x000000805caa77e4 in __read_nocancel () from /lib64/libc.so.6
>> #1  0x00000fffa4543d48 in .read () from
>> /home/sridhar/librdmacm/src/preload.so
>> #2  0x000000805ccfe68c in .ibv_get_cq_event () from
>> /usr/lib64/libibverbs.so.1
>> #3  0x00000fffa456a534 in rs_get_cq_event (rs=0x100203105f0)
>>       at src/rsocket.c:825
>> #4  0x00000fffa456bb48 in rs_process_cq (rs=0x100203105f0,
>>       nonblock=<value optimized out>,
>>       test=@0xfffa457e7d8: 0xfffa45697b0<rs_have_rdata>) at
>> src/rsocket.c:894
>> #5  0x00000fffa456d4e0 in rrecv (socket=<value optimized out>,
>>       buf=0xffff97e15f8, len=1, flags=<value optimized out>)
>>       at src/rsocket.c:1007
>> #6  0x00000fffa4543a08 in .recv () from
>> /home/sridhar/librdmacm/src/preload.so
>> #7  0x00000000100497f4 in .disconnect_data_socket ()
>> #8  0x000000001004c528 in .send_omni_inner ()
>> #9  0x00000000100502e0 in .send_tcp_stream ()
>> #10 0x00000000100028a4 in .main ()
> Thanks - I'll look into this.  Do you know offhand if the server calls shutdown/close?
looks like the server is doing a shutdown().
but i have seen the same issue even with a simple tcp client/server test 
that does a close().
>
> I do know that abrupt shutdown is an area that needs work, along with real support for KEEPALIVE.
>
>> iperf
>> - Successfully got iperf working using rsockets via preload library.
>>     With default 128K message sizes, on P6 systems with Mellanox ConnectX
>> cards, here are
>>     some test results for comparision.
>>           IPoIB  : 1.5Gb/s
>>           IPoIB Connected Mode: 5.5Gb/s
>>           rsockets using RDMA : 8.9Gb/s
>>           ib_write_bw(native RDMA using ibverbs): 12Gb/s
> Note that I've recently added additional controls to rsockets to adjust the size of the QP and inline data.  I haven't posted these patches yet (still testing), but I've been able to use them to increase bandwidth that I see on my systems from 24 Gbps to 26 Gbps.  I'll add both netperf and iperf to my list of apps to test.  From the above trace, it looks like netperf uses blocking calls.  I'm not sure about iperf, but the use of blocking calls impacts rsocket performance considerably.  I'm actively working on trying to reduce that impact.  rstream uses nonblocking calls and should be able to come close to ib_write_bw() performance.
Even iperf seems to be doing blocking calls.

Thanks
Sridhar

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Vivek Kashyap]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1268 bytes --]

On Wed, 23 May 2012, Hefty, Sean wrote:

> > netperf
> > - By default netserver forks a child process for each netperf client. As
> > rsockets
> >    doesn't support fork() yet, this doesn't work.
> 
> Btw, fork() is unlikely to work anytime soon, if ever...

rsockets() is a perfect as a wrapper providing Sockets over RDMA, and so is
very attractive. Really nice work.

But to map standard networking applications to rsockets we will run into the
above problem i.e. fork() will not work.  It would be very useful to 
allow for the standard networking paradigm of: bind()->listen()->accept()-
->fork(), and then the server goes back to accept(). That would allow us to 
use rsockets transparently with existing applications.

Is it possible to support fork() for this standard model where the parent
closes its copy of the newly created socket and goes back to waiting on the
original socket in accept, and the child continues with the newly created
socket?

Have you considered some other options to make this happen?

Could we  skip creating the endpoint in raccept(), but actually create the
new endpoint during the frist receive or send? This would then allow the
fork() to occur without the complications with the pinned data.

thoughts?

thanks
         Vivek

RE: rsockets and standard socket based TCP benchmarks

[Hefty, Sean]

> But to map standard networking applications to rsockets we will run into the
> above problem i.e. fork() will not work.  It would be very useful to
> allow for the standard networking paradigm of: bind()->listen()->accept()-
> ->fork(), and then the server goes back to accept(). That would allow us to
> use rsockets transparently with existing applications.

I'm aware that fork() support does not work.  The problems lie within the RDMA stack, and maybe there's some way around this, but I'm not sure what it would be at this point.  I can't say that I fully grasp all the low level issues involved in 'fixing' it.

> Is it possible to support fork() for this standard model where the parent
> closes its copy of the newly created socket and goes back to waiting on the
> original socket in accept, and the child continues with the newly created
> socket?

IMO, it seems that if we can limit the usage model to what you're describing, then we may at least have a chance at supporting fork().

> Have you considered some other options to make this happen?
> 
> Could we  skip creating the endpoint in raccept(), but actually create the
> new endpoint during the frist receive or send? This would then allow the
> fork() to occur without the complications with the pinned data.

That's an interesting possibility.  I need to give something like this more thought, but establishing a connection over a normal socket, then using that to migrate communication to a QP on the first data exchange may allow for fork support.  That would have an effect on trying to handle fallback support (i.e. using standard sockets when RDMA fails), plus may have other issues, but there may some be ideas here worth considering.

- Sean 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Vivek Kashyap]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2503 bytes --]

On Sat, 9 Jun 2012, Hefty, Sean wrote:

>> But to map standard networking applications to rsockets we will run into the
>> above problem i.e. fork() will not work.  It would be very useful to
>> allow for the standard networking paradigm of: bind()->listen()->accept()-
>> ->fork(), and then the server goes back to accept(). That would allow us to
>> use rsockets transparently with existing applications.
>
> I'm aware that fork() support does not work.  The problems lie within the RDMA stack, and maybe there's some way around this, but I'm not sure what it would be at this point.  I can't say that I fully grasp all the low level issues involved in 'fixing' it.

Yeah..I looked a bit inot it. It seems easy enough to, within this restricted
paradigm, to use shared memory for the data buffer and share it between the
parent and the child. The QP/CQ accesses are not that straightforward and I 
need to understand more before I can suggest anything myself.


>
>> Is it possible to support fork() for this standard model where the parent
>> closes its copy of the newly created socket and goes back to waiting on the
>> original socket in accept, and the child continues with the newly created
>> socket?
>
> IMO, it seems that if we can limit the usage model to what you're describing, then we may at least have a chance at supporting fork().
>
>> Have you considered some other options to make this happen?
>>
>> Could we  skip creating the endpoint in raccept(), but actually create the
>> new endpoint during the frist receive or send? This would then allow the
>> fork() to occur without the complications with the pinned data.
>
> That's an interesting possibility.  I need to give something like this more thought, but establishing a connection over a normal socket, then using that to migrate communication to a QP on the first data exchange may allow for fork support.

yes.


>That would have an effect on trying to handle fallback support (i.e. using standard sockets when RDMA fails), plus may have other issues, but there may some be ideas here worth considering.
>

Though one can consider the fall-back in reverse order i.e. if the 
rdma connection fails  continue with the already established connection (over 
the normal inet socket).

Vivek

> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

RE: rsockets and standard socket based TCP benchmarks

[Hefty, Sean]

> Though one can consider the fall-back in reverse order i.e. if the
> rdma connection fails  continue with the already established connection (over
> the normal inet socket).

When I consider fallback, one of the issues is handling the case where one of the two sides is not using rsockets.  This prevents us from injecting new traffic into the TCP stream.

Still, if there's a way for rsockets to support fork for a significant number of apps, I'd be willing to live with these trade-offs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Vivek Kashyap]

On Mon, 11 Jun 2012, Hefty, Sean wrote:

>> Though one can consider the fall-back in reverse order i.e. if the
>> rdma connection fails  continue with the already established connection (over
>> the normal inet socket).
>
> When I consider fallback, one of the issues is handling the case where one of the two sides is not using rsockets.  This prevents us from injecting new traffic into the TCP stream.
>

Yes, good point. If the other side does not have rsockets then it is not that
straightforward.

Some thoughts:

1. The best option might be if we exchanged an option during
connection setup. This tells the peers if the other side is capable of RDMA.
If it is then one can switch to support of RDMA.

One obvious area of capability exchange is with TCP option. This requires TCP
extension though.

Is there any other mechanism to exchange capabilities?


2. In IPoIB, the address resolution returns RC or UD flags. This can allow 
one to determine if the remote peer is capabile of RDMA i.e. use of RC but 
does not tell us if rsockets is present on the peer. 
Can one still attmept an rsockets based connection over RC mode?


3. Use a service discovery mechanism to determine if the remote peer supports
rsockets. This will need to determine if the remote port is enabled; not just 
that rsockets is present on the system.

thanks
Vivek



> Still, if there's a way for rsockets to support fork for a significant number of apps, I'd be willing to live with these trade-offs.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Hefty, Sean]

> Yes, good point. If the other side does not have rsockets then it is not that
> straightforward.
> 
> Some thoughts:
> 
> 1. The best option might be if we exchanged an option during
> connection setup. This tells the peers if the other side is capable of RDMA.
> If it is then one can switch to support of RDMA.
> 
> One obvious area of capability exchange is with TCP option. This requires TCP
> extension though.
> 
> Is there any other mechanism to exchange capabilities?
> 
> 
> 2. In IPoIB, the address resolution returns RC or UD flags. This can allow
> one to determine if the remote peer is capabile of RDMA i.e. use of RC but
> does not tell us if rsockets is present on the peer.
> Can one still attmept an rsockets based connection over RC mode?
> 
> 
> 3. Use a service discovery mechanism to determine if the remote peer supports
> rsockets. This will need to determine if the remote port is enabled; not just
> that rsockets is present on the system.

Just to be clear, rsockets has limited fallback support today, mainly targeting the client side of the connection.

Another option, which I think ends up being a requirement for iWarp, is to use a port mapper function.  But I'd also be willing to live without fallback support on apps that require fork support.

Regarding the choices above, I have given thought to introducing a service for supporting rsocket UDP (to handle UDP port to QP mappings).  Maybe that would give us the most flexibility to solving this and all future problems.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Vivek Kashyap]

>
>> Have you considered some other options to make this happen?
>>
>> Could we  skip creating the endpoint in raccept(), but actually create the
>> new endpoint during the frist receive or send? This would then allow the
>> fork() to occur without the complications with the pinned data.
>
> That's an interesting possibility.  I need to give something like this more thought, but establishing a connection over a normal socket, then using that to migrate communication to a QP on the first data exchange may allow for fork support.  That would have an effect on trying to handle fallback support (i.e. using standard sockets when RDMA fails), plus may have other issues, but there may some be ideas here worth considering.
>
Hi Sean,

Have you had a chance to look more into the above for fork() support?

thanks
Vivek

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rsockets and standard socket based TCP benchmarks

[Hefty, Sean]

> Have you had a chance to look more into the above for fork() support?

Actually, I just started working on it last Friday.  I'll post a patch once I have at least something working.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html