ucma question - backlog limit

ucma question - backlog limit

[Steve Wise]

Hey Sean,

What is the logic behind limiting the listen backlog to 128?

------

snipit from ucma.c:

enum {
         UCMA_MAX_BACKLOG        = 128
};


snipit from ucma_listen():

         ctx->backlog = cmd.backlog > 0 && cmd.backlog < UCMA_MAX_BACKLOG ?
                        cmd.backlog : UCMA_MAX_BACKLOG;
------



Thanks,

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ucma question - backlog limit

[Bernard Metzler]

I think that could be relaxed.
Running OFED (and softiwarp) on larger installations,
we already had to set this to some KByte...

Bernard.

linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 08/09/2010 04:34:31 PM:

> Hey Sean,
>
> What is the logic behind limiting the listen backlog to 128?
>
> ------
>
> snipit from ucma.c:
>
> enum {
>          UCMA_MAX_BACKLOG        = 128
> };
>
>
> snipit from ucma_listen():
>
>          ctx->backlog = cmd.backlog > 0 && cmd.backlog <
UCMA_MAX_BACKLOG ?
>                         cmd.backlog : UCMA_MAX_BACKLOG;
> ------
>
>
>
> Thanks,
>
> Steve.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ucma question - backlog limit

[Steve Wise]

I agree.  I think it should default to say 1024.  Further, I propose we 
make this a sysctl variable such that admins can tweak it as needed.



On 08/09/2010 12:06 PM, Bernard Metzler wrote:
> I think that could be relaxed.
> Running OFED (and softiwarp) on larger installations,
> we already had to set this to some KByte...
>
> Bernard.
>
> linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 08/09/2010 04:34:31 PM:
>
>    
>> Hey Sean,
>>
>> What is the logic behind limiting the listen backlog to 128?
>>
>> ------
>>
>> snipit from ucma.c:
>>
>> enum {
>>           UCMA_MAX_BACKLOG        = 128
>> };
>>
>>
>> snipit from ucma_listen():
>>
>>           ctx->backlog = cmd.backlog>  0&&  cmd.backlog<
>>      
> UCMA_MAX_BACKLOG ?
>    
>>                          cmd.backlog : UCMA_MAX_BACKLOG;
>> ------
>>
>>
>>
>> Thanks,
>>
>> Steve.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ucma question - backlog limit

[Hefty, Sean]

> I agree.  I think it should default to say 1024.  Further, I propose we
> make this a sysctl variable such that admins can tweak it as needed.

The default is based on:

#define SOMAXCONN       128

I have no objection to making it adjustable.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ucma question - backlog limit

[Steve Wise]

On 08/09/2010 12:51 PM, Hefty, Sean wrote:
>> I agree.  I think it should default to say 1024.  Further, I propose we
>> make this a sysctl variable such that admins can tweak it as needed.
>>      
> The default is based on:
>
> #define SOMAXCONN       128
>
> I have no objection to making it adjustable.
>
> - Sean
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    


For TCP, a dropped SYN will cause the peer to retransmit the SYN, and 
the connection will be eventually established.  So a small backlog limit 
is probably ok for TCP.  With iWARP, the connect request is TCP payload 
an on established connection.  So the connection request is transmitted 
and acked at the TCP level by the time the connect request gets dropped 
in the ucma.  What finally happens is the peer times out (after many 
seconds typically) and kills the connection.  Further, a 32 node 256NP 
OpenMPI job will generate > 128 connect requests on some ranks.

So I think its reasonable to up default max value.   I'll post a patch 
to change the max and make it a sysctl variable as well.

Ok?

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ucma question - backlog limit

[Hefty, Sean]

> For TCP, a dropped SYN will cause the peer to retransmit the SYN, and
> the connection will be eventually established.  So a small backlog limit
> is probably ok for TCP.  With iWARP, the connect request is TCP payload
> an on established connection.  So the connection request is transmitted
> and acked at the TCP level by the time the connect request gets dropped
> in the ucma.  What finally happens is the peer times out (after many
> seconds typically) and kills the connection.  Further, a 32 node 256NP
> OpenMPI job will generate > 128 connect requests on some ranks.
> 
> So I think its reasonable to up default max value.   I'll post a patch
> to change the max and make it a sysctl variable as well.
> 
> Ok?

Yes - I think that makes sense.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ucma question - backlog limit

[Bernard Metzler]

Agreed.

Currently, if running out of backlog, the cm actively
rejects the connection with -ENOMEM back to the provider
(see ucma_event_handler(), and cm_conn_req_handler()).
That should allow the provider to zap the TCP connection,
but its not very productive ;)

Bernard.

Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote on 08/09/2010 08:33:18 PM:

> On 08/09/2010 12:51 PM, Hefty, Sean wrote:
> >> I agree.  I think it should default to say 1024.  Further, I propose
we
> >> make this a sysctl variable such that admins can tweak it as needed.
> >>
> > The default is based on:
> >
> > #define SOMAXCONN       128
> >
> > I have no objection to making it adjustable.
> >
> > - Sean
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
>
> For TCP, a dropped SYN will cause the peer to retransmit the SYN, and
> the connection will be eventually established.  So a small backlog limit
> is probably ok for TCP.  With iWARP, the connect request is TCP payload
> an on established connection.  So the connection request is transmitted
> and acked at the TCP level by the time the connect request gets dropped
> in the ucma.  What finally happens is the peer times out (after many
> seconds typically) and kills the connection.  Further, a 32 node 256NP
> OpenMPI job will generate > 128 connect requests on some ranks.
>
> So I think its reasonable to up default max value.   I'll post a patch
> to change the max and make it a sysctl variable as well.
>
> Ok?
>
> Steve.
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html