Re: SO_REUSEPORT?

[Stephen Hemminger <stephen.hemminger@vyatta.com>]

On Thu, 07 Aug 2008 11:17:55 -0700
Rick Jones <rick.jones2@hp.com> wrote:

> Tom Herbert wrote:
> >>>We are looking at ways to scale TCP listeners.  I think we like is the
> >>>ability to listen on a port from multiple threads (sockets bound to
> >>>same port,  INADDR_ANY, and no interface binding) , which is what
> >>>SO_REUSEPORT would seem to allow.  Has this ever been implemented for
> >>>Linux or is there a good reason not to have it?
> >>
> >>On Linux, SO_REUSEADDR provide most of what SO_REUSEPORT provides on BSD.
> >>
> >>In any case, there is absolutely no point in creating multiple TCP listeners.
> >>Multiple threads can accept() on the same listener - at the same time.
> >>
> > 
> > 
> > We've been doing that, but then on wakeup it would seem that we're at
> > the mercy of scheduling-- basically which ever threads wakes up first
> > will get to process accept queue first.  This seems to bias towards
> > threads running on the same CPU as the wakeup is called, and   so this
> > method doesn't give us an even distribution of new connections across
> > the threads that we'd like.
> 
> How would the presence of multiple TCP LISTEN endpoints change that? 
> You'd then be at the mercy of whatever "scheduling" there was inside the 
> stack.
> 
> If you want to balance the threads, perhaps a dispatch thread, or a 
> virtual one - each thread knows how many connections it is servicing, 
> let them know how many the other threads are servicing, and if a thread 
> has N more connections than the other threads have it not go into 
> accept() that time around.  Might need some tweaking to handle 
> pathological starvation cases like all the other threads are hung I 
> suppose but the basic idea is there.
> 
> rick jones

I suspect thread balancing would actually hurt performance!
You would be better off to have a couple of "hot" threads that are doing
all the work and stay in cache. If you push the work around to all the
threads, you have worst case cache behaviour.