Re: SO_REUSEPORT?

[Rick Jones <rick.jones2@hp.com>]

> I'm not sure that's applicable for us since the server application and
> networking will max out all the CPUs on host anyway; one way or
> another we need to dispatch the work of incoming connections to
> threads on different CPUs.  If we do this in user space and do all
> accepts in one thread, the CPU of that  thread becomes the bottleneck
> (we're accepting about 40,000 connections per second).  If we have
> multiple accept threads running on different CPUs, this helps some,
> but the load is spread unevenly across the CPUs and we still can't get
> the highest connection rate.  So it seems we're looking for a method
> that distributes the incoming connection load across CPUs pretty
> evenly.

Well, if you _really_ want the load spread, you may need to use a 
multiqueue (at least inbound if not also later outbound) interface, 
"know" how the NIC will hash and then have N distinct port numbers each 
assigned to a LISTEN endpoint.  The old song and dance about making an N 
CPU system look as much like N single-CPU systems and all that...

Unless there are NICs you can "tell" where to send the interrupts, which 
IMO is preferable - I have a preference for the application/scheduler 
telling "networking" where to work rather than networking (or the NIC) 
telling the scheduler where to run a thread - the archives of either 
here or netnews will probalby pull-up stuff were I've talked about 
Inbound Packet Scheduling (IPS) vs Thread Optimized Packet Scheduling 
(TOPS) and limitations of simplistic address hashing to pick a 
queue/processor/whatnot :)

rick jones