Re: SO_REUSEPORT - can it be done in kernel?

Herbert Xu
2011-02-26T00:57:52Z

David Miller <davem@davemloft.net> wrote:
>
> I think this is fundamentally a bind problem as well.

I'm fairly certain the bottleneck is indeed in the kernel, and
in the UDP stack in particular.

This is born out by a test where I used two named worker threads,
both working on the same socket. Stracing shows that they're
working flat out only doing sendmsg/recvmsg.

The result was that they obtained (in aggregate) half the throughput
of a single worker thread.

I then retested by having them use two sockets and the performance
greatly improved.

Now this is actually expected since our UDP stack is essentially
single-threaded on the send side when only one socket is being
used, mostly due to the corking functionality.

I'm unsure how big a role the receive side scalability actually
plays in this case, but I suspect it isn't great.

Which is why I'm quite skeptical about this REUSEPORT patch as
IMHO the only reason it produces a great result is solely because
it is allowing parallel sends going out.

Rather than modifying all UDP applications out there to fix what
is fundamentally a kernel problem, I think what we should do is
fix the UDP stack so that it actually scales.

It isn't all that hard since the easy way would be to only take
the lock if we're already corked or about to cork.

For the receive side we also don't need REUSEPORT as we can simply
make our UDP stack multiqueue.

Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt