Re: Kernel BUG: epoll_wait() (and epoll_pwait) stall for 206 ms per call on sockets with a small-ish snd/rcv buffer.

[Carlo Wood <carlo@alinoe.com>]

While investigating this further, I read on
http://www.masterraghu.com/subjects/np/introduction/unix_network_programming_v1.3/ch07lev1sec5.html
under "SO_RCVBUF and SO_SNDBUF Socket Options":

    When setting the size of the TCP socket receive buffer, the
    ordering of the function calls is important. This is because of
    TCP's window scale option (Section 2.6), which is exchanged with
    the peer on the SYN segments when the connection is established.
    For a client, this means the SO_RCVBUF socket option must be set
    before calling connect. For a server, this means the socket option
    must be set for the listening socket before calling listen. Setting
    this option for the connected socket will have no effect whatsoever
    on the possible window scale option because accept does not return
    with the connected socket until TCP's three-way handshake is
    complete. That is why this option must be set for the listening
    socket. (The sizes of the socket buffers are always inherited from
    the listening socket by the newly created connected socket: pp.
    462–463 of TCPv2.)

As mentioned in a previous post, I had already discovered about
needing to set the socket buffers before connect, but I didn't know
about setting them before the call to listen() in order to get the
buffer sizes inherited by the accepted sockets.

After fixing this in my test program, all problems disappeared when
keeping the send and receive buffers the same on both sides.

However, when only setting the send and receive buffers on the client
socket (not on the (accepted or) listen socket), epoll_wait() still
stalls 43ms. When the SO_SNDBUF is smaller than 33182 bytes.

Here is the latest version of my test program:

https://github.com/CarloWood/ai-evio-testsuite/blob/master/src/epoll_bug.c

I have to retract most of my "bug" report, it might even not really be
a bug then... but nevertheless, what remains strange is the fact
that setting the socket buffer sizes on the accepted sockets can lead
to so much crippling effect, while the quoted website states:

    Setting this option for the connected socket will have no effect
    whatsoever on the possible window scale option because accept does
    not return with the connected socket until TCP's three-way
    handshake is complete.

And when only setting the socket buffer sizes for the client socket
(that I use to send back received data; so this is the sending
side now) then why does epoll_wait() stall 43 ms per call when the
receiving side is using the default (much larger) socket buffer sizes?

That 43 ms is STILL crippling-- slowing down the transmission of the
data to a trickling speed compared to what it should be.
 
-- 
Carlo Wood <carlo@alinoe.com>