Unsafe TCP connection close handling

Unsafe TCP connection close handling

[Josh Durgin]

Yesterday during an ffsb run on the ceph kernel client, both
the client and the osd processes hit the max open fds limit
(there was only one osd up at the time). There were 1006 sockets
in the CLOSING state on the client, and 1006 in the FIN_WAIT2
state on the osd.

 From the tcp state machine [1], it seems that the
sequence of events was something like this, with both sides
initially in the ESTABLISHED state:

       Kernel Client            OSD
            |                    |
            |                   /| Send FIN, go to FIN_WAIT1
Send FIN,  |\                 / |
go to      | \               /  |
FIN_WAIT1  |  \             /   |
            |   \           /    |
Recv FIN   |<--------------     |
            |     \              |
Send ACK,  |------\------------>| Recv ACK, go to FIN_WAIT2
go to      |       \            |
CLOSING    |        -----------x| FIN not read

That is, after closing its half of the connection, the osd isn't
reading anything from the socket anymore, and thus ignores the
FIN from the client. We have bug #1803 to track this, but we
should make sure libceph in the kernel handles simultaneous
TCP connection close correctly as well.

[1] http://www.tcpipguide.com/free/diagrams/tcpfsm.png