Re: [PATCH 1/1] SUNRPC: fix handling of half-closed connection

[Trond Myklebust <trondmy@hammerspace.com>]

On Fri, 2019-02-22 at 12:02 -0500, Olga Kornievskaia wrote:
> On Fri, Feb 22, 2019 at 10:50 AM Trond Myklebust
> <trondmy@hammerspace.com> wrote:
> > On Fri, 2019-02-22 at 10:11 -0500, Olga Kornievskaia wrote:
> > > On Fri, Feb 22, 2019 at 10:06 AM Trond Myklebust
> > > <trondmy@hammerspace.com> wrote:
> > > > On Fri, 2019-02-22 at 09:46 -0500, Olga Kornievskaia wrote:
> > > > > On Fri, Feb 22, 2019 at 8:45 AM Trond Myklebust <
> > > > > trondmy@hammerspace.com> wrote:
> > > > > > On Fri, 2019-02-22 at 07:12 -0500, Dave Wysochanski wrote:
> > > > > > > Hi Olga,
> > > > > > > 
> > > > > > > Do you have a reproducer for this?  A number of months
> > > > > > > ago I
> > > > > > > did
> > > > > > > a
> > > > > > > significant amount of testing with half-closed
> > > > > > > connections,
> > > > > > > after
> > > > > > > we
> > > > > > > had reports of connections stuck in FIN_WAIT2 in some
> > > > > > > older
> > > > > > > kernels.
> > > > > > > What I found was with kernels that had the tcp keepalives
> > > > > > > (commit
> > > > > > > 7f260e8575bf53b93b77978c1e39f8e67612759c), I could only
> > > > > > > reproduce
> > > > > > > a
> > > > > > > hang of a few minutes, after which time the tcp keepalive
> > > > > > > code
> > > > > > > would
> > > > > > > reset the connection.
> > > > > > > 
> > > > > > > That said it was a while ago and something subtle may
> > > > > > > have
> > > > > > > changed.
> > > > > > > Also I'm not not sure if your header implies an
> > > > > > > indefinite
> > > > > > > hang
> > > > > > > or
> > > > > > > just
> > > > > > > a few minutes.
> > > > > > > 
> > > > > > > Thanks.
> > > > > > > 
> > > > > > > 
> > > > > > > On Wed, 2019-02-20 at 09:56 -0500, Olga Kornievskaia
> > > > > > > wrote:
> > > > > > > > From: Olga Kornievskaia <kolga@netapp.com>
> > > > > > > > 
> > > > > > > > When server replies with an ACK to client's FIN/ACK,
> > > > > > > > client
> > > > > > > > ends
> > > > > > > > up stuck in a TCP_FIN_WAIT2 state and client's mount
> > > > > > > > hangs.
> > > > > > > > Instead, make sure to close and reset client's socket
> > > > > > > > and
> > > > > > > > transport
> > > > > > > > when transitioned into that state.
> > > > > 
> > > > > Hi Trond,
> > > > > 
> > > > > > So, please do note that we do not want to ignore the
> > > > > > FIN_WAIT2
> > > > > > state
> > > > > 
> > > > > But we do ignore the FIN_WAIT2 state.
> > > > 
> > > > We do not. We wait for the server to send a FIN, which is
> > > > precisely
> > > > the
> > > > reason for which FIN_WAIT2 exists.
> > > > 
> > > > > > because it implies that the server has not closed the
> > > > > > socket on
> > > > > > its
> > > > > > side.
> > > > > 
> > > > > That's correct.
> > > > > 
> > > > > > That again means that we cannot re-establish a connection
> > > > > > using
> > > > > > the same source IP+port to the server, which is problematic
> > > > > > for
> > > > > > protocols such as NFSv3 which rely on standard duplicate
> > > > > > reply
> > > > > > cache
> > > > > > for correct replay semantics.
> > > > > 
> > > > > that's exactly what's happening that a client is unable to
> > > > > establish
> > > > > a
> > > > > new connection to the server. With the patch, the client does
> > > > > an
> > > > > RST
> > > > > and it re-uses the port and all is well for NFSv3.
> > > > 
> > > > RST is not guaranteed to be delivered to the recipient. That's
> > > > why
> > > > the
> > > > TCP protocol defines FIN: it is a guaranteed to be delivered
> > > > because it
> > > > is ACKed.
> > > > 
> > > > > > This is why we don't just set the TCP_LINGER2 socket option
> > > > > > and
> > > > > > call
> > > > > > sock_release(). The choice to try to wait it out is
> > > > > > deliberate
> > > > > > because
> > > > > > the alternative is that we end up with busy-waiting re-
> > > > > > connection
> > > > > > attempts.
> > > > > 
> > > > > Why would it busy-wait? In my testing, RST happens and new
> > > > > connection
> > > > > is established?
> > > > 
> > > > Only if the server has dropped the connection without notifying
> > > > the
> > > > client.
> > > 
> > > Yes the server dropped the connection without notifying the
> > > client
> > > (or
> > > perhaps something in the middle did it as an attack). Again, I
> > > raise
> > > this concern for the sake of dealing with this as an attack. I
> > > have
> > > no
> > > intentions of catering to broken servers. If this is not a
> > > possible
> > > attack, then we don't have to deal with it.
> > 
> > A man in the middle might be able to intercept the FIN from the
> > server
> > and ACK it, causing the connection to be closed on that server.
> > However, as Dave pointed out, why wouldn't the keepalive mechanism
> > then
> > eventually kick in and close the socket on the client as well?
> 
> The mechanism is already kicked in and got stuck in FIN_WAIT2. NFS
> connection was idle, so TCP layer was sending keep-alives. Then it
> sent a FIN/ACK to which the server replied with just an ACK. Kernel
> notified NFS that we are in FIN_WAIT2 and I believe it is NFS
> responsibility to act accordingly. Kernel then keeps sending
> "keep-alives" forever.  Because of this code:
> 
>         case TCP_FIN_WAIT1:
>         case TCP_FIN_WAIT2:
>                 /* RFC 793 says to queue data in these states,
>                  * RFC 1122 says we MUST send a reset.
>                  * BSD 4.4 also does reset.
>                  */
>                 if (sk->sk_shutdown & RCV_SHUTDOWN) {
>                         if (TCP_SKB_CB(skb)->end_seq !=
> TCP_SKB_CB(skb)->seq &&
>                             after(TCP_SKB_CB(skb)->end_seq - th->fin,
> tp->rcv_nxt)) {
>                                 NET_INC_STATS(sock_net(sk),
> LINUX_MIB_TCPABORTONDATA);
>                                 tcp_reset(sk);   << this is never
> triggered
>                                 return 1;
>                         }
>                 }
> 
> In our case TCP_SKB_CB(skb)->end_seq always equals
> TCP_SKB_CB(skb)->seq. (No i don't know the meaning of end_seq and seq
> :-/).

Right, but if the connection is closed on the server, then it should be
sending RST replies to all these keepalives, one of which will
presumably eventually reach the client.

> > If the FIN is not ACKed, then the server is supposed to keep
> > retransmitting it. Until that ACK is received, it cannot close the
> > socket without violating the TCP protocol.
> 
> Something in the middle can keep intercepting the the FIN/ACK from
> the
> server and keep sending an ACK back?

Sure, but if can do that (which would entail being able to guess the
TCP segment seq nums) it can also be in a position to generally mess
with the TCP connection. Why do we care about that case?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com