Established sockets remain open after iface down or address lost

Established sockets remain open after iface down or address lost

[Chris Verges]

Hello all,

I've encountered a behavior that appears to be known, but am seeking
some clarity on its rationale.  The scenario is as follows:

  (0) A TCP server socket listens on :: (v4/v6).
  (1) Connect a USB/Ethernet adapter to a Linux system.
  (2) Adapter is brought up as 'eth0' with an IP address.
  (3) A remote TCP client connects to the server socket.
  (4) 'netstat -anp' shows the socket as ESTABLISHED
  (5) The TCP server starts a blocking read waiting for data.
  (6) Physically disconnect the USB/Ethernet adapter from the USB bus.
  (7) Linux removes the 'eth0' interface and associated IP address.

At this point, the socket _still_ shows as ESTABLISHED under netstat.

This is the paradox.  Why is the blocking read not interrupted with a
socket error to indicate that the socket is no longer viable?

Thank you in advance,
Chris

P.S.  I apologize in advance if I missed this answer in the netdev
archives.

Re: Established sockets remain open after iface down or address lost

[Eric Dumazet]

On Wed, 2013-09-25 at 23:04 -0700, Chris Verges wrote:
> Hello all,
> 
> I've encountered a behavior that appears to be known, but am seeking
> some clarity on its rationale.  The scenario is as follows:
> 
>   (0) A TCP server socket listens on :: (v4/v6).
>   (1) Connect a USB/Ethernet adapter to a Linux system.
>   (2) Adapter is brought up as 'eth0' with an IP address.
>   (3) A remote TCP client connects to the server socket.
>   (4) 'netstat -anp' shows the socket as ESTABLISHED
>   (5) The TCP server starts a blocking read waiting for data.
>   (6) Physically disconnect the USB/Ethernet adapter from the USB bus.
>   (7) Linux removes the 'eth0' interface and associated IP address.
> 
> At this point, the socket _still_ shows as ESTABLISHED under netstat.
> 
> This is the paradox.  Why is the blocking read not interrupted with a
> socket error to indicate that the socket is no longer viable?

Because TCP layer is not sensitive to such temporary events.

You can plug again your iface, and IP is valid again.

Why should we give a permanent error for such case ?

If network communication is cut somewhere, TCP is not supposed to
immediately react. Normal timeouts and retransmits take place. 

Re: Established sockets remain open after iface down or address lost

[Chris Verges]

On Thu, Sep 26, 2013 at 06:49:43AM -0700, Eric Dumazet wrote:
> On Wed, 2013-09-25 at 23:04 -0700, Chris Verges wrote:
> >   (6) Physically disconnect the USB/Ethernet adapter from the USB
> >       bus.
> >   (7) Linux removes the 'eth0' interface and associated IP address.
> > 
> > At this point, the socket _still_ shows as ESTABLISHED under
> > netstat.
> > 
> > This is the paradox.  Why is the blocking read not interrupted with
> > a socket error to indicate that the socket is no longer viable?
> 
> Because TCP layer is not sensitive to such temporary events.  You can
> plug again your iface, and IP is valid again.  Why should we give a
> permanent error for such case ?

Yes, the interface could be reconnected ... or it could not be.
Consider an embedded device where a PPP-based radio module is powered
off for a decent amount of time (hours).

  +--------+          +-----------------+
  | Client |----------| Embedded Server |
  +--------+          +-----------------+

The client establishes a connection to the server.  It requests some
data and gets a response.  The socket remains open.  The server then
decides, through some asynchronous process, that the radio needs to be
duty cycled.  So the radio is turned off.

The client attempts to make another request to the device, but
determines that the connection is dead through the normal retry
mechanisms.  It's write() operation returns something like EPIPE.  So on
the client's side, the connection is dead.

But on the server side, the socket is still open and waiting for some
more data.  The interface and IP address and even the remote client are
long gone, but the socket still persists and uses system resources.

This is primarily an issue when the server binds/listens/accepts without
using a socket pool to process sockets after the accept().  That is, the
server processes only one socket.  If the socket resource is held by
this now-dead connection, there is generally no way to reset it without
lengthy or costly keepalives (depending on whether the keepalive timer
is set long or short, respectively.)

> If network communication is cut somewhere, TCP is not supposed to
> immediately react. Normal timeouts and retransmits take place. 

I agree in the sense that "somewhere" is between the remote station
(inclusive) and the local station (exclusive.)  I would argue that the
local station could be aware of its own state changes and may choose to
respond accordingly.

I can see the arguments for why the existing behavior would be viewed
favorably.  From this particular real-world scenario that I am
encountering, I can also see why a modified behavior would be useful.
Would you consider accepting a patch that adds a new socket option to
optionally control this?  The effect would be to cause the socket to
automatically close (interrupting any blocking reads) if the underlying
address used by the socket is unregistered from the stack.  Default
behavior would be maintained.

Thanks,
Chris

Re: Established sockets remain open after iface down or address lost

[Rick Jones]

On 10/01/2013 06:27 AM, Chris Verges wrote:
> The client establishes a connection to the server.  It requests some
> data and gets a response.  The socket remains open.  The server then
> decides, through some asynchronous process, that the radio needs to be
> duty cycled.  So the radio is turned off.
>
> The client attempts to make another request to the device, but
> determines that the connection is dead through the normal retry
> mechanisms.  It's write() operation returns something like EPIPE.  So on
> the client's side, the connection is dead.
>
> But on the server side, the socket is still open and waiting for some
> more data.  The interface and IP address and even the remote client are
> long gone, but the socket still persists and uses system resources.

The protocol between client and server needs to have an 
application-layer "keepalive" mechanism added, and then the server will 
be able to detect a dangling connection without need of any further 
kernel modifications.

If that is not possible, the server can/should set SO_KEEPALIVE and 
perhaps tweak the TCP keepalive settings.  Not as good (IMO) as an 
application-layer keepalive because it only shows that the connection is 
good as far as TCP, but I suppose it could do in a pinch.

rick jones

Re: Established sockets remain open after iface down or address lost

[Chris Verges]

On Tue, Oct 01, 2013 at 08:44:17AM -0700, Rick Jones wrote:
> The protocol between client and server needs to have an
> application-layer "keepalive" mechanism added, and then the server
> will be able to detect a dangling connection without need of any
> further kernel modifications.
> 
> If that is not possible, the server can/should set SO_KEEPALIVE and
> perhaps tweak the TCP keepalive settings.  Not as good (IMO) as an
> application-layer keepalive because it only shows that the connection
> is good as far as TCP, but I suppose it could do in a pinch.

I agree that some form of keepalives would solve the problem where
blocking reads need to be interrupted.  However, this creates traffic
across the link -- directly proportional to the keepalive interval.

The underlying physical layer is such that we pay for all traffic going
across it -- including any keepalives at either the application or TCP
layers.  Paying for this keepalive traffic when the link is operational
is not desired.

Thanks for the suggestion.  I, too, am hoping that a kernel mod isn't
required to do what is being asked.  But I'm also willing to put in the
work if needed.  My hope on engaging with the netdev list early in this
process is to (1) figure out whether this has already been fully solved
and (2) determine whether there would be interest in this patch for
general use.

Thanks,
Chris

Re: Established sockets remain open after iface down or address lost

[Rick Jones]

On 10/01/2013 09:08 AM, Chris Verges wrote:
> On Tue, Oct 01, 2013 at 08:44:17AM -0700, Rick Jones wrote:
>> The protocol between client and server needs to have an
>> application-layer "keepalive" mechanism added, and then the server
>> will be able to detect a dangling connection without need of any
>> further kernel modifications.
>>
>> If that is not possible, the server can/should set SO_KEEPALIVE and
>> perhaps tweak the TCP keepalive settings.  Not as good (IMO) as an
>> application-layer keepalive because it only shows that the connection
>> is good as far as TCP, but I suppose it could do in a pinch.
>
> I agree that some form of keepalives would solve the problem where
> blocking reads need to be interrupted.  However, this creates traffic
> across the link -- directly proportional to the keepalive interval.
>
> The underlying physical layer is such that we pay for all traffic going
> across it -- including any keepalives at either the application or TCP
> layers.  Paying for this keepalive traffic when the link is operational
> is not desired.

Pick your poison :)

If the server application is in a "I know there should be (more) data 
arriving on this connection" mode, then you can simply have an 
application-layer timeout in the server code that does not rely on 
active probing the connection.

Otherwise, even if you do get some sort of "nuke connections using a 
source IP matching an interface we just brought down" option into the 
kernel, you will still have the small matter of something else between 
the client and server going down that neither can see directly.

rick jones

Re: Established sockets remain open after iface down or address lost

[Alexey Kuznetsov]

Hello!

> P.S.  I apologize in advance if I missed this answer in the netdev archives.

FYI googling f.e. "netdev tcp remove local address" instantly  finds
all that you want to know. Namrly, subj: Re: "TCP shutdown behaviour
when deleting local IP addresses"

Re: Established sockets remain open after iface down or address lost

[Chris Verges]

On Tue, Oct 01, 2013 at 08:33:04PM +0400, Alexey Kuznetsov wrote:
> > P.S.  I apologize in advance if I missed this answer in the netdev
> > archives.
> 
> FYI googling f.e. "netdev tcp remove local address" instantly  finds
> all that you want to know. Namrly, subj: Re: "TCP shutdown behaviour
> when deleting local IP addresses"

Oustanding.  Thank you!  I really appreciate the pointer to bring me up
to speed on the history.  It sounds like Mikael and I share a similar
desire and rationale for why such a thing would be useful.

Your comment regarding the tcp hash table scanning is what I was
planning to code up, though it sounds like some amount of effort was
made in this attempt and abandoned.  Would you happen to know where any
out-of-kernel code related to this might be parked if one wanted to
continue this effort?

I do agree with Mikael that offloading this from the kernel to some kind
of connection manager would be ideal.  I'll ponder this some more....

Again, many thanks for the help.

Chris

Re: Established sockets remain open after iface down or address lost

[Alexey Kuznetsov]

Hello!

On Tue, Oct 1, 2013 at 9:07 PM, Chris Verges
<cverges@sentient-energy.com> wrote:
>  Would you happen to know where any
> out-of-kernel code related to this might be parked if one wanted to
> continue this effort?

Not a big surpize, but reference is in the same thead, I am am doing
cut-n-paste for you:

From: Andi Kleen <andi@xxxxxxxxxxxxxx>
Date: Fri, 19 Oct 2012 13:43:39 -0700

I had a patch for this a long time ago to speed up recovery.
It was very useful with dial on demand ISDN links or DSL
connections that hang up occasionally to force a new IP.

It was in SUSE kernels for a long time, but never made it
into mainline

http://www.firstfloor.org/pub/ak/quilt-before-arch-merge/patches/iff-dynamic

-Andi

Don't forget reference to Andi Kleen in case you push some descendant
of this code
to mainline.