conntrack, idle TCP connection and keep-alives

conntrack, idle TCP connection and keep-alives

[WGH]

Hello!

It seems that, when masquerading, conntrack silently drops idle
connection after nf_conntrack_tcp_timeout_established seconds. This's
pretty terrible, as application inside the network, if it never sends
anything, will never know that connection was dropped.

RFC 5382 gives us a solution to this:
> A NAT can check if an endpoint for a session has crashed by sending a
> TCP keep-alive packet and receiving a TCP RST packet in response.

However, it I couldn't find such feature in netfilter. It would be
pretty nice to have.

It would be much more effective than enabling keep-alives system-wide
(which is not even possible in practice). It makes sense that NAT has to
manage such things, as only NAT knows the timeouts of itself. If there's
a NAT along the route, it will send keep-alives (overhead, but
inevitable). If there's no NATs, there will be no keep-alives. Simple.

AFAIK, Cisco implements this under name Dead Connection Detection.

Re: conntrack, idle TCP connection and keep-alives

[Phil Oester]

On Sun, Oct 27, 2013 at 12:14:48AM +0400, WGH wrote:
> It seems that, when masquerading, conntrack silently drops idle
> connection after nf_conntrack_tcp_timeout_established seconds. This's
> pretty terrible, as application inside the network, if it never sends
> anything, will never know that connection was dropped.

If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
to an insanely high value.  You do realize, of course, that the conntrack
table has a finite number of entries.
 
> RFC 5382 gives us a solution to this:
> > A NAT can check if an endpoint for a session has crashed by sending a
> > TCP keep-alive packet and receiving a TCP RST packet in response.
> 
> However, it I couldn't find such feature in netfilter. It would be
> pretty nice to have.

Keepalives should be done in the application, not in the firewall.

Phil

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 08:34:09AM -0700, Phil Oester wrote:
> On Sun, Oct 27, 2013 at 12:14:48AM +0400, WGH wrote:
> > It seems that, when masquerading, conntrack silently drops idle
> > connection after nf_conntrack_tcp_timeout_established seconds. This's
> > pretty terrible, as application inside the network, if it never sends
> > anything, will never know that connection was dropped.
> 
> If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
> to an insanely high value.  You do realize, of course, that the conntrack
> table has a finite number of entries.
>  
> > RFC 5382 gives us a solution to this:
> > > A NAT can check if an endpoint for a session has crashed by sending a
> > > TCP keep-alive packet and receiving a TCP RST packet in response.
> > 
> > However, it I couldn't find such feature in netfilter. It would be
> > pretty nice to have.
> 
> Keepalives should be done in the application, not in the firewall.

Actually I think its a pretty nice idea to reduce breakage introduced
by NATs. There a millions of embedded devices that use very small timeout
values to reduce memory usage, at the cost of frequently breaking idle
connections.

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 06:01:30PM +0000, Patrick McHardy wrote:
> On Sun, Oct 27, 2013 at 08:34:09AM -0700, Phil Oester wrote:
> > On Sun, Oct 27, 2013 at 12:14:48AM +0400, WGH wrote:
> > > It seems that, when masquerading, conntrack silently drops idle
> > > connection after nf_conntrack_tcp_timeout_established seconds. This's
> > > pretty terrible, as application inside the network, if it never sends
> > > anything, will never know that connection was dropped.
> > 
> > If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
> > to an insanely high value.  You do realize, of course, that the conntrack
> > table has a finite number of entries.
> >  
> > > RFC 5382 gives us a solution to this:
> > > > A NAT can check if an endpoint for a session has crashed by sending a
> > > > TCP keep-alive packet and receiving a TCP RST packet in response.
> > > 
> > > However, it I couldn't find such feature in netfilter. It would be
> > > pretty nice to have.
> > 
> > Keepalives should be done in the application, not in the firewall.
> 
> Actually I think its a pretty nice idea to reduce breakage introduced
> by NATs. There a millions of embedded devices that use very small timeout
> values to reduce memory usage, at the cost of frequently breaking idle
> connections.

The downside seems to be that we'd need to keep track of timestamp values
to send valid keepalives, which also costs extra memory. I don't think that
cost is justifiable for NAT keepalives alone.

Re: conntrack, idle TCP connection and keep-alives

[Jozsef Kadlecsik]

On Sun, 27 Oct 2013, Patrick McHardy wrote:

> On Sun, Oct 27, 2013 at 06:01:30PM +0000, Patrick McHardy wrote:
> > On Sun, Oct 27, 2013 at 08:34:09AM -0700, Phil Oester wrote:
> > > On Sun, Oct 27, 2013 at 12:14:48AM +0400, WGH wrote:
> > > > It seems that, when masquerading, conntrack silently drops idle
> > > > connection after nf_conntrack_tcp_timeout_established seconds. This's
> > > > pretty terrible, as application inside the network, if it never sends
> > > > anything, will never know that connection was dropped.
> > > 
> > > If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
> > > to an insanely high value.  You do realize, of course, that the conntrack
> > > table has a finite number of entries.
> > >  
> > > > RFC 5382 gives us a solution to this:
> > > > > A NAT can check if an endpoint for a session has crashed by sending a
> > > > > TCP keep-alive packet and receiving a TCP RST packet in response.
> > > > 
> > > > However, it I couldn't find such feature in netfilter. It would be
> > > > pretty nice to have.
> > > 
> > > Keepalives should be done in the application, not in the firewall.
> > 
> > Actually I think its a pretty nice idea to reduce breakage introduced
> > by NATs. There a millions of embedded devices that use very small timeout
> > values to reduce memory usage, at the cost of frequently breaking idle
> > connections.
> 
> The downside seems to be that we'd need to keep track of timestamp values
> to send valid keepalives, which also costs extra memory. I don't think that
> cost is justifiable for NAT keepalives alone.

I think a single flag could be sufficient: if the timer in conntrack goes 
off and the entry is in the ESTABLISHED state and this flag is not set, 
then send a TCP keepalive packet and start the timer with a short timeout. 
If we receive the reply packet, then the long ESTABLISHED timeout value 
can be restored and the flag cleared.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> On Sun, 27 Oct 2013, Patrick McHardy wrote:
> 
> > On Sun, Oct 27, 2013 at 06:01:30PM +0000, Patrick McHardy wrote:
> > > On Sun, Oct 27, 2013 at 08:34:09AM -0700, Phil Oester wrote:
> > > > On Sun, Oct 27, 2013 at 12:14:48AM +0400, WGH wrote:
> > > > > It seems that, when masquerading, conntrack silently drops idle
> > > > > connection after nf_conntrack_tcp_timeout_established seconds. This's
> > > > > pretty terrible, as application inside the network, if it never sends
> > > > > anything, will never know that connection was dropped.
> > > > 
> > > > If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
> > > > to an insanely high value.  You do realize, of course, that the conntrack
> > > > table has a finite number of entries.
> > > >  
> > > > > RFC 5382 gives us a solution to this:
> > > > > > A NAT can check if an endpoint for a session has crashed by sending a
> > > > > > TCP keep-alive packet and receiving a TCP RST packet in response.
> > > > > 
> > > > > However, it I couldn't find such feature in netfilter. It would be
> > > > > pretty nice to have.
> > > > 
> > > > Keepalives should be done in the application, not in the firewall.
> > > 
> > > Actually I think its a pretty nice idea to reduce breakage introduced
> > > by NATs. There a millions of embedded devices that use very small timeout
> > > values to reduce memory usage, at the cost of frequently breaking idle
> > > connections.
> > 
> > The downside seems to be that we'd need to keep track of timestamp values
> > to send valid keepalives, which also costs extra memory. I don't think that
> > cost is justifiable for NAT keepalives alone.
> 
> I think a single flag could be sufficient: if the timer in conntrack goes 
> off and the entry is in the ESTABLISHED state and this flag is not set, 
> then send a TCP keepalive packet and start the timer with a short timeout. 
> If we receive the reply packet, then the long ESTABLISHED timeout value 
> can be restored and the flag cleared.

Sure, I think we wouldn't even need that flag, we can just send the keepalive
and set a short timeout. If a RST is received, the connection is killed
anyway, otherwise it will be refreshed with the ESTABLISHED timeout.

But we do need a timestamp value to pass PAWS.

Re: conntrack, idle TCP connection and keep-alives

[WGH]

On 27.10.2013 23:20, Patrick McHardy wrote:
> On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
>> I think a single flag could be sufficient: if the timer in conntrack goes 
>> off and the entry is in the ESTABLISHED state and this flag is not set, 
>> then send a TCP keepalive packet and start the timer with a short timeout. 
>> If we receive the reply packet, then the long ESTABLISHED timeout value 
>> can be restored and the flag cleared.
> Sure, I think we wouldn't even need that flag, we can just send the keepalive
> and set a short timeout. If a RST is received, the connection is killed
> anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
>
> But we do need a timestamp value to pass PAWS.
I believe you forgot the third scenario: neither ACK nor RST is received
in reply.

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 11:23:17PM +0400, WGH wrote:
> On 27.10.2013 23:20, Patrick McHardy wrote:
> > On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> >> I think a single flag could be sufficient: if the timer in conntrack goes 
> >> off and the entry is in the ESTABLISHED state and this flag is not set, 
> >> then send a TCP keepalive packet and start the timer with a short timeout. 
> >> If we receive the reply packet, then the long ESTABLISHED timeout value 
> >> can be restored and the flag cleared.
> > Sure, I think we wouldn't even need that flag, we can just send the keepalive
> > and set a short timeout. If a RST is received, the connection is killed
> > anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
> >
> > But we do need a timestamp value to pass PAWS.
> I believe you forgot the third scenario: neither ACK nor RST is received
> in reply.

Actually no, "... and set a short timeout ...".

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 07:32:44PM +0000, Patrick McHardy wrote:
> On Sun, Oct 27, 2013 at 11:23:17PM +0400, WGH wrote:
> > On 27.10.2013 23:20, Patrick McHardy wrote:
> > > On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> > >> I think a single flag could be sufficient: if the timer in conntrack goes 
> > >> off and the entry is in the ESTABLISHED state and this flag is not set, 
> > >> then send a TCP keepalive packet and start the timer with a short timeout. 
> > >> If we receive the reply packet, then the long ESTABLISHED timeout value 
> > >> can be restored and the flag cleared.
> > > Sure, I think we wouldn't even need that flag, we can just send the keepalive
> > > and set a short timeout. If a RST is received, the connection is killed
> > > anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
> > >
> > > But we do need a timestamp value to pass PAWS.
> > I believe you forgot the third scenario: neither ACK nor RST is received
> > in reply.
> 
> Actually no, "... and set a short timeout ...".

Well, OK, we do need a flag to distinguish normal timeout from probe timeout.
But still I don't see how we can do this without increasing the size of
every conntrack by at least 4 bytes.

Re: conntrack, idle TCP connection and keep-alives

[Jozsef Kadlecsik]

On Sun, 27 Oct 2013, Patrick McHardy wrote:

> On Sun, Oct 27, 2013 at 07:32:44PM +0000, Patrick McHardy wrote:
> > On Sun, Oct 27, 2013 at 11:23:17PM +0400, WGH wrote:
> > > On 27.10.2013 23:20, Patrick McHardy wrote:
> > > > On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> > > >> I think a single flag could be sufficient: if the timer in conntrack goes 
> > > >> off and the entry is in the ESTABLISHED state and this flag is not set, 
> > > >> then send a TCP keepalive packet and start the timer with a short timeout. 
> > > >> If we receive the reply packet, then the long ESTABLISHED timeout value 
> > > >> can be restored and the flag cleared.
> > > > Sure, I think we wouldn't even need that flag, we can just send the keepalive
> > > > and set a short timeout. If a RST is received, the connection is killed
> > > > anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
> > > >
> > > > But we do need a timestamp value to pass PAWS.
> > > I believe you forgot the third scenario: neither ACK nor RST is received
> > > in reply.
> > 
> > Actually no, "... and set a short timeout ...".
> 
> Well, OK, we do need a flag to distinguish normal timeout from probe 
> timeout. But still I don't see how we can do this without increasing the 
> size of every conntrack by at least 4 bytes.

Yes, you're right: PAWS assumes all packets carry timestamps option, an 
option-less ACK isn't sufficient. And increasing every conntrack entry 
does seem too expensive when the application itself could send keep-alive 
packets.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: conntrack, idle TCP connection and keep-alives

[Jozsef Kadlecsik]

On Sun, 27 Oct 2013, Jozsef Kadlecsik wrote:

> On Sun, 27 Oct 2013, Patrick McHardy wrote:
> 
> > On Sun, Oct 27, 2013 at 07:32:44PM +0000, Patrick McHardy wrote:
> > > On Sun, Oct 27, 2013 at 11:23:17PM +0400, WGH wrote:
> > > > On 27.10.2013 23:20, Patrick McHardy wrote:
> > > > > On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> > > > >> I think a single flag could be sufficient: if the timer in conntrack goes 
> > > > >> off and the entry is in the ESTABLISHED state and this flag is not set, 
> > > > >> then send a TCP keepalive packet and start the timer with a short timeout. 
> > > > >> If we receive the reply packet, then the long ESTABLISHED timeout value 
> > > > >> can be restored and the flag cleared.
> > > > > Sure, I think we wouldn't even need that flag, we can just send the keepalive
> > > > > and set a short timeout. If a RST is received, the connection is killed
> > > > > anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
> > > > >
> > > > > But we do need a timestamp value to pass PAWS.
> > > > I believe you forgot the third scenario: neither ACK nor RST is received
> > > > in reply.
> > > 
> > > Actually no, "... and set a short timeout ...".
> > 
> > Well, OK, we do need a flag to distinguish normal timeout from probe 
> > timeout. But still I don't see how we can do this without increasing the 
> > size of every conntrack by at least 4 bytes.
> 
> Yes, you're right: PAWS assumes all packets carry timestamps option, an 
> option-less ACK isn't sufficient. And increasing every conntrack entry 
> does seem too expensive when the application itself could send keep-alive 
> packets.

Looking at the source code, actually, the Linux TCP stack handles 
gracefully TCP packets without timestamp options when sender originally 
announced TCP timestamps. So it seems to me we could send a simple, 
option-less "keep-alive" packet. RFC1323 does not discuss the case but if 
the option is missing, stacks should fall back to the base handling, 
isn't it? ;-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: conntrack, idle TCP connection and keep-alives

[Patrick McHardy]

On Sun, Oct 27, 2013 at 09:49:47PM +0100, Jozsef Kadlecsik wrote:
> On Sun, 27 Oct 2013, Jozsef Kadlecsik wrote:
> 
> > On Sun, 27 Oct 2013, Patrick McHardy wrote:
> > 
> > > On Sun, Oct 27, 2013 at 07:32:44PM +0000, Patrick McHardy wrote:
> > > > On Sun, Oct 27, 2013 at 11:23:17PM +0400, WGH wrote:
> > > > > On 27.10.2013 23:20, Patrick McHardy wrote:
> > > > > > On Sun, Oct 27, 2013 at 08:14:19PM +0100, Jozsef Kadlecsik wrote:
> > > > > >> I think a single flag could be sufficient: if the timer in conntrack goes 
> > > > > >> off and the entry is in the ESTABLISHED state and this flag is not set, 
> > > > > >> then send a TCP keepalive packet and start the timer with a short timeout. 
> > > > > >> If we receive the reply packet, then the long ESTABLISHED timeout value 
> > > > > >> can be restored and the flag cleared.
> > > > > > Sure, I think we wouldn't even need that flag, we can just send the keepalive
> > > > > > and set a short timeout. If a RST is received, the connection is killed
> > > > > > anyway, otherwise it will be refreshed with the ESTABLISHED timeout.
> > > > > >
> > > > > > But we do need a timestamp value to pass PAWS.
> > > > > I believe you forgot the third scenario: neither ACK nor RST is received
> > > > > in reply.
> > > > 
> > > > Actually no, "... and set a short timeout ...".
> > > 
> > > Well, OK, we do need a flag to distinguish normal timeout from probe 
> > > timeout. But still I don't see how we can do this without increasing the 
> > > size of every conntrack by at least 4 bytes.
> > 
> > Yes, you're right: PAWS assumes all packets carry timestamps option, an 
> > option-less ACK isn't sufficient. And increasing every conntrack entry 
> > does seem too expensive when the application itself could send keep-alive 
> > packets.
> 
> Looking at the source code, actually, the Linux TCP stack handles 
> gracefully TCP packets without timestamp options when sender originally 
> announced TCP timestamps. So it seems to me we could send a simple, 
> option-less "keep-alive" packet. RFC1323 does not discuss the case but if 
> the option is missing, stacks should fall back to the base handling, 
> isn't it? ;-)

That would be one option. Alternatively we could just make it optional
by putting it into an extend or, looking at the NAT structure, shrink
it a bit more for the NAT case to make up for the size increase by
moving the PPtP helper data to an extend.

Re: conntrack, idle TCP connection and keep-alives

[WGH]

On 27.10.2013 19:34, Phil Oester wrote:
> If this is a problem for you, then increase nf_conntrack_tcp_timeout_established
> to an insanely high value.  You do realize, of course, that the conntrack
> table has a finite number of entries.
It'll delay the problem, but not fix it. Besides, it'll worsen the
situtation that established timeout intended to fix - genuinely crashed
connections will linger for said insane value.
> Keepalives should be done in the application, not in the firewall. 
Why not, actually? It isn't strictly keep-alive in application sense,
but rather a way that NAT may use to detect broken connections. It
addresses breakage caused by NAT itself, while keep-alives issued by
application will address other problems (like physical connection loss),
if necessary.