conntrack table filling up due to missing ACKs near the end of a TCP session

conntrack table filling up due to missing ACKs near the end of a TCP session

[Menno Smits]

Hi all,

I'm seeing a problem on a customer box that acts as a primary MX and is 
  running netfilter. The secondary MXs (run by the ISP) appear to be 
broken and cause the conntrack table on the primary MX to fill up.

What happens is this:

- secondary connects to deliver mail
- connection is established and SMTP conversation takes place
- secondary issues QUIT
- primary sends response "221 Bye"
- primary sends FIN
- secondary sends FIN
- primary ACKs FIN from secondary

The secondary never ACKs the last data packet ("221 Bye") or FIN sent by 
the primary. Later the primary retries the "221 Bye" packets several 
times. Interestingly the secondary also sends a Keep-Alive soon after it 
sends it's FIN.

This happens for every inbound connection from the secondaries and the 
result is 1000's of ESTABLISHED entries with 5 day timeouts in the 
conntrack table. The table eventually fills up causing packet loss. The 
conntrack entries are all of the form:

tcp      6 24826 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip 
sport=25 dport=54334 packets=3 bytes=183 [UNREPLIED] src=sec.ond.ary.ip 
dst=pri.ma.ry.ip sport=54334 dport=25 packets=0 bytes=0 mark=0 use=1

The primary is blocking packets that aren't NEW, ESTALISHED or RELATED. 
This could be contributing to the problem but I haven't looked closely 
at this.

Clearly the secondary mail servers are broken but shouldn't netfilter 
handle this situation better. Shouldn't the timeout of a connection be 
greatly reduced if a FIN has been sent?

I have an isolated packet dump for one connection. I'm happy to send it 
   to a netfilter developer privately but am hesitant to post it to the 
list since it is from a customer site.

Any thoughts or advice on this would be greatly appreciated.

Regards,
Menno Smits

NetBox Blue
http://netboxblue.com/

Re: conntrack table filling up due to missing ACKs near the end of a TCP session

[Patrick McHardy]

Menno Smits wrote:
> Hi all,
> 
> I'm seeing a problem on a customer box that acts as a primary MX and is
>  running netfilter. The secondary MXs (run by the ISP) appear to be
> broken and cause the conntrack table on the primary MX to fill up.
> 
> What happens is this:
> 
> - secondary connects to deliver mail
> - connection is established and SMTP conversation takes place
> - secondary issues QUIT
> - primary sends response "221 Bye"
> - primary sends FIN
> - secondary sends FIN
> - primary ACKs FIN from secondary
> 
> The secondary never ACKs the last data packet ("221 Bye") or FIN sent by
> the primary. Later the primary retries the "221 Bye" packets several
> times. Interestingly the secondary also sends a Keep-Alive soon after it
> sends it's FIN.
> 
> This happens for every inbound connection from the secondaries and the
> result is 1000's of ESTABLISHED entries with 5 day timeouts in the
> conntrack table. The table eventually fills up causing packet loss. The
> conntrack entries are all of the form:
> 
> tcp      6 24826 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip
> sport=25 dport=54334 packets=3 bytes=183 [UNREPLIED] src=sec.ond.ary.ip
> dst=pri.ma.ry.ip sport=54334 dport=25 packets=0 bytes=0 mark=0 use=1
> 
> The primary is blocking packets that aren't NEW, ESTALISHED or RELATED.
> This could be contributing to the problem but I haven't looked closely
> at this.

If you look at the byte counters, the connection seems to consist
only of three packets sent by the primary. I guess one of them
was an ACK, which through connection pick-up places the conntrack
in ESTABLISHED state. The first question would be why they're not
properly associated with the real connection. Please try
"echo 255 > /proc/net/ipv4/netfilter/ip_conntrack_log_invalid".
As a work-around you could disable connection pickup or drop
outgoing TCP packets with state NEW but no SYN.

Re: conntrack table filling up due to missing ACKs near the end of a TCP session

[Menno Smits]

Hi Patrick,

Thanks for your reply.

Patrick McHardy wrote:
> 
> If you look at the byte counters, the connection seems to consist
> only of three packets sent by the primary. I guess one of them
> was an ACK, which through connection pick-up places the conntrack
> in ESTABLISHED state. The first question would be why they're not
> properly associated with the real connection. Please try
> "echo 255 > /proc/net/ipv4/netfilter/ip_conntrack_log_invalid".
> As a work-around you could disable connection pickup or drop
> outgoing TCP packets with state NEW but no SYN.

I tried enabling invalid packet logging like you suggested but nothing 
was logged.

Turning off connection pickup via ip_conntrack_tcp_loose fixes the 
problem. There definitely seems to be an issue with packets being 
associated with the wrong connection.

I've looked a little closer at the dodgy connections in the table 
(before the workaround). They are always of one of two forms, 2 packets 
sent or 3 packets sent:

tcp      6 51949 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip 
sport=25 dport=38755 packets=3 bytes=183 [UNREPLIED] src=sec.ond.ary.ip 
dst=pri.ma.ry.ip sport=38755 dport=25 packets=0 bytes=0 mark=0 use=1

tcp      6 48900 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip 
sport=25 dport=57868 packets=2 bytes=122 [UNREPLIED] src=sec.ond.ary.ip 
dst=pri.ma.ry.ip sport=57868 dport=25 packets=0 bytes=0 mark=0 use=1

The IPs, packet counts and byte counts are always identical. Only the 
non-port-25 ports change.

Are you interested in packet dumps of the connections that cause this 
behaviour? I have some here.

Menno

Re: conntrack table filling up due to missing ACKs near the end of a TCP session

[Patrick McHardy]

Menno Smits wrote:
> I tried enabling invalid packet logging like you suggested but nothing
> was logged.

Forgot to mention, you need to have one of ipt_LOG or ipt_ULOG loaded,
in case of ipt_ULOG you also need the ulog daemon running.

> Turning off connection pickup via ip_conntrack_tcp_loose fixes the
> problem. There definitely seems to be an issue with packets being
> associated with the wrong connection.
> 
> I've looked a little closer at the dodgy connections in the table
> (before the workaround). They are always of one of two forms, 2 packets
> sent or 3 packets sent:
> 
> tcp      6 51949 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip
> sport=25 dport=38755 packets=3 bytes=183 [UNREPLIED] src=sec.ond.ary.ip
> dst=pri.ma.ry.ip sport=38755 dport=25 packets=0 bytes=0 mark=0 use=1
> 
> tcp      6 48900 ESTABLISHED src=pri.ma.ry.ip dst=sec.ond.ary.ip
> sport=25 dport=57868 packets=2 bytes=122 [UNREPLIED] src=sec.ond.ary.ip
> dst=pri.ma.ry.ip sport=57868 dport=25 packets=0 bytes=0 mark=0 use=1
> 
> The IPs, packet counts and byte counts are always identical. Only the
> non-port-25 ports change.
> 
> Are you interested in packet dumps of the connections that cause this
> behaviour? I have some here.

Yes, that might give some clues. Just send it to me in private or
on-list, as you like.