Re: 6.1: possible bug with netfilter conntrack?

["Russell King (Oracle)" <linux@armlinux.org.uk>]

On Fri, Jan 13, 2023 at 12:45:03AM +0100, Florian Westphal wrote:
> Russell King (Oracle) <linux@armlinux.org.uk> wrote:
> > Given the packet counts as per my example above, it looks like
> > conntrack only saw:
> > 
> > src=180.173.2.183 dst=78.32.30.218	SYN
> > src=78.32.30.218 dst=180.173.2.183	SYN+ACK
> > src=180.173.2.183 dst=78.32.30.218	ACK
> > 
> > and I suspect at that point, the connection went silent - until
> > Exim timed out and closed the connection, as does seem to be the
> > case:
> > 
> > 2023-01-11 21:32:04 no host name found for IP address 180.173.2.183
> > 2023-01-11 21:33:05 SMTP command timeout on connection from [180.173.2.183]:64332 I=[78.32.30.218]:25
> > 
> > but if Exim closed the connection, why didn't conntrack pick it up?
> 
> Yes, thats the question.  Exim closing the connection should have
> conntrack at least pick up a fin packet from the mail server (which
> should move the entry to the 2 minute fin timeout).

Okay, update this morning. I left tcpdump running overnight having
cleared conntrack of all port 25 and 587 connections. This morning,
there's a whole bunch of new entries on conntrack.

Digging through the tcpdump and logs, it seems what is going on is:

public interface			dmz interface
origin -> mailserver SYN		origin -> mailserver SYN
mailserver -> origin SYNACK		mailserver -> origin SYNACK
origin -> mailserver ACK
mailserver -> origin RST
mailserver -> origin SYNACK		mailserver -> origin SYNACK
mailserver -> origin SYNACK		mailserver -> origin SYNACK
mailserver -> origin SYNACK		mailserver -> origin SYNACK
mailserver -> origin SYNACK		mailserver -> origin SYNACK
...

Here is an example from the public interface:

09:52:36.599398 IP 103.14.225.112.63461 > 78.32.30.218.587: Flags [SEW], seq 3387227814, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
09:52:36.599893 IP 78.32.30.218.587 > 103.14.225.112.63461: Flags [S.], seq 816385329, ack 3387227815, win 64240, options [mss 1452,nop,nop,sackOK,nop,wscale 7], length 0
09:52:36.820464 IP 103.14.225.112.63461 > 78.32.30.218.587: Flags [.], ack 1, win 260, length 0
09:52:36.820549 IP 78.32.30.218.587 > 103.14.225.112.63461: Flags [R], seq 816385330, win 0, length 0
09:52:37.637548 IP 78.32.30.218.587 > 103.14.225.112.63461: Flags [S.], seq 816385329, ack 3387227815, win 64240, options [mss 1452,nop,nop,sackOK,nop,wscale 7], length 0

and the corresponding trace on the mailserver:
09:52:36.599729 IP 103.14.225.112.63461 > 78.32.30.218.587: Flags [SEW], seq 3387227814, win 8192, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
09:52:36.599772 IP 78.32.30.218.587 > 103.14.225.112.63461: Flags [S.], seq 816385329, ack 3387227815, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
09:52:37.637421 IP 78.32.30.218.587 > 103.14.225.112.63461: Flags [S.], seq 816385329, ack 3387227815, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

So, my first observation is that conntrack is reacting to the ACK
packet on the public interface, and marking the connection established,
but a firewall rule is rejecting the connection when that ACK packet is
received by sending a TCP reset. It looks like conntrack does not see 
this packet, and also conntrack does not see the SYNACK retransmissions
(which is odd, because it saw the first one.)

As to why we're responding with a TCP reset to the ACK packet, it's
because iptables is hitting a reject rule as the IP address has been
temporarily banned due to preceding known spammer signatures a few
seconds before.

I probably ought to pick up on the initial SYN rather than the 3rd
packet of the connection... but even so, I don't think conntrack
should be missing the TCP reset from the reject rule.

The rule path that leads to the reject rule is currently:
  -A TCP -p tcp -m multiport --dports 25,587 -m conntrack --ctstate ESTABLISHED -j TCP-smtp-in
  -A TCP-smtp-in -p tcp -m set --match-set ip4-banned-smtp src -j TCP-smtp-s
  -A TCP-smtp-s -j SET --add-set ip4-banned-smtp src --exist --timeout N
  -A TCP-smtp-s -p tcp -j REJECT --reject-with tcp-reset

(I've omitted the timeout.)

There definitely seems to be a change in behaviour - looking back to
the logs prior to upgrading to 6.1, there were never any conntrack
table overflows, and that older kernel had been running for hundreds
of days.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!