connection tracking and kernel dropping packets

connection tracking and kernel dropping packets

[Matt Zagrabelny]

Hello,

The kernel of my DNS server is dropping packets:

kernel: [48074703.302657] nf_conntrack: table full, dropping packet.

I've followed this knowledge base article:

https://kb.isc.org/docs/aa-01183

for configuring iptables to not track the connections of DNS packets.

...but it is still dropping packets due to the CT.

I know I could bump the conntrack memory, but I'd still like to know
why my iptables rules aren't sufficient for not dropping DNS packets.

I'm running Linux 3.2.0-4-amd64

Here are my iptables rules:

# iptables -vnL -t raw
Chain PREROUTING (policy ACCEPT 24M packets, 2112M bytes)
 pkts bytes target     prot opt in     out     source               destination
  16M 1153M CT         udp  --  *      *       0.0.0.0/0
0.0.0.0/0            udp dpt:53 CT notrack
3723K  701M CT         udp  --  *      *       0.0.0.0/0
0.0.0.0/0            udp spt:53 CT notrack

Chain OUTPUT (policy ACCEPT 24M packets, 5436M bytes)
 pkts bytes target     prot opt in     out     source               destination
3760K  327M CT         udp  --  *      *       0.0.0.0/0
0.0.0.0/0            udp dpt:53 CT notrack
  16M 4680M CT         udp  --  *      *       0.0.0.0/0
0.0.0.0/0            udp spt:53 CT notrack

# iptables -vnL
Chain INPUT (policy DROP 45173 packets, 2842K bytes)
 pkts bytes target     prot opt in     out     source               destination
  23M 2065M ACCEPT     all  --  *      *       0.0.0.0/0
0.0.0.0/0            ctstate RELATED,ESTABLISHED,UNTRACKED
   11   804 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    4   284 ACCEPT     udp  --  *      *       0.0.0.0/0
0.0.0.0/0            udp dpt:53
 709K   37M ACCEPT     tcp  --  *      *       0.0.0.0/0
0.0.0.0/0            tcp dpt:53

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 24M packets, 5439M bytes)
 pkts bytes target     prot opt in     out     source               destination

Any ideas what I'm missing?

Thanks for the help!

-m

Re: connection tracking and kernel dropping packets

[Slavko]

Dňa 29. októbra 2024 15:11:34 UTC používateľ Matt Zagrabelny <mzagrabe@d.umn.edu> napísal:

>...but it is still dropping packets due to the CT.

You have first to inspect what is filling your conntrack table:

    conntrack -L

Then you have to decide, if you have to add more notrack rules, or
you are under eg. SYN flood or so... IMO you forget that DNS can
use TCP too (not mentioned in that article"s rules).

Do not forget, that "lo" traffic can create (a lot of) conntrack entries
too.

>I'm running Linux 3.2.0-4-amd64

IMO quite old, AFAIK modern kernels can do better with SYN floods
(via better SYN cookies approach), if that is source of your problems.

regards


-- 
Slavko
https://www.slavino.sk/

Re: connection tracking and kernel dropping packets

[Matt Zagrabelny]

Hi Slavko,

On Tue, Oct 29, 2024 at 10:48 AM Slavko <linux@slavino.sk> wrote:
>
> Dňa 29. októbra 2024 15:11:34 UTC používateľ Matt Zagrabelny <mzagrabe@d.umn.edu> napísal:
>
> >...but it is still dropping packets due to the CT.
>
> You have first to inspect what is filling your conntrack table:
>
>     conntrack -L

Ah.. Thanks for that hint!

>
> Then you have to decide, if you have to add more notrack rules, or
> you are under eg. SYN flood or so... IMO you forget that DNS can
> use TCP too (not mentioned in that article"s rules).

Agreed. I added the TCP no track after sending the initial email.


> Do not forget, that "lo" traffic can create (a lot of) conntrack entries
> too.

Sure. I'll look at the conntrack output.

> >I'm running Linux 3.2.0-4-amd64
>
> IMO quite old, AFAIK modern kernels can do better with SYN floods
> (via better SYN cookies approach), if that is source of your problems.

Agreed. It's on the docket to upgrade.

Thanks for the helpful reply.

Cheers!

-m

Re: connection tracking and kernel dropping packets

[Matt Zagrabelny]

Greetings...

On Tue, Oct 29, 2024 at 10:48 AM Slavko <linux@slavino.sk> wrote:
>
> Dňa 29. októbra 2024 15:11:34 UTC používateľ Matt Zagrabelny <mzagrabe@d.umn.edu> napísal:
>
> >...but it is still dropping packets due to the CT.
>
> You have first to inspect what is filling your conntrack table:
>
>     conntrack -L

I've waited a week to let the TCP streams in the conntrack table time
out. I'm still seeing the kernel drop packets:

# tail -f /var/log/kern.log
Nov  6 11:29:02 netadm kernel: [48773744.961053] nf_conntrack: table
full, dropping packet.

...and confirmed with /proc:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
65536

...but there aren't that many flows in the conntrack table, and none
in the expect table.

# conntrack -L > /dev/null
conntrack v1.2.1 (conntrack-tools): 22 flow entries have been shown.

# conntrack -L expect
conntrack v1.2.1 (conntrack-tools): 0 expectations have been shown.

Any idea where to find what is still causing the kernel to drop packets?

I still need to handle the INVALID state, but here is my current rule-set:

# Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
*raw
:PREROUTING ACCEPT [559078817:46637850935]
:OUTPUT ACCEPT [539455981:114717831525]
[311121010:22258566347] -A PREROUTING -p udp -m udp --dport 53 -j CT --notrack
[82158007:15791189816] -A PREROUTING -p udp -m udp --sport 53 -j CT --notrack
[160638557:7854716900] -A PREROUTING -p tcp -m tcp --dport 53 -j CT --notrack
[55174:13694242] -A PREROUTING -p tcp -m tcp --sport 53 -j CT --notrack
[82898530:7187312985] -A OUTPUT -p udp -m udp --dport 53 -j CT --notrack
[310815684:92319143568] -A OUTPUT -p udp -m udp --sport 53 -j CT --notrack
[81356:4991618] -A OUTPUT -p tcp -m tcp --dport 53 -j CT --notrack
[134221259:13321766219] -A OUTPUT -p tcp -m tcp --sport 53 -j CT --notrack
COMMIT
# Completed on Wed Nov  6 11:40:00 2024
# Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
*nat
:PREROUTING ACCEPT [2196269:120347484]
:INPUT ACCEPT [1191964:55756667]
:OUTPUT ACCEPT [43401:4029238]
:POSTROUTING ACCEPT [43401:4029238]
COMMIT
# Completed on Wed Nov  6 11:40:00 2024
# Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
*mangle
:PREROUTING ACCEPT [559078013:46637808723]
:INPUT ACCEPT [559078013:46637808723]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [539456018:114717838594]
:POSTROUTING ACCEPT [539456018:114717838594]
COMMIT
# Completed on Wed Nov  6 11:40:00 2024
# Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
*filter
:INPUT DROP [1028561:66625259]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [539455943:114717822181]
[555959622:46411126809] -A INPUT -m conntrack --ctstate
RELATED,ESTABLISHED,UNTRACKED -j ACCEPT
[158:11928] -A INPUT -i lo -j ACCEPT
[0:0] -A INPUT -s 10.101.0.0/16 -p icmp -j ACCEPT
[1283248:91447747] -A INPUT -s 10.212.0.0/16 -p icmp -j ACCEPT
[0:0] -A INPUT -s 10.84.0.0/16 -p icmp -j ACCEPT
[23101:2217696] -A INPUT -s 10.57.0.0/16 -p icmp -j ACCEPT
[0:0] -A INPUT -s 10.94.0.0/16 -p icmp -j ACCEPT
[0:0] -A INPUT -s 127.0.0.0/8 -p icmp -j ACCEPT
[767154:60545933] -A INPUT -s 10.0.0.0/8 -p icmp -j ACCEPT
[0:0] -A INPUT -s 172.16.0.0/12 -p icmp -j ACCEPT
[12243:5597503] -A INPUT -s 192.168.0.0/16 -p icmp -j ACCEPT
[0:0] -A INPUT -s 100.64.0.0/10 -p icmp -j ACCEPT
[3:243] -A INPUT -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
[0:0] -A INPUT -s 10.212.109.49/32 -p tcp -m tcp --dport 8139 -j ACCEPT
[3888:233280] -A INPUT -s 10.212.0.0/17 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 10.212.128.0/18 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 10.212.192.0/19 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 127.0.0.0/8 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 10.0.0.0/8 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 172.16.0.0/12 -p tcp -m tcp --dport 22 -j ACCEPT
[8:416] -A INPUT -s 192.168.0.0/16 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 100.72.0.0/15 -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -s 10.94.202.160/27 -p icmp -j ACCEPT
COMMIT
# Completed on Wed Nov  6 11:40:00 2024

Thanks for any help!

-m

Re: connection tracking and kernel dropping packets

[Slavko]

On 6. novembra 2024 17:44:50 UTC, Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:

>I've waited a week to let the TCP streams in the conntrack table time
>out. I'm still seeing the kernel drop packets:

># cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
>65536

># conntrack -L > /dev/null
>conntrack v1.2.1 (conntrack-tools): 22 flow entries have been shown.

Your system seems to be broken, you can have zombie connections or
you meet same (old) bug.

Try to reboot it, with a little of luck it will solve that (i roughly remember
some problems, in old kernels, after conntrack become full, but i am not
sure), othervise you can start to track it from zero and try to find source.

Perhaps unload/load the conntrack module will help too...

BTW, you can show conntracks count by:

    conntrack -C

You can test which number it display, mine matches...

regards


-- 
Slavko
https://www.slavino.sk/

Re: connection tracking and kernel dropping packets

[Florian Westphal]

Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> On Tue, Oct 29, 2024 at 10:48 AM Slavko <linux@slavino.sk> wrote:
> >
> > Dňa 29. októbra 2024 15:11:34 UTC používateľ Matt Zagrabelny <mzagrabe@d.umn.edu> napísal:
> >
> > >...but it is still dropping packets due to the CT.
> >
> > You have first to inspect what is filling your conntrack table:
> >
> >     conntrack -L
> 
> I've waited a week to let the TCP streams in the conntrack table time
> out. I'm still seeing the kernel drop packets:

conntrack -F not working?

> # tail -f /var/log/kern.log
> Nov  6 11:29:02 netadm kernel: [48773744.961053] nf_conntrack: table
> full, dropping packet.
> 
> ...and confirmed with /proc:
> 
> # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
> 65536
> 
> ...but there aren't that many flows in the conntrack table, and none
> in the expect table.
> 
> # conntrack -L > /dev/null
> conntrack v1.2.1 (conntrack-tools): 22 flow entries have been shown.

Depending on userspace/kernel version this may only list ipv4.

> Any idea where to find what is still causing the kernel to drop packets?
> 
> I still need to handle the INVALID state, but here is my current rule-set:
> 
> # Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
> *raw
> :PREROUTING ACCEPT [559078817:46637850935]
> :OUTPUT ACCEPT [539455981:114717831525]
> [311121010:22258566347] -A PREROUTING -p udp -m udp --dport 53 -j CT --notrack
> [82158007:15791189816] -A PREROUTING -p udp -m udp --sport 53 -j CT --notrack
> [160638557:7854716900] -A PREROUTING -p tcp -m tcp --dport 53 -j CT --notrack
> [55174:13694242] -A PREROUTING -p tcp -m tcp --sport 53 -j CT --notrack
> [82898530:7187312985] -A OUTPUT -p udp -m udp --dport 53 -j CT --notrack
> [310815684:92319143568] -A OUTPUT -p udp -m udp --sport 53 -j CT --notrack
> [81356:4991618] -A OUTPUT -p tcp -m tcp --dport 53 -j CT --notrack
> [134221259:13321766219] -A OUTPUT -p tcp -m tcp --sport 53 -j CT --notrack
> COMMIT

Is this an ipv4 only system?

Re: connection tracking and kernel dropping packets

[Matt Zagrabelny]

Hi Florian,


On Sun, Nov 10, 2024 at 3:48 PM Florian Westphal <fw@strlen.de> wrote:
>
> Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
> > On Tue, Oct 29, 2024 at 10:48 AM Slavko <linux@slavino.sk> wrote:
> > >
> > > Dňa 29. októbra 2024 15:11:34 UTC používateľ Matt Zagrabelny <mzagrabe@d.umn.edu> napísal:
> > >
> > > >...but it is still dropping packets due to the CT.
> > >
> > > You have first to inspect what is filling your conntrack table:
> > >
> > >     conntrack -L
> >
> > I've waited a week to let the TCP streams in the conntrack table time
> > out. I'm still seeing the kernel drop packets:
>
> conntrack -F not working?

I didn't know about it. Just tried it. Seemed to work.

>
> > # tail -f /var/log/kern.log
> > Nov  6 11:29:02 netadm kernel: [48773744.961053] nf_conntrack: table
> > full, dropping packet.
> >
> > ...and confirmed with /proc:
> >
> > # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
> > 65536
> >
> > ...but there aren't that many flows in the conntrack table, and none
> > in the expect table.
> >
> > # conntrack -L > /dev/null
> > conntrack v1.2.1 (conntrack-tools): 22 flow entries have been shown.
>
> Depending on userspace/kernel version this may only list ipv4.

Ahh...

>
> > Any idea where to find what is still causing the kernel to drop packets?
> >
> > I still need to handle the INVALID state, but here is my current rule-set:
> >
> > # Generated by iptables-save v1.4.14 on Wed Nov  6 11:40:00 2024
> > *raw
> > :PREROUTING ACCEPT [559078817:46637850935]
> > :OUTPUT ACCEPT [539455981:114717831525]
> > [311121010:22258566347] -A PREROUTING -p udp -m udp --dport 53 -j CT --notrack
> > [82158007:15791189816] -A PREROUTING -p udp -m udp --sport 53 -j CT --notrack
> > [160638557:7854716900] -A PREROUTING -p tcp -m tcp --dport 53 -j CT --notrack
> > [55174:13694242] -A PREROUTING -p tcp -m tcp --sport 53 -j CT --notrack
> > [82898530:7187312985] -A OUTPUT -p udp -m udp --dport 53 -j CT --notrack
> > [310815684:92319143568] -A OUTPUT -p udp -m udp --sport 53 -j CT --notrack
> > [81356:4991618] -A OUTPUT -p tcp -m tcp --dport 53 -j CT --notrack
> > [134221259:13321766219] -A OUTPUT -p tcp -m tcp --sport 53 -j CT --notrack
> > COMMIT
>
> Is this an ipv4 only system?

Nope. Dual-homed. Thanks for the reminder. I've added the
corresponding "notrack" rules to the v6 side.

Thanks again for the help!

-m

Re: connection tracking and kernel dropping packets

[Kerin Millar]

On Tue, 29 Oct 2024, at 3:11 PM, Matt Zagrabelny wrote:
> Hello,
>
> The kernel of my DNS server is dropping packets:
>
> kernel: [48074703.302657] nf_conntrack: table full, dropping packet.
>
> I've followed this knowledge base article:
>
> https://kb.isc.org/docs/aa-01183
>
> for configuring iptables to not track the connections of DNS packets.
>
> ...but it is still dropping packets due to the CT.
>
> I know I could bump the conntrack memory, but I'd still like to know
> why my iptables rules aren't sufficient for not dropping DNS packets.
>
> I'm running Linux 3.2.0-4-amd64
>
> Here are my iptables rules:
>
> # iptables -vnL -t raw
> Chain PREROUTING (policy ACCEPT 24M packets, 2112M bytes)
>  pkts bytes target     prot opt in     out     source               destination
>   16M 1153M CT         udp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            udp dpt:53 CT notrack
> 3723K  701M CT         udp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            udp spt:53 CT notrack
>
> Chain OUTPUT (policy ACCEPT 24M packets, 5436M bytes)
>  pkts bytes target     prot opt in     out     source               destination
> 3760K  327M CT         udp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            udp dpt:53 CT notrack
>   16M 4680M CT         udp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            udp spt:53 CT notrack
>
> # iptables -vnL
> Chain INPUT (policy DROP 45173 packets, 2842K bytes)
>  pkts bytes target     prot opt in     out     source               destination
>   23M 2065M ACCEPT     all  --  *      *       0.0.0.0/0
> 0.0.0.0/0            ctstate RELATED,ESTABLISHED,UNTRACKED
>    11   804 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
>     4   284 ACCEPT     udp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            udp dpt:53
>  709K   37M ACCEPT     tcp  --  *      *       0.0.0.0/0
> 0.0.0.0/0            tcp dpt:53
>
> Chain FORWARD (policy DROP 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Chain OUTPUT (policy ACCEPT 24M packets, 5439M bytes)
>  pkts bytes target     prot opt in     out     source               destination
>
> Any ideas what I'm missing?

Please use iptables-save -c for dumping rulesets.

You can inspect the contents of the conntrack table by using the conntrack(8) utility from conntrack-tools and, in so doing, better understand the manner in which it comes to be full.

By default, TCP sessions that aren't formally ended can occupy a conntrack table entry for up to a week, per the value of the "net.netfilter.nf_conntrack_tcp_timeout_established" sysctl. Also, "net.nf_conntrack_max" can be used to increase the maximum number of entries. There is nothing wrong with tuning these controls in accordance with your workload, if needs be.

Incidentally, your ruleset allows for packets classified as INVALID to traverse the INPUT chain in full, which isn't advisable. As was discussed recently, either contend with INVALID packets up front or additionally match on the NEW state for subsequent rules, where appropriate.

Finally, you might consider whether you need bother using iptables at all, since this ruleset doesn't appear to be doing a great deal. Are you trying to limit the exposure of other network services that this host is running? If so, binding them to a particular address may be an option. Further, there are various scenarios in which connection tracking isn't necessarily useful, in which case one is certainly not obliged to use it. That your host isn't tasked with forwarding packets indicates that you should at least ponder upon this.

-- 
Kerin Millar