[bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

Hello all,

I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
Running cat /proc/net/nf_conntrack showed a large number of
connections in the SYN_SENT state.
As is well known, if we attempt to connect to a non-existent port, the
system will respond with an RST and then delete the conntrack entry.
However, when we frequently connect to non-existent ports, the
conntrack entries are not deleted, eventually causing the nf_conntrack
table to fill up.

The problem can be reproduced using the following command:
hping3 -S -V -p 9007 --flood xx.x.xxx.xxx

~$ cat /proc/net/nf_conntrack
ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=2642 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=2642 mark=0 zone=0 use=2
ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=11510 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=11510 mark=0 zone=0 use=2
ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=28611 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=28611 mark=0 zone=0 use=2
ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=62849 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=62849 mark=0 zone=0 use=2
ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=3410 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=3410 mark=0 zone=0 use=2
ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=44185 dport=9007 [UNREPLIED] src=xx.xx.xx.xx dst=xx.x.xxx.xxx
sport=9007 dport=44185 mark=0 zone=0 use=2
ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=51099 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=51099 mark=0 zone=0 use=2
ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
sport=23609 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
dport=23609 mark=0 zone=0 use=2

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Eric Dumazet]

On Wed, May 28, 2025 at 5:52 AM ying chen <yc1082463@gmail.com> wrote:
>
> Hello all,
>
> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> Running cat /proc/net/nf_conntrack showed a large number of
> connections in the SYN_SENT state.
> As is well known, if we attempt to connect to a non-existent port, the
> system will respond with an RST and then delete the conntrack entry.
> However, when we frequently connect to non-existent ports, the
> conntrack entries are not deleted, eventually causing the nf_conntrack
> table to fill up.
>
> The problem can be reproduced using the following command:
> hping3 -S -V -p 9007 --flood xx.x.xxx.xxx
>
> ~$ cat /proc/net/nf_conntrack
> ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=2642 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=2642 mark=0 zone=0 use=2
> ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=11510 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=11510 mark=0 zone=0 use=2
> ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=28611 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=28611 mark=0 zone=0 use=2
> ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=62849 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=62849 mark=0 zone=0 use=2
> ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=3410 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=3410 mark=0 zone=0 use=2
> ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=44185 dport=9007 [UNREPLIED] src=xx.xx.xx.xx dst=xx.x.xxx.xxx
> sport=9007 dport=44185 mark=0 zone=0 use=2
> ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=51099 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=51099 mark=0 zone=0 use=2
> ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> sport=23609 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> dport=23609 mark=0 zone=0 use=2

The default timeout is 120 seconds.

/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_syn_sent

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Florian Westphal]

ying chen <yc1082463@gmail.com> wrote:
> Hello all,
> 
> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> Running cat /proc/net/nf_conntrack showed a large number of
> connections in the SYN_SENT state.
> As is well known, if we attempt to connect to a non-existent port, the
> system will respond with an RST and then delete the conntrack entry.
> However, when we frequently connect to non-existent ports, the
> conntrack entries are not deleted, eventually causing the nf_conntrack
> table to fill up.

Yes, what do you expect to happen?

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
>
> ying chen <yc1082463@gmail.com> wrote:
> > Hello all,
> >
> > I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> > Running cat /proc/net/nf_conntrack showed a large number of
> > connections in the SYN_SENT state.
> > As is well known, if we attempt to connect to a non-existent port, the
> > system will respond with an RST and then delete the conntrack entry.
> > However, when we frequently connect to non-existent ports, the
> > conntrack entries are not deleted, eventually causing the nf_conntrack
> > table to fill up.
>
> Yes, what do you expect to happen?
I understand that the conntrack entry should be deleted immediately
after receiving the RST reply.

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

On Wed, May 28, 2025 at 8:59 PM Eric Dumazet <edumazet@google.com> wrote:
>
> On Wed, May 28, 2025 at 5:52 AM ying chen <yc1082463@gmail.com> wrote:
> >
> > Hello all,
> >
> > I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> > Running cat /proc/net/nf_conntrack showed a large number of
> > connections in the SYN_SENT state.
> > As is well known, if we attempt to connect to a non-existent port, the
> > system will respond with an RST and then delete the conntrack entry.
> > However, when we frequently connect to non-existent ports, the
> > conntrack entries are not deleted, eventually causing the nf_conntrack
> > table to fill up.
> >
> > The problem can be reproduced using the following command:
> > hping3 -S -V -p 9007 --flood xx.x.xxx.xxx
> >
> > ~$ cat /proc/net/nf_conntrack
> > ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=2642 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=2642 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=11510 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=11510 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=28611 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=28611 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=62849 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=62849 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=3410 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=3410 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=44185 dport=9007 [UNREPLIED] src=xx.xx.xx.xx dst=xx.x.xxx.xxx
> > sport=9007 dport=44185 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 111 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=51099 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=51099 mark=0 zone=0 use=2
> > ipv4     2 tcp      6 112 SYN_SENT src=xx.x.xxx.xxx dst=xx.xx.xx.xx
> > sport=23609 dport=9007 src=xx.xx.xx.xx dst=xx.x.xxx.xxx sport=9007
> > dport=23609 mark=0 zone=0 use=2
>
> The default timeout is 120 seconds.
>
> /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_syn_sent
Yes，The  timeout is 120 seconds.

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Eric Dumazet]

On Wed, May 28, 2025 at 6:26 AM ying chen <yc1082463@gmail.com> wrote:
>
> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > ying chen <yc1082463@gmail.com> wrote:
> > > Hello all,
> > >
> > > I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> > > Running cat /proc/net/nf_conntrack showed a large number of
> > > connections in the SYN_SENT state.
> > > As is well known, if we attempt to connect to a non-existent port, the
> > > system will respond with an RST and then delete the conntrack entry.
> > > However, when we frequently connect to non-existent ports, the
> > > conntrack entries are not deleted, eventually causing the nf_conntrack
> > > table to fill up.
> >
> > Yes, what do you expect to happen?
> I understand that the conntrack entry should be deleted immediately
> after receiving the RST reply.

Then it probably hints that you do not receive RST for all your SYN packets.

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Jozsef Kadlecsik]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Wed, 28 May 2025, ying chen wrote:

> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
>>
>> ying chen <yc1082463@gmail.com> wrote:
>>> Hello all,
>>>
>>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
>>> Running cat /proc/net/nf_conntrack showed a large number of
>>> connections in the SYN_SENT state.
>>> As is well known, if we attempt to connect to a non-existent port, the
>>> system will respond with an RST and then delete the conntrack entry.
>>> However, when we frequently connect to non-existent ports, the
>>> conntrack entries are not deleted, eventually causing the nf_conntrack
>>> table to fill up.
>>
>> Yes, what do you expect to happen?
> I understand that the conntrack entry should be deleted immediately 
> after receiving the RST reply.

No, the conntrack entry will be in the CLOSE state with the timeout value 
of /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close

Best regards,
Jozsef

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Jozsef Kadlecsik]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Wed, 28 May 2025, Eric Dumazet wrote:

> On Wed, May 28, 2025 at 6:26 AM ying chen <yc1082463@gmail.com> wrote:
>>
>> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
>>>
>>> ying chen <yc1082463@gmail.com> wrote:
>>>> Hello all,
>>>>
>>>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
>>>> Running cat /proc/net/nf_conntrack showed a large number of
>>>> connections in the SYN_SENT state.
>>>> As is well known, if we attempt to connect to a non-existent port, the
>>>> system will respond with an RST and then delete the conntrack entry.
>>>> However, when we frequently connect to non-existent ports, the
>>>> conntrack entries are not deleted, eventually causing the nf_conntrack
>>>> table to fill up.
>>>
>>> Yes, what do you expect to happen?
>> I understand that the conntrack entry should be deleted immediately
>> after receiving the RST reply.
>
> Then it probably hints that you do not receive RST for all your SYN 
> packets.

And Eric has got right: because the states are in SYN_SENT then either the 
RST packets were not received or out of the window or invalid from other 
reasons.

Best regards,
Jozsef

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

On Wed, May 28, 2025 at 9:41 PM Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
>
> On Wed, 28 May 2025, ying chen wrote:
>
> > On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
> >>
> >> ying chen <yc1082463@gmail.com> wrote:
> >>> Hello all,
> >>>
> >>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> >>> Running cat /proc/net/nf_conntrack showed a large number of
> >>> connections in the SYN_SENT state.
> >>> As is well known, if we attempt to connect to a non-existent port, the
> >>> system will respond with an RST and then delete the conntrack entry.
> >>> However, when we frequently connect to non-existent ports, the
> >>> conntrack entries are not deleted, eventually causing the nf_conntrack
> >>> table to fill up.
> >>
> >> Yes, what do you expect to happen?
> > I understand that the conntrack entry should be deleted immediately
> > after receiving the RST reply.
>
> No, the conntrack entry will be in the CLOSE state with the timeout value
> of /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close
>
> Best regards,
> Jozsef
The conntrack entry does not transition to the CLOSE state and remains
in the SYN_SENT state until the nf_conntrack_tcp_timeout_syn_sent
timeout is reached.
According to the code, the conntrack  entry should be deleted
immediately after the RST reply.
int nf_conntrack_tcp_packet(struct nf_conn *ct,
                            struct sk_buff *skb,
                            unsigned int dataoff,
                            enum ip_conntrack_info ctinfo,
                            const struct nf_hook_state *state)
{
        ......
        if (!test_bit(IPS_SEEN_REPLY_BIT, &ct->status)) {
                /* If only reply is a RST, we can consider ourselves not to
                   have an established connection: this is a fairly common
                   problem case, so we can delete the conntrack
                   immediately.  --RR */
                if (th->rst) {
                        nf_ct_kill_acct(ct, ctinfo, skb);
                        return NF_ACCEPT;
                }

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

On Wed, May 28, 2025 at 9:45 PM Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
>
> On Wed, 28 May 2025, Eric Dumazet wrote:
>
> > On Wed, May 28, 2025 at 6:26 AM ying chen <yc1082463@gmail.com> wrote:
> >>
> >> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
> >>>
> >>> ying chen <yc1082463@gmail.com> wrote:
> >>>> Hello all,
> >>>>
> >>>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> >>>> Running cat /proc/net/nf_conntrack showed a large number of
> >>>> connections in the SYN_SENT state.
> >>>> As is well known, if we attempt to connect to a non-existent port, the
> >>>> system will respond with an RST and then delete the conntrack entry.
> >>>> However, when we frequently connect to non-existent ports, the
> >>>> conntrack entries are not deleted, eventually causing the nf_conntrack
> >>>> table to fill up.
> >>>
> >>> Yes, what do you expect to happen?
> >> I understand that the conntrack entry should be deleted immediately
> >> after receiving the RST reply.
> >
> > Then it probably hints that you do not receive RST for all your SYN
> > packets.
>
> And Eric has got right: because the states are in SYN_SENT then either the
> RST packets were not received or out of the window or invalid from other
> reasons.
>
> Best regards,
> Jozsef
I also suspect it's due to being "out of the window", but I'm not sure why.

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[Jozsef Kadlecsik]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On Wed, 28 May 2025, ying chen wrote:

> On Wed, May 28, 2025 at 9:45 PM Jozsef Kadlecsik
> <kadlec@blackhole.kfki.hu> wrote:
>>
>> On Wed, 28 May 2025, Eric Dumazet wrote:
>>
>>> On Wed, May 28, 2025 at 6:26 AM ying chen <yc1082463@gmail.com> wrote:
>>>>
>>>> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
>>>>>
>>>>> ying chen <yc1082463@gmail.com> wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
>>>>>> Running cat /proc/net/nf_conntrack showed a large number of
>>>>>> connections in the SYN_SENT state.
>>>>>> As is well known, if we attempt to connect to a non-existent port, the
>>>>>> system will respond with an RST and then delete the conntrack entry.
>>>>>> However, when we frequently connect to non-existent ports, the
>>>>>> conntrack entries are not deleted, eventually causing the nf_conntrack
>>>>>> table to fill up.
>>>>>
>>>>> Yes, what do you expect to happen?
>>>> I understand that the conntrack entry should be deleted immediately
>>>> after receiving the RST reply.
>>>
>>> Then it probably hints that you do not receive RST for all your SYN
>>> packets.
>>
>> And Eric has got right: because the states are in SYN_SENT then either the
>> RST packets were not received or out of the window or invalid from other
>> reasons.
> I also suspect it's due to being "out of the window", but I'm not sure why.

tcpdump of the traffic from the targeted machine with both the SYN and RST 
packets could help (raw pcap or at least the output with absolute seqs).

Best regards,
Jozsef

Re: [bug report, linux 6.15-rc4] A large number of connections in the SYN_SENT state caused the nf_conntrack table to be full.

[ying chen]

On Wed, May 28, 2025 at 10:18 PM Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
>
> On Wed, 28 May 2025, ying chen wrote:
>
> > On Wed, May 28, 2025 at 9:45 PM Jozsef Kadlecsik
> > <kadlec@blackhole.kfki.hu> wrote:
> >>
> >> On Wed, 28 May 2025, Eric Dumazet wrote:
> >>
> >>> On Wed, May 28, 2025 at 6:26 AM ying chen <yc1082463@gmail.com> wrote:
> >>>>
> >>>> On Wed, May 28, 2025 at 9:10 PM Florian Westphal <fw@strlen.de> wrote:
> >>>>>
> >>>>> ying chen <yc1082463@gmail.com> wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> I encountered an "nf_conntrack: table full" warning on Linux 6.15-rc4.
> >>>>>> Running cat /proc/net/nf_conntrack showed a large number of
> >>>>>> connections in the SYN_SENT state.
> >>>>>> As is well known, if we attempt to connect to a non-existent port, the
> >>>>>> system will respond with an RST and then delete the conntrack entry.
> >>>>>> However, when we frequently connect to non-existent ports, the
> >>>>>> conntrack entries are not deleted, eventually causing the nf_conntrack
> >>>>>> table to fill up.
> >>>>>
> >>>>> Yes, what do you expect to happen?
> >>>> I understand that the conntrack entry should be deleted immediately
> >>>> after receiving the RST reply.
> >>>
> >>> Then it probably hints that you do not receive RST for all your SYN
> >>> packets.
> >>
> >> And Eric has got right: because the states are in SYN_SENT then either the
> >> RST packets were not received or out of the window or invalid from other
> >> reasons.
> > I also suspect it's due to being "out of the window", but I'm not sure why.
>
> tcpdump of the traffic from the targeted machine with both the SYN and RST
> packets could help (raw pcap or at least the output with absolute seqs).
>
> Best regards,
> Jozsef

Using bpftrace, I found that the RST is under the lower bound and
printed the values of the following variables:
receiver->td_maxwin = 1
sender->td_end = 0
receiver->td_maxwin =1