[RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[S.S.Vikash]

Hi,

While investigating connection tracking exhaustion behavior under a spoofed
SYN flood, I observed that the effectiveness of early eviction in
net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
nf_conntrack_max is configured to be substantially smaller than
nf_conntrack_buckets.

Setup & Reproduction:
- Host kernel: Linux 6.14 (Proxmox host)
- Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
  connections generated by spoofed SYN traffic.
- Test: Verifying whether an inbound, legitimate SSH connection can be
  established while the table is at capacity.

Results observed:

nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
------------------------------+-----------------------------------+----------------------------------
              4096              |              262144
|         Failed (drops)
              4096              |              4096
  |         Established
              65536            |              262144
|         Established
              262144          |              262144                  |
        Established

Root Cause Analysis:
In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
consecutive hash buckets starting from the calculated hash index.

With nf_conntrack_max=4096 and nf_conntrack_buckets=262144, the hash table
is sparsely populated. Since early_drop() examines only eight consecutive
buckets, it can fail to locate an eligible entry even when many non-ASSURED
connections remain in the table. This appears consistent with the connection
failures observed during testing.

When the bucket count is reduced or when the table density is increased,
early eviction succeeds reliably and permits legitimate inbound connections.

Proposed Improvements:
I am interested in addressing this behavior upstream and would appreciate
feedback on which direction would be most useful:

1. Diagnostic:
   Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
   significantly lower than nf_conntrack_buckets via sysctl, notifying the
   operator that early_drop() heuristics may become ineffective under load.

2. Documentation:
   Update Documentation/networking/nf_conntrack-sysctl.rst to document the
   tuning implications of sparse hash tables on early eviction.

3. Algorithmic:
   Investigate a bounded improvement to candidate selection in sparse tables
   without introducing unacceptable CPU overhead on the drop path.

I'm happy to prepare a patch or provide additional test measurements based
on the discussion.

Thanks,
Vikash SS

Re: [RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[S.S.Vikash]

Correction: The results table was mangled by Gmail
formatting. Reposting the results below.

nf_conntrack_max=4096
nf_conntrack_buckets=262144
SSH: Failed (drops)

nf_conntrack_max=4096
nf_conntrack_buckets=4096
SSH: Established

nf_conntrack_max=65536
nf_conntrack_buckets=262144
SSH: Established

nf_conntrack_max=262144
nf_conntrack_buckets=262144
SSH: Established

On Fri, Nov 6, 2026 at 4:23 PM S.S.Vikash <arisivikash@gmail.com> wrote:
>
> Hi,
>
> While investigating connection tracking exhaustion behavior under a spoofed
> SYN flood, I observed that the effectiveness of early eviction in
> net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
> nf_conntrack_max is configured to be substantially smaller than
> nf_conntrack_buckets.
>
> Setup & Reproduction:
> - Host kernel: Linux 6.14 (Proxmox host)
> - Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
>   connections generated by spoofed SYN traffic.
> - Test: Verifying whether an inbound, legitimate SSH connection can be
>   established while the table is at capacity.
>
> Results observed:
>
> nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
> ------------------------------+-----------------------------------+----------------------------------
>               4096              |              262144
> |         Failed (drops)
>               4096              |              4096
>   |         Established
>               65536            |              262144
> |         Established
>               262144          |              262144                  |
>         Established
>
> Root Cause Analysis:
> In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
> consecutive hash buckets starting from the calculated hash index.
>
> With nf_conntrack_max=4096 and nf_conntrack_buckets=262144, the hash table
> is sparsely populated. Since early_drop() examines only eight consecutive
> buckets, it can fail to locate an eligible entry even when many non-ASSURED
> connections remain in the table. This appears consistent with the connection
> failures observed during testing.
>
> When the bucket count is reduced or when the table density is increased,
> early eviction succeeds reliably and permits legitimate inbound connections.
>
> Proposed Improvements:
> I am interested in addressing this behavior upstream and would appreciate
> feedback on which direction would be most useful:
>
> 1. Diagnostic:
>    Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
>    significantly lower than nf_conntrack_buckets via sysctl, notifying the
>    operator that early_drop() heuristics may become ineffective under load.
>
> 2. Documentation:
>    Update Documentation/networking/nf_conntrack-sysctl.rst to document the
>    tuning implications of sparse hash tables on early eviction.
>
> 3. Algorithmic:
>    Investigate a bounded improvement to candidate selection in sparse tables
>    without introducing unacceptable CPU overhead on the drop path.
>
> I'm happy to prepare a patch or provide additional test measurements based
> on the discussion.
>
> Thanks,
> Vikash SS

Re: [RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[Pablo Neira Ayuso]

On Fri, Nov 06, 2026 at 04:23:08PM +0530, S.S.Vikash wrote:
> Hi,
> 
> While investigating connection tracking exhaustion behavior under a spoofed
> SYN flood, I observed that the effectiveness of early eviction in
> net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
> nf_conntrack_max is configured to be substantially smaller than
> nf_conntrack_buckets.
> 
> Setup & Reproduction:
> - Host kernel: Linux 6.14 (Proxmox host)
> - Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
>   connections generated by spoofed SYN traffic.
> - Test: Verifying whether an inbound, legitimate SSH connection can be
>   established while the table is at capacity.
> 
> Results observed:
> 
> nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
> ------------------------------+-----------------------------------+----------------------------------
>               4096              |              262144
> |         Failed (drops)
>               4096              |              4096
>   |         Established
>               65536            |              262144
> |         Established
>               262144          |              262144                  |
>         Established
> 
> Root Cause Analysis:
> In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
> consecutive hash buckets starting from the calculated hash index.
> 
> With nf_conntrack_max=4096 and nf_conntrack_buckets=262144,

makes no sense to make to configure it in this way, there are well
define rules aiming at maintaining O(1) in hashtables and this
configuration is clearly suboptimal.

> the hash table
> is sparsely populated. Since early_drop() examines only eight consecutive
> buckets, it can fail to locate an eligible entry even when many non-ASSURED
> connections remain in the table. This appears consistent with the connection
> failures observed during testing.
> 
> When the bucket count is reduced or when the table density is increased,
> early eviction succeeds reliably and permits legitimate inbound connections.
> 
> Proposed Improvements:
> I am interested in addressing this behavior upstream and would appreciate
> feedback on which direction would be most useful:
> 
> 1. Diagnostic:
>    Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
>    significantly lower than nf_conntrack_buckets via sysctl, notifying the
>    operator that early_drop() heuristics may become ineffective under load.
> 
> 2. Documentation:
>    Update Documentation/networking/nf_conntrack-sysctl.rst to document the
>    tuning implications of sparse hash tables on early eviction.

Documentation is good to have, I would suggest you start with this.

> 3. Algorithmic:
>    Investigate a bounded improvement to candidate selection in sparse tables
>    without introducing unacceptable CPU overhead on the drop path.

You mean, to improve the situation for this suboptimal configuration?

> I'm happy to prepare a patch or provide additional test measurements based
> on the discussion.
> 
> Thanks,
> Vikash SS

Re: [RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[S.S.Vikash]

Hi Pablo,

I understand your point about the suboptimal configuration.

I came across this behavior while experimenting with conntrack and
investigated the source to understand why it was happening.

I'm relatively new to kernel development, so I appreciate the guidance
on what would be useful upstream.

I'll start with the documentation improvement.

Thanks for your time!

On Fri, Nov 6, 2026 at 11:40 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> On Fri, Nov 06, 2026 at 04:23:08PM +0530, S.S.Vikash wrote:
> > Hi,
> >
> > While investigating connection tracking exhaustion behavior under a spoofed
> > SYN flood, I observed that the effectiveness of early eviction in
> > net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
> > nf_conntrack_max is configured to be substantially smaller than
> > nf_conntrack_buckets.
> >
> > Setup & Reproduction:
> > - Host kernel: Linux 6.14 (Proxmox host)
> > - Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
> >   connections generated by spoofed SYN traffic.
> > - Test: Verifying whether an inbound, legitimate SSH connection can be
> >   established while the table is at capacity.
> >
> > Results observed:
> >
> > nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
> > ------------------------------+-----------------------------------+----------------------------------
> >               4096              |              262144
> > |         Failed (drops)
> >               4096              |              4096
> >   |         Established
> >               65536            |              262144
> > |         Established
> >               262144          |              262144                  |
> >         Established
> >
> > Root Cause Analysis:
> > In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
> > consecutive hash buckets starting from the calculated hash index.
> >
> > With nf_conntrack_max=4096 and nf_conntrack_buckets=262144,
>
> makes no sense to make to configure it in this way, there are well
> define rules aiming at maintaining O(1) in hashtables and this
> configuration is clearly suboptimal.
>
> > the hash table
> > is sparsely populated. Since early_drop() examines only eight consecutive
> > buckets, it can fail to locate an eligible entry even when many non-ASSURED
> > connections remain in the table. This appears consistent with the connection
> > failures observed during testing.
> >
> > When the bucket count is reduced or when the table density is increased,
> > early eviction succeeds reliably and permits legitimate inbound connections.
> >
> > Proposed Improvements:
> > I am interested in addressing this behavior upstream and would appreciate
> > feedback on which direction would be most useful:
> >
> > 1. Diagnostic:
> >    Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
> >    significantly lower than nf_conntrack_buckets via sysctl, notifying the
> >    operator that early_drop() heuristics may become ineffective under load.
> >
> > 2. Documentation:
> >    Update Documentation/networking/nf_conntrack-sysctl.rst to document the
> >    tuning implications of sparse hash tables on early eviction.
>
> Documentation is good to have, I would suggest you start with this.
>
> > 3. Algorithmic:
> >    Investigate a bounded improvement to candidate selection in sparse tables
> >    without introducing unacceptable CPU overhead on the drop path.
>
> You mean, to improve the situation for this suboptimal configuration?
>
> > I'm happy to prepare a patch or provide additional test measurements based
> > on the discussion.
> >
> > Thanks,
> > Vikash SS

Re: [RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[Pablo Neira Ayuso]

On Fri, Nov 06, 2026 at 11:52:47AM +0000, S.S.Vikash wrote:
> Hi Pablo,
> 
> I understand your point about the suboptimal configuration.
> 
> I came across this behavior while experimenting with conntrack and
> investigated the source to understand why it was happening.

OK, looks like a LLM finding then.

> I'm relatively new to kernel development, so I appreciate the guidance
> on what would be useful upstream.
> 
> I'll start with the documentation improvement.

Yes, thats sufficient. I don't see the need to improve the situation
for a suboptiomal configuration like the one described in this
scenario.

Thanks.

> Thanks for your time!
> 
> On Fri, Nov 6, 2026 at 11:40 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> >
> > On Fri, Nov 06, 2026 at 04:23:08PM +0530, S.S.Vikash wrote:
> > > Hi,
> > >
> > > While investigating connection tracking exhaustion behavior under a spoofed
> > > SYN flood, I observed that the effectiveness of early eviction in
> > > net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
> > > nf_conntrack_max is configured to be substantially smaller than
> > > nf_conntrack_buckets.
> > >
> > > Setup & Reproduction:
> > > - Host kernel: Linux 6.14 (Proxmox host)
> > > - Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
> > >   connections generated by spoofed SYN traffic.
> > > - Test: Verifying whether an inbound, legitimate SSH connection can be
> > >   established while the table is at capacity.
> > >
> > > Results observed:
> > >
> > > nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
> > > ------------------------------+-----------------------------------+----------------------------------
> > >               4096              |              262144
> > > |         Failed (drops)
> > >               4096              |              4096
> > >   |         Established
> > >               65536            |              262144
> > > |         Established
> > >               262144          |              262144                  |
> > >         Established
> > >
> > > Root Cause Analysis:
> > > In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
> > > consecutive hash buckets starting from the calculated hash index.
> > >
> > > With nf_conntrack_max=4096 and nf_conntrack_buckets=262144,
> >
> > makes no sense to make to configure it in this way, there are well
> > define rules aiming at maintaining O(1) in hashtables and this
> > configuration is clearly suboptimal.
> >
> > > the hash table
> > > is sparsely populated. Since early_drop() examines only eight consecutive
> > > buckets, it can fail to locate an eligible entry even when many non-ASSURED
> > > connections remain in the table. This appears consistent with the connection
> > > failures observed during testing.
> > >
> > > When the bucket count is reduced or when the table density is increased,
> > > early eviction succeeds reliably and permits legitimate inbound connections.
> > >
> > > Proposed Improvements:
> > > I am interested in addressing this behavior upstream and would appreciate
> > > feedback on which direction would be most useful:
> > >
> > > 1. Diagnostic:
> > >    Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
> > >    significantly lower than nf_conntrack_buckets via sysctl, notifying the
> > >    operator that early_drop() heuristics may become ineffective under load.
> > >
> > > 2. Documentation:
> > >    Update Documentation/networking/nf_conntrack-sysctl.rst to document the
> > >    tuning implications of sparse hash tables on early eviction.
> >
> > Documentation is good to have, I would suggest you start with this.
> >
> > > 3. Algorithmic:
> > >    Investigate a bounded improvement to candidate selection in sparse tables
> > >    without introducing unacceptable CPU overhead on the drop path.
> >
> > You mean, to improve the situation for this suboptimal configuration?
> >
> > > I'm happy to prepare a patch or provide additional test measurements based
> > > on the discussion.
> > >
> > > Thanks,
> > > Vikash SS

Re: [RFC] netfilter: conntrack: early_drop eviction limitation under sparse table configurations (max << buckets)

[S.S.Vikash]

Hi Pablo,

Thanks for the suggestion. I've submitted a documentation patch
describing the early eviction limitation with sparse hash tables.

Patch: https://lore.kernel.org/netfilter-devel/20261106135816.391285-1-thisisforspotifyvikash@gmail.com/

Thanks,
Vikash

On Fri, Nov 6, 2026 at 1:56 PM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> On Fri, Nov 06, 2026 at 11:52:47AM +0000, S.S.Vikash wrote:
> > Hi Pablo,
> >
> > I understand your point about the suboptimal configuration.
> >
> > I came across this behavior while experimenting with conntrack and
> > investigated the source to understand why it was happening.
>
> OK, looks like a LLM finding then.
>
> > I'm relatively new to kernel development, so I appreciate the guidance
> > on what would be useful upstream.
> >
> > I'll start with the documentation improvement.
>
> Yes, thats sufficient. I don't see the need to improve the situation
> for a suboptiomal configuration like the one described in this
> scenario.
>
> Thanks.
>
> > Thanks for your time!
> >
> > On Fri, Nov 6, 2026 at 11:40 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> > >
> > > On Fri, Nov 06, 2026 at 04:23:08PM +0530, S.S.Vikash wrote:
> > > > Hi,
> > > >
> > > > While investigating connection tracking exhaustion behavior under a spoofed
> > > > SYN flood, I observed that the effectiveness of early eviction in
> > > > net/netfilter/nf_conntrack_core.c:early_drop() degrades significantly when
> > > > nf_conntrack_max is configured to be substantially smaller than
> > > > nf_conntrack_buckets.
> > > >
> > > > Setup & Reproduction:
> > > > - Host kernel: Linux 6.14 (Proxmox host)
> > > > - Workload: Conntrack table saturated with non-ASSURED, unreplied TCP
> > > >   connections generated by spoofed SYN traffic.
> > > > - Test: Verifying whether an inbound, legitimate SSH connection can be
> > > >   established while the table is at capacity.
> > > >
> > > > Results observed:
> > > >
> > > > nf_conntrack_max       |   nf_conntrack_buckets     |     New SSH Connection
> > > > ------------------------------+-----------------------------------+----------------------------------
> > > >               4096              |              262144
> > > > |         Failed (drops)
> > > >               4096              |              4096
> > > >   |         Established
> > > >               65536            |              262144
> > > > |         Established
> > > >               262144          |              262144                  |
> > > >         Established
> > > >
> > > > Root Cause Analysis:
> > > > In early_drop(), the kernel examines at most NF_CT_EVICTION_RANGE (8)
> > > > consecutive hash buckets starting from the calculated hash index.
> > > >
> > > > With nf_conntrack_max=4096 and nf_conntrack_buckets=262144,
> > >
> > > makes no sense to make to configure it in this way, there are well
> > > define rules aiming at maintaining O(1) in hashtables and this
> > > configuration is clearly suboptimal.
> > >
> > > > the hash table
> > > > is sparsely populated. Since early_drop() examines only eight consecutive
> > > > buckets, it can fail to locate an eligible entry even when many non-ASSURED
> > > > connections remain in the table. This appears consistent with the connection
> > > > failures observed during testing.
> > > >
> > > > When the bucket count is reduced or when the table density is increased,
> > > > early eviction succeeds reliably and permits legitimate inbound connections.
> > > >
> > > > Proposed Improvements:
> > > > I am interested in addressing this behavior upstream and would appreciate
> > > > feedback on which direction would be most useful:
> > > >
> > > > 1. Diagnostic:
> > > >    Warn (e.g., via pr_warn_once()) when nf_conntrack_max is configured to be
> > > >    significantly lower than nf_conntrack_buckets via sysctl, notifying the
> > > >    operator that early_drop() heuristics may become ineffective under load.
> > > >
> > > > 2. Documentation:
> > > >    Update Documentation/networking/nf_conntrack-sysctl.rst to document the
> > > >    tuning implications of sparse hash tables on early eviction.
> > >
> > > Documentation is good to have, I would suggest you start with this.
> > >
> > > > 3. Algorithmic:
> > > >    Investigate a bounded improvement to candidate selection in sparse tables
> > > >    without introducing unacceptable CPU overhead on the drop path.
> > >
> > > You mean, to improve the situation for this suboptimal configuration?
> > >
> > > > I'm happy to prepare a patch or provide additional test measurements based
> > > > on the discussion.
> > > >
> > > > Thanks,
> > > > Vikash SS