[conntrack-tools] conntrack.8: Document --stats counters

[conntrack-tools] conntrack.8: Document --stats counters

[Phil Sutter]

Provide at least a brief description of each counter's meaning based on
code-analysis in kernel's nf_conntrack_core.c.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
While most values are pretty obvious, I am not entirely sure I got the
insert_failed and drop reasons right.
---
 conntrack.8 | 37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/conntrack.8 b/conntrack.8
index 2bfd80e5d6aa4..b562e16839a32 100644
--- a/conntrack.8
+++ b/conntrack.8
@@ -108,7 +108,42 @@ Flush the whole given table
 Show the table counter.
 .TP
 .BI "-S, --stats "
-Show the in-kernel connection tracking system statistics.
+Show the in-kernel connection tracking system statistics. The returned values
+for each CPU are:
+.RS
+.TP
+.B found
+Number of successful conntrack table lookups
+.TP
+.B invalid
+Number of invalid packets encountered
+.TP
+.B insert
+Number of entries inserted into conntrack table
+.TP
+.B insert_failed
+Number of failed inserts (clash resolution failure, conntrack extensions in
+inconsistent state, entry in dying state, oversized hash bucket encountered)
+.TP
+.B drop
+Number of packets dropped (clash resolution failure, removed conntrack helper,
+stress, TCP connection aborted)
+.TP
+.B early_drop
+Number of packets dropped up front due to full table
+.TP
+.B error
+Number of invalid ICMP/ICMPv6 packets received
+.TP
+.B search_restart
+Number of times a table lookup had to be restarted due to table reorg
+.TP
+.B clash_resolve
+Number of entry insert clashes resolved
+.TP
+.B chaintoolong
+Number of oversized hash bucket encounters
+.RE
 .TP
 .BI "-R, --load-file "
 Load entries from a given file. To read from stdin, "\-" should be specified.
-- 
2.54.0

Re: [conntrack-tools] conntrack.8: Document --stats counters

[Florian Westphal]

Phil Sutter <phil@nwl.cc> wrote:
> Provide at least a brief description of each counter's meaning based on
> code-analysis in kernel's nf_conntrack_core.c.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> While most values are pretty obvious, I am not entirely sure I got the
> insert_failed and drop reasons right.

Thanks for working on this.   I think the hardest part is to explain
how to interpret this, i.e. which counters may indicate issues and which
ones do not.

> ---
>  conntrack.8 | 37 ++++++++++++++++++++++++++++++++++++-
>  1 file changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/conntrack.8 b/conntrack.8
> index 2bfd80e5d6aa4..b562e16839a32 100644
> --- a/conntrack.8
> +++ b/conntrack.8
> @@ -108,7 +108,42 @@ Flush the whole given table
>  Show the table counter.
>  .TP
>  .BI "-S, --stats "
> -Show the in-kernel connection tracking system statistics.
> +Show the in-kernel connection tracking system statistics. The returned values
> +for each CPU are:
> +.RS
> +.TP
> +.B found
> +Number of successful conntrack table lookups

We don't count those anymore, its too expensive.
This is only incremented when a new connection cannot reuse
the tuple and has to create a new nat mapping.

See nf_conntrack_tuple_taken().

> +.B insert
> +Number of entries inserted into conntrack table

IIRC its only incremented for insertions from netlink path.

> +.B insert_failed
> +Number of failed inserts (clash resolution failure, conntrack extensions in
> +inconsistent state, entry in dying state, oversized hash bucket encountered)

oversized hash buckets should not be mentioned here, they have their own
counter.  Should probably not increment both counters in kernel when it
happens.

Maybe: "Number of NEW connections dropped because of clashes with existing entry."?

> +.B drop
> +Number of packets dropped (clash resolution failure, removed conntrack helper,
> +stress, TCP connection aborted)

Yes, I think we need to fix this in the kernel and not increment two
counters for the same reason.

Drop should mostly mean "incomplete packet / out of memory".

> +.B early_drop
> +Number of packets dropped up front due to full table

AFAICS its number of connections dropped because of a full table.

> +.B search_restart
> +Number of times a table lookup had to be restarted due to table reorg

Not sure about this one.  This is an implementation detail (SLAB_TYPESAFE_BY_RCU).
This can happen at any time when entry is removed from table and
other CPU is re-instantiating a new conection using the just-unlinked
entry.

> +.TP
> +.B clash_resolve
> +Number of entry insert clashes resolved

Maybe mention that this is not a cause for alarm and mostly
expected with DNS these days?

> +.TP
> +.B chaintoolong
> +Number of oversized hash bucket encounters

Maybe mention that this happens only on insert and results in conntrack
to drop the packet.

Re: [conntrack-tools] conntrack.8: Document --stats counters

[Phil Sutter]

Hi Florian,

Finally trying to return to this open end.

On Thu, May 28, 2026 at 09:46:07AM +0200, Florian Westphal wrote:
> Phil Sutter <phil@nwl.cc> wrote:
> > Provide at least a brief description of each counter's meaning based on
> > code-analysis in kernel's nf_conntrack_core.c.
> > 
> > Signed-off-by: Phil Sutter <phil@nwl.cc>
> > ---
> > While most values are pretty obvious, I am not entirely sure I got the
> > insert_failed and drop reasons right.
> 
> Thanks for working on this.   I think the hardest part is to explain
> how to interpret this, i.e. which counters may indicate issues and which
> ones do not.

Thanks for your feedback so far. What I don't like is how we all seem to
reverse engineer meaning into the counter values. Maybe this is an
opportunity to clarify what these counters *should* represent and with
that in mind treat any deviation in the code as a bug which needs
fixing.

> > ---
> >  conntrack.8 | 37 ++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 36 insertions(+), 1 deletion(-)
> > 
> > diff --git a/conntrack.8 b/conntrack.8
> > index 2bfd80e5d6aa4..b562e16839a32 100644
> > --- a/conntrack.8
> > +++ b/conntrack.8
> > @@ -108,7 +108,42 @@ Flush the whole given table
> >  Show the table counter.
> >  .TP
> >  .BI "-S, --stats "
> > -Show the in-kernel connection tracking system statistics.
> > +Show the in-kernel connection tracking system statistics. The returned values
> > +for each CPU are:
> > +.RS
> > +.TP
> > +.B found
> > +Number of successful conntrack table lookups
> 
> We don't count those anymore, its too expensive.
> This is only incremented when a new connection cannot reuse
> the tuple and has to create a new nat mapping.
> 
> See nf_conntrack_tuple_taken().

Ah, I was obviously confused. Will put as "Number of collisions seen
while inserting a new conntrack entry." Hope that covers all cases.

> > +.B insert
> > +Number of entries inserted into conntrack table
> 
> IIRC its only incremented for insertions from netlink path.

Since nf_conntrack_hash_check_insert is called from
net/netfilter/nf_conntrack_bpf.c, too I'll put this as "Number of
entries manually inserted (via netlink or eBPF)."

> > +.B insert_failed
> > +Number of failed inserts (clash resolution failure, conntrack extensions in
> > +inconsistent state, entry in dying state, oversized hash bucket encountered)
> 
> oversized hash buckets should not be mentioned here, they have their own
> counter.  Should probably not increment both counters in kernel when it
> happens.

I see it incremented in three places:

1) nf_ct_resolve_clash
   - Increments 'drop' at the same time
   - This is the "clash resolution failure" case I mentioned
   - Happens either if nf_conntrack_l4proto does not set 'allow_clash' or
     both __nf_ct_resolve_clash and nf_ct_resolve_clash_harder fail

2) First spot in __nf_conntrack_confirm
   - Called "entry in dying state" by me above
   - AIUI, happens when confirming a conntrack entry (SYN-ACK?) for which
     nf_ct_is_dying() returns true

3) Second spot in __nf_conntrack_confirm
   - Increments 'chaintoolong' at the same time
   - Happens if oversized hash bucket is encountered(?)

> Maybe: "Number of NEW connections dropped because of clashes with existing entry."?

So this would be just #1 above. As a remedy, I would:

- Remove the 'drop' increment in #1, make it *the* insert_failed counter
  case
- Remove the 'insert_failed' increment from #3

But what about #2? Introduce a new counter? Increment 'drop' instead?

> > +.B drop
> > +Number of packets dropped (clash resolution failure, removed conntrack helper,
> > +stress, TCP connection aborted)
> 
> Yes, I think we need to fix this in the kernel and not increment two
> counters for the same reason.
> 
> Drop should mostly mean "incomplete packet / out of memory".

Hmm. Assuming we no longer increment it in nf_ct_resolve_clash, the
following incrementers remain:

1) nf_conntrack_in
  - If resolve_normal_ct returns error, which happens only if
    init_conntrack returns -ENOMEM

2) nf_conntrack_in
   - Also increments 'invalid' counter
   - If nf_conntrack_handle_packet returned 0 (= NF_DROP), which seems
     to be a special case with TCP (added in commit 6b69fe0c73c0
     ("netfilter: nf_conntrack_tcp: fix endless loop"))

3) nf_confirm_cthelper
   - If TCP sequence number adjustment fails, which seems to happen only
     on ENOMEM, missing seqadj CT extension or malformed TCP packet
     (having a tcpopt header with invalid optlen)

Maybe one should split #3 to increment 'invalid' upon invalid TCP option
header? What about #2? Should this increment 'invalid' at all? It does
not seem to indicate an invalid packet, but rather some internal
corner-case, right?

> > +.B early_drop
> > +Number of packets dropped up front due to full table
> 
> AFAICS its number of connections dropped because of a full table.

This is incremented by early_drop() only, called by __nf_conntrack_alloc
if a netns's conntrack table exceeds nf_conntrack_max. Am I getting this
right and early_drop() tries to drop existing entries to free up space
for the new one? I don't see any particular preference in
early_drop_list(), is this just "best effort"? (I guess the system is
under pressure and the goal is to resolve the situation as quickly as
possible?)

> > +.B search_restart
> > +Number of times a table lookup had to be restarted due to table reorg
> 
> Not sure about this one.  This is an implementation detail (SLAB_TYPESAFE_BY_RCU).
> This can happen at any time when entry is removed from table and
> other CPU is re-instantiating a new conection using the just-unlinked
> entry.

Ah, I see. Guess the comment in ____nf_conntrack_find misled me into
believing this was about hash table reorgs.

Maybe describe like this (which avoids too many internal details):

"Number of table lookups which had to be restarted. Due to optimized
storage reuse, a table lookup may occasionally find an already deleted
entry. This is detected and remedied by restarting the search."

> > +.TP
> > +.B clash_resolve
> > +Number of entry insert clashes resolved
> 
> Maybe mention that this is not a cause for alarm and mostly
> expected with DNS these days?

Sure!

> 
> > +.TP
> > +.B chaintoolong
> > +Number of oversized hash bucket encounters
> 
> Maybe mention that this happens only on insert and results in conntrack
> to drop the packet.

ACK.

Thanks, Phil

Re: [conntrack-tools] conntrack.8: Document --stats counters

[Florian Westphal]

Phil Sutter <phil@nwl.cc> wrote:
> Thanks for your feedback so far. What I don't like is how we all seem to
> reverse engineer meaning into the counter values. Maybe this is an
> opportunity to clarify what these counters *should* represent and with
> that in mind treat any deviation in the code as a bug which needs
> fixing.

Agreed.

> Since nf_conntrack_hash_check_insert is called from
> net/netfilter/nf_conntrack_bpf.c, too I'll put this as "Number of
> entries manually inserted (via netlink or eBPF)."

Oh, did not realize this was also called via ebpf.

> > oversized hash buckets should not be mentioned here, they have their own
> > counter.  Should probably not increment both counters in kernel when it
> > happens.
> 
> I see it incremented in three places:
> 
> 1) nf_ct_resolve_clash
>    - Increments 'drop' at the same time
>    - This is the "clash resolution failure" case I mentioned
>    - Happens either if nf_conntrack_l4proto does not set 'allow_clash' or
>      both __nf_ct_resolve_clash and nf_ct_resolve_clash_harder fail
> 
> 2) First spot in __nf_conntrack_confirm
>    - Called "entry in dying state" by me above
>    - AIUI, happens when confirming a conntrack entry (SYN-ACK?) for which
>      nf_ct_is_dying() returns true

SYN, we only ever confirm the NEW / first packet of a flow.

> 3) Second spot in __nf_conntrack_confirm
>    - Increments 'chaintoolong' at the same time
>    - Happens if oversized hash bucket is encountered(?)

Yes.

> > Maybe: "Number of NEW connections dropped because of clashes with existing entry."?
> So this would be just #1 above. As a remedy, I would:
> 
> - Remove the 'drop' increment in #1, make it *the* insert_failed counter
>   case
> - Remove the 'insert_failed' increment from #3

No objections.

> But what about #2? Introduce a new counter? Increment 'drop' instead?

I'm not sure this can ever be true anymore (dying on unconfirmed).
I would leave it as-is.  In both nf_ct_resolve_clash and this case there
is nothing the user could do.  At least I can't think of anything.

> > Yes, I think we need to fix this in the kernel and not increment two
> > counters for the same reason.
> > 
> > Drop should mostly mean "incomplete packet / out of memory".
> 
> Hmm. Assuming we no longer increment it in nf_ct_resolve_clash, the
> following incrementers remain:
> 
> 1) nf_conntrack_in
>   - If resolve_normal_ct returns error, which happens only if
>     init_conntrack returns -ENOMEM
> 
> 2) nf_conntrack_in
>    - Also increments 'invalid' counter
>    - If nf_conntrack_handle_packet returned 0 (= NF_DROP), which seems
>      to be a special case with TCP (added in commit 6b69fe0c73c0
>      ("netfilter: nf_conntrack_tcp: fix endless loop"))
> 
> 3) nf_confirm_cthelper
>    - If TCP sequence number adjustment fails, which seems to happen only
>      on ENOMEM, missing seqadj CT extension or malformed TCP packet
>      (having a tcpopt header with invalid optlen)
> 
> Maybe one should split #3 to increment 'invalid' upon invalid TCP option
> header?

Good idea.

> What about #2? Should this increment 'invalid' at all? It does
> not seem to indicate an invalid packet, but rather some internal
> corner-case, right?

Yes, I think its best to remove the invalid increment here.

> > > +.B early_drop
> > > +Number of packets dropped up front due to full table
> > 
> > AFAICS its number of connections dropped because of a full table.
> 
> This is incremented by early_drop() only, called by __nf_conntrack_alloc
> if a netns's conntrack table exceeds nf_conntrack_max. Am I getting this
> right and early_drop() tries to drop existing entries to free up space
> for the new one? I don't see any particular preference in
> early_drop_list(), is this just "best effort"? (I guess the system is
> under pressure and the goal is to resolve the situation as quickly as
> possible?)

Yes, it will attempt to remove another competing entry (not confirmed,
meaning short-lived UDP or TCP that did not progress past 3whs yet).

> > > +.B search_restart
> > > +Number of times a table lookup had to be restarted due to table reorg
> > 
> > Not sure about this one.  This is an implementation detail (SLAB_TYPESAFE_BY_RCU).
> > This can happen at any time when entry is removed from table and
> > other CPU is re-instantiating a new conection using the just-unlinked
> > entry.
> 
> Ah, I see. Guess the comment in ____nf_conntrack_find misled me into
> believing this was about hash table reorgs.
> 
> Maybe describe like this (which avoids too many internal details):
> 
> "Number of table lookups which had to be restarted. Due to optimized
> storage reuse, a table lookup may occasionally find an already deleted
> entry. This is detected and remedied by restarting the search."

Maybe even more terse, e.g.

"Number of table lookups which had to be restarted.  In rare cases a
lookup may encounter an already deleted entry which causes a
search restart."

Up to you, this is hard to explain without the implementation details,
the question would be "what does that counter tell me" and "what should
I do about this".  And I have no answer.  I thought about removing this
counter, but then this would allow to detect programming bugs (chain end
tag corrupted) for example.

Thanks for working on this!