[BUG] netfilter: unbounded nftables reject recursion on inet OUTPUT

[Yu Junzhe <junzheyu1@gmail.com>]

Hello,

I am reporting unbounded hook recursion on inet OUTPUT: an nftables
output chain that does a TCP flags payload-set followed by
"reject with tcp reset" makes nf_send_reset() re-enter the same
NF_INET_LOCAL_OUT hook until the kernel stack is exhausted.

Summary
=======

nft_reject_inet_eval() calls nf_send_reset(), which builds a RST skb
and transmits it with ip_local_out(). That walks the same OUTPUT
hooks again.

The only loop guard in nf_reject_ip_tcphdr_get() is the packet-content
check:

  /* No RST for RST. */
  if (oth->rst)
          return NULL;

If a payload-set runs first and rewrites the TCP flags byte to SYN,
that guard never sees RST, so another RST is generated. The cycle is:

  nft_do_chain
    -> nft_reject_inet_eval
      -> nf_send_reset
        -> ip_local_out
          -> nf_hook_slow
            -> nft_do_chain

nft_reject_inet_validate() allows NF_INET_LOCAL_OUT.

This is a legal nftables configuration (payload set + reject with tcp
reset on an output base chain), including with CAP_NET_ADMIN in a
network namespace. I do not claim privilege escalation; the result we
see is local DoS (stack overflow, then panic).

I am aware of the existing recursion bound on the netdev fwd/neigh
xmit path. nf_send_reset() via ip_local_out() does not use it.

Affected
========

- Confirmed on Linux 6.6.144, KASAN guest (CONFIG_VMAP_STACK not set).
  On that config the overflowing SP walks into adjacent slab; KASAN
  first reports a write in __unwind_start while tracking __alloc_skb
  from the next RST. Those frames are collateral. The defect is
  unbounded hook recursion.
- Source review of torvalds/linux master as of 2026-09-21:
  nf_send_reset() still calls ip_local_out(); the only loop guard is
  still if (oth->rst); nft_reject_inet_validate() still allows
  NF_INET_LOCAL_OUT. The PoC was not re-run on tip.
- Files: net/ipv4/netfilter/nf_reject_ipv4.c,
  net/netfilter/nft_reject_inet.c
- Config: CONFIG_NF_TABLES=y, CONFIG_NF_TABLES_INET=y,
  CONFIG_NFT_REJECT_INET=y (KASAN used for the report)

Ruleset that triggers it
========================

  table inet t {
          chain out {
                  type filter hook output priority 0;
                  tcp flags set syn
                  reject with tcp reset
          }
  }

Then connect() to 127.0.0.1 in that netns (a SYN on lo).

Crash excerpt (minimized PoC, Linux 6.6.144 KASAN)
==================================================

BUG: KASAN: use-after-free in __unwind_start+0x36/0x840
Write of size 112 at addr ff1100000659ff88 by task repro/221
CPU: 1 PID: 221 Comm: repro Not tainted 6.6.144 #1
Call Trace:
  __alloc_skb
  nf_send_reset
  nft_reject_inet_eval
  nft_do_chain
  nft_do_chain_inet
  nf_hook_slow
  __ip_local_out
  ip_local_out
  nf_send_reset
  nft_reject_inet_eval
  nft_do_chain
  ... (repeats until stack exhaustion; then GPF / panic)

Reproducer
==========

A self-contained C PoC (raw NETLINK_NETFILTER batch, no nft binary)
and a Docker/QEMU package are available on request. VM-only: the
program overflows the guest kernel stack. Please do not run it on a
host kernel.

I am happy to test patches or send the reproducer.

Thanks,
Yu Junzhe
FuzzAnything <fuzzanything@gmail.com>