[PATCH net-next v1 00/11] net: flow_dissector: opt-in byte-identical fast paths for common shapes

[PATCH net-next v1 00/11] net: flow_dissector: opt-in byte-identical fast paths for common shapes

[Dave Seddon]

G'day,

__skb_flow_dissect() parses packet headers with a generic protocol-graph
parser. The graph walk is what makes the dissector so flexible, but most
machines spend that flexibility on the same traffic all day long: plain
eth + IPv4/IPv6 + TCP/UDP, perhaps with a VLAN tag or a tunnel header in
front. The intuition behind this series is that the common shapes can be
dissected by straight-line code instead, without giving up the graph
walk for everything else.

Concretely: the series adds an opt-in, static-branch-gated fast path per
packet shape, covering the common cases and the IP-in-IP/GRE
encapsulations. Each shape has its own gate under
/proc/sys/net/flow_dissector/, every gate is disabled by default, and
when a gate is off its added cost is one not-taken branch per dissect.
The gates are manual in this series; a follow-up RFC will propose an
optional auto mode that flips them from the measured traffic mix (via
the patch-8 counters), so nobody has to hand-tune seven knobs
(eth_ip, vlan, qinq, pppoe, mpls, ipip, gre). Nothing
here depends on it -- it is where this is heading, not a prerequisite.

The series is byte-identical by contract: for any packet, the fast path
either writes exactly the flow_keys bytes the slow path would have
written, or it returns false and the slow path runs. That contract is
enforced in-tree by a KUnit equivalence suite (patch 10).

Patch 1 stands slightly apart: it is a small win for the existing path --
today every dissect takes rcu_read_lock() and does two dependent
rcu_dereference() loads to look for a netns BPF flow dissector program
even when none is attached anywhere; a static key (mirroring
bpf_sk_lookup_enabled) skips all of it in the common no-program case.

Patches 2-7 are the byte-identical common shapes (eth+IPv4/IPv6 x
TCP/UDP, VLAN/QinQ, PPPoE, single MPLS, IP-in-IP, plain GRE). Patch 8
adds per-shape counters (/proc/net/flow_dissector_stats); patch 9 bounds
the fast-path tunnel recursion at MAX_FLOW_DISSECT_HDRS.

A companion series (posted separately) builds on this one to add opt-in
inner-flow descent for the UDP tunnels (VXLAN, Geneve, GTP-U, FOU/GUE)
-- that one changes hashing when enabled, so it is a separate policy
discussion; this series is purely a transparent optimisation.

Why not just optimise the existing loop
========================================
The slow path is a generic graph walk by design: one loop, one
proto/nhoff state machine, re-entered per header, able to express every
protocol relationship the dissector knows. That generality is the cost --
per-header dispatch, state saves, and branchy key-writing that the
compiler cannot flatten because the shape is only known at runtime. A
per-shape straight-line function knows the whole shape at compile time,
so the loads, checks and key writes schedule as one block. The two
approaches are complementary: the loop stays the single source of
semantic truth for every shape, and a handful of high-volume shapes get
a verified shortcut.

Performance summary
===================
Measured across 3 ISAs and 8 microarchitectures in total, with
byte-identical verification per shape (userspace A/B harness compiling
the dissector both ways from the same source). Two instruments, each
run on 7 of the 8 microarchitectures: "allshapes" compiles every fast
path in (the realistic icache footprint) and measures each shape in
isolation, on x86 Zen1/Zen2/Skylake, ARM Cortex-A53/A72/A76 and RISC-V
X60; "isolated A/B" compiles one shape's path against the unmodified
dissector (the upper bound for that shape), on the same set with
Haswell in place of Zen 1.

Reduction in dissector CPU cost, allshapes microbench. Each row is the
range across its 7 microarchitectures, worst to best:

    shape     worst    best
    ------   ------   ------
    eth_ip    -4.7%   -31.6%
    vlan     -18.1%   -38.5%
    qinq     -16.1%   -39.8%
    pppoe     -8.6%   -28.0%
    mpls      -6.7%   -15.0%
    ipip     -36.8%   -47.0%   largest: slow path re-parses inner IP
    gre        (byte-identical descent family, tracks ipip; verified
                by the KUnit suite rather than separately benchmarked)

Isolated A/B, eth + IPv4 + TCP (same source compiled two ways),
nanoseconds per packet, lower is better:

    microarchitecture                      slow    fast   delta
    ------------------------------------  ------  ------  -----
    x86 Zen 2 (Threadripper PRO 3945WX)    12.4     6.6    -47%
    x86 Skylake (Core i9-10885H)           10.6     5.6    -47%
    x86 Haswell (Celeron 2955U)            35.9    18.0    -50%
    ARM Cortex-A53                        119.0    61.3    -48%
    ARM Cortex-A72                         42.4    19.1    -55%
    ARM Cortex-A76                         19.4     9.3    -52%
    RISC-V SpacemiT X60                    98.2    49.3    -50%

The isolated dissect costs 47-55% less per packet, consistent across
the three ISAs and the seven microarchitectures measured with that
instrument; the relative reduction is largest on in-order cores,
indicating the saving is fewer instructions rather than a
microarchitectural artifact. IPv6 uses the same gate and is
byte-identical (x86 -39%, RISC-V -25%).

flow_keys -> skb->hash via skb_get_hash() is the flow identity behind
RPS/RFS/aRFS, the fq / fq_codel / cake fairness queues, bonding/LAG
transmit hashing and more; those callers pass no restricting flags, so
each of them gets the per-dissect saving at once. Dissector callers
that pass STOP flags (ECMP multipath, xfrm policy lookup) or run their
own dissector instances (tc-flower) deliberately stay on the slow
path -- unchanged, not regressed.

Counters (patch 8) are the one part with a cost in the all-gates-off
default: one this_cpu_inc per dissect plus one per matched shape.
Measured on a CPU-bound pktgen soak that is +0.74% dissector time
(+0.44 sigma -- within run-to-run noise). The gates themselves stay
free: off means one not-taken static branch.

Relationship to the BPF flow dissector
======================================
The fast path is invoked strictly after the netns BPF flow dissector
hook in __skb_flow_dissect(): if a program is attached and returns any
verdict other than BPF_FLOW_DISSECTOR_CONTINUE, the function returns
before flow_dissect_fast() is reached. Attached BPF dissectors therefore
always take precedence, and a system running one sees bit-for-bit
unchanged behaviour from this series (patch 1 additionally makes its
program lookup cheaper for everyone else). Dissects fully handled by a
BPF program are deliberately not counted in
/proc/net/flow_dissector_stats (patch 8).

The gates are global (a static key patches code shared by every netns);
per-netns dissector policy already has a mechanism -- the netns BPF
flow dissector -- and it keeps full precedence here.

Maintenance burden: why two paths won't diverge
===============================================
The slow path remains the single source of semantic truth and the fast
path is verified against it, not vice versa. Three layers enforce that:

  1. The eligibility gate: only the two standard dissectors and a flag
     subset ever enter flow_dissect_fast(); custom tc-flower dissector
     instances are structurally unaffected.
  2. Every shape is a default-off static branch: a kernel with gates off
     executes the not-taken-branch slow path only.
  3. The KUnit equivalence suite (patch 10) dissects a corpus of
     eligible shapes, deliberate fast-path misses, truncations at every
     byte boundary, and skb-mode entry conditions through both paths
     and memcmp()s the resulting struct flow_keys (and their
     flow_hash_from_keys()). Future divergence is a failing test, not a
     silent behaviour change.

The test earns its place: during development it caught real divergences
the out-of-tree A/B harness had missed -- the MPLS fast path skipping the
slow path's out_good terminal writes, the 4in6 descent not reproducing
the outer-IPv6 residue, and -- caught by a hardware boot smoke with
counter positive-controls, then pinned by a regression test -- the
dispatcher rejecting the STOP_AT_FLOW_LABEL flag that skb_get_hash()
passes on every dissect, which had made the fast path unreachable from
RPS/RFS/fq/cake entirely. All fixed in this posting.

Robustness and safety limits
============================
The fast path descends into IP-in-IP and GRE by recursion. The slow
path bounds header descents at MAX_FLOW_DISSECT_HDRS; patch 9 gives the
fast path the same bound, so a crafted nested chain falls back to the
slow path at the same depth rather than driving one C stack frame per
~20 bytes of linear header. The KUnit suite covers the bound boundary,
the non-linear-skb fallback, a gates-off negative (the fast path must
never run when disabled, observed via the counters), and a seeded
fuzzer as a general guard.


Patch 11 adds Documentation/networking/flow_dissector.rst -- there was no
general flow_dissector overview in the tree. It explains what the dissector
is and who consumes its output, then documents this series' fast paths: the
byte-identical contract and the break-even model for when a gate is worth
enabling. The per-knob reference stays in
Documentation/admin-guide/sysctl/net.rst; the new doc links to it.

On tooling: this series was developed with substantial LLM assistance
(Claude, Anthropic) -- identifying the optimisation opportunity,
drafting the patches, and building and running the test matrix. Every
patch carries an Assisted-by: trailer in the
Documentation/process/coding-assistants.rst format. All of it was
human-reviewed and human-measured: the performance numbers come from
months of runs across the hardware above, and the byte-identical
contract is enforced mechanically by the in-tree KUnit suite rather
than by trust in either the tool or the author.

This series grew out of work on Tom Herbert's XDP2 parser project:
porting the kernel flow dissector into XDP2's userspace benchmarking
harness, to compare parsers on equal footing, is what made the
per-shape cost of the generic loop visible -- this opt-in fast path is
the kernel-side result of that analysis.

Testing: 3 ISAs / 8 uarches (x86 Zen1/Zen2/Skylake/Haswell, ARM
Cortex-A53/A72/A76, RISC-V X60), in-tree KUnit (patch 10: 55 tests,
also run under KASAN+UBSAN; per-patch W=1, sparse and smatch clean;
CONFIG_NET_FOU n/m/y config matrix for the companion series) and
out-of-tree A/B. Supporting data (flow-distribution study,
per-shape/per-arch matrix, break-even derivation) lives in the
author's XDP2 fork: https://github.com/randomizedcoder/xdp2 , under
perf-results/ .

Dave Seddon (11):
  net: flow_dissector: gate BPF program lookup behind a static key
  net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}
  net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP
  net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} +
    TCP/UDP
  net: flow_dissector: add fast-path for single MPLS label + IP
  net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 /
    6in4)
  net: flow_dissector: add byte-identical fast-path for plain GRE inner
  net: flow_dissector: per-shape counters +
    /proc/net/flow_dissector_stats
  net: flow_dissector: bound fast-path tunnel recursion
  net: flow_dissector: add KUnit fast/slow path equivalence tests
  Documentation: networking: add flow_dissector overview and fast-path
    guide

 Documentation/admin-guide/sysctl/net.rst    |  178 ++-
 Documentation/bpf/index.rst                 |    1 +
 Documentation/bpf/prog_flow_dissector.rst   |    2 +
 Documentation/networking/flow_dissector.rst |  133 +++
 Documentation/networking/index.rst          |    1 +
 include/linux/skbuff.h                      |    1 +
 include/net/flow_dissector.h                |   34 +
 kernel/bpf/net_namespace.c                  |   17 +-
 net/Kconfig                                 |   12 +
 net/core/Makefile                           |    1 +
 net/core/flow_dissector.c                   | 1116 ++++++++++++++++++-
 net/core/flow_dissector_test.c              | 1053 +++++++++++++++++
 12 files changed, 2500 insertions(+), 49 deletions(-)
 create mode 100644 Documentation/networking/flow_dissector.rst
 create mode 100644 net/core/flow_dissector_test.c


base-commit: f6f3b36c15ed44de1fbb44e645e4fae8c4a4453e
-- 
2.54.0

[PATCH net-next v1 01/11] net: flow_dissector: gate BPF program lookup behind a static key

[Dave Seddon]

__skb_flow_dissect() takes rcu_read_lock(), resolves the netns from
skb->dev/skb->sk, and does two dependent rcu_dereference() loads of
the netns-BPF run_array (init_net's first - a shared cacheline - then
the packet netns') on every dissect, even though the overwhelmingly
common case is that no BPF flow dissector program is attached
anywhere on the system.

Add a static key that counts attached flow dissector programs and
skip the whole lookup block when it is zero, exactly as
bpf_sk_lookup_enabled does for the sibling NETNS_BPF_SK_LOOKUP attach
type (also see cgroup_bpf_enabled_key[] for the same pattern per
cgroup-BPF attach type).

The sk_lookup type is link-only, so its inc/dec sit only in
netns_bpf_attach_type_need()/unneed(), which the link attach/release
paths already call. The flow dissector type also supports legacy
BPF_PROG_ATTACH, so additionally inc on a fresh (non-replacement)
prog attach, dec in __netns_bpf_prog_detach(), and dec for a
remaining legacy prog in netns_bpf_pernet_pre_exit() (links are
already handled by the existing per-link unneed loop there). All
these paths run under netns_bpf_mutex.

As with sk_lookup, there is a brief window where a just-attached
program can miss a concurrent dissect (key flips after
rcu_assign_pointer) or a just-detached one can cause a wasted lookup;
both are benign and match the existing semantics of the sibling keys.

DEBUG_NET_WARN_ON_ONCE(!net) moves inside the gated block: it exists
to catch a missing netns for the BPF hook, which is only meaningful
when a program can actually be attached.

With the key disabled this removes the rcu_read_lock/unlock pair, the
netns resolution, and both run_array loads from the per-packet path.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 include/linux/skbuff.h     |   1 +
 kernel/bpf/net_namespace.c |  17 +++++-
 net/core/flow_dissector.c  | 104 +++++++++++++++++++++----------------
 3 files changed, 75 insertions(+), 47 deletions(-)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 22eda1d54a0e..6448c59dc807 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1588,6 +1588,7 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector,
 			     unsigned int key_count);
 
 struct bpf_flow_dissector;
+extern struct static_key_false netns_bpf_flow_dissector_enabled;
 u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
 		     __be16 proto, int nhoff, int hlen, unsigned int flags);
 
diff --git a/kernel/bpf/net_namespace.c b/kernel/bpf/net_namespace.c
index 25f30f9edaef..0d48ea65697e 100644
--- a/kernel/bpf/net_namespace.c
+++ b/kernel/bpf/net_namespace.c
@@ -28,6 +28,9 @@ DEFINE_MUTEX(netns_bpf_mutex);
 static void netns_bpf_attach_type_unneed(enum netns_bpf_attach_type type)
 {
 	switch (type) {
+	case NETNS_BPF_FLOW_DISSECTOR:
+		static_branch_dec(&netns_bpf_flow_dissector_enabled);
+		break;
 #ifdef CONFIG_INET
 	case NETNS_BPF_SK_LOOKUP:
 		static_branch_dec(&bpf_sk_lookup_enabled);
@@ -41,6 +44,9 @@ static void netns_bpf_attach_type_unneed(enum netns_bpf_attach_type type)
 static void netns_bpf_attach_type_need(enum netns_bpf_attach_type type)
 {
 	switch (type) {
+	case NETNS_BPF_FLOW_DISSECTOR:
+		static_branch_inc(&netns_bpf_flow_dissector_enabled);
+		break;
 #ifdef CONFIG_INET
 	case NETNS_BPF_SK_LOOKUP:
 		static_branch_inc(&bpf_sk_lookup_enabled);
@@ -352,6 +358,11 @@ int netns_bpf_prog_attach(const union bpf_attr *attr, struct bpf_prog *prog)
 	net->bpf.progs[type] = prog;
 	if (attached)
 		bpf_prog_put(attached);
+	else
+		/* Mark attach point as used on a fresh attach; a
+		 * replacement keeps the existing count.
+		 */
+		netns_bpf_attach_type_need(type);
 
 out_unlock:
 	mutex_unlock(&netns_bpf_mutex);
@@ -373,6 +384,8 @@ static int __netns_bpf_prog_detach(struct net *net,
 	attached = net->bpf.progs[type];
 	if (!attached || attached != old)
 		return -ENOENT;
+	/* Mark attach point as unused */
+	netns_bpf_attach_type_unneed(type);
 	netns_bpf_run_array_detach(net, type);
 	net->bpf.progs[type] = NULL;
 	bpf_prog_put(attached);
@@ -546,8 +559,10 @@ static void __net_exit netns_bpf_pernet_pre_exit(struct net *net)
 			net_link->net = NULL; /* auto-detach link */
 			netns_bpf_attach_type_unneed(type);
 		}
-		if (net->bpf.progs[type])
+		if (net->bpf.progs[type]) {
+			netns_bpf_attach_type_unneed(type);
 			bpf_prog_put(net->bpf.progs[type]);
+		}
 	}
 	mutex_unlock(&netns_bpf_mutex);
 }
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 8aa4f9b4df81..b8c59209b460 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -73,6 +73,14 @@ void skb_flow_dissector_init(struct flow_dissector *flow_dissector,
 }
 EXPORT_SYMBOL(skb_flow_dissector_init);
 
+/* Enabled while any netns has a BPF flow dissector program attached;
+ * lets __skb_flow_dissect() skip the program lookup entirely in the
+ * common no-program case. Maintained by the netns-BPF attach/detach
+ * paths in kernel/bpf/net_namespace.c.
+ */
+DEFINE_STATIC_KEY_FALSE(netns_bpf_flow_dissector_enabled);
+EXPORT_SYMBOL(netns_bpf_flow_dissector_enabled);
+
 #ifdef CONFIG_BPF_SYSCALL
 int flow_dissector_bpf_prog_attach_check(struct net *net,
 					 struct bpf_prog *prog)
@@ -1117,59 +1125,63 @@ bool __skb_flow_dissect(const struct net *net,
 					      FLOW_DISSECTOR_KEY_BASIC,
 					      target_container);
 
-	rcu_read_lock();
+	if (static_branch_unlikely(&netns_bpf_flow_dissector_enabled)) {
+		rcu_read_lock();
 
-	if (skb) {
-		if (!net) {
-			if (skb->dev)
-				net = dev_net_rcu(skb->dev);
-			else if (skb->sk)
-				net = sock_net(skb->sk);
+		if (skb) {
+			if (!net) {
+				if (skb->dev)
+					net = dev_net_rcu(skb->dev);
+				else if (skb->sk)
+					net = sock_net(skb->sk);
+			}
 		}
-	}
 
-	DEBUG_NET_WARN_ON_ONCE(!net);
-	if (net) {
-		enum netns_bpf_attach_type type = NETNS_BPF_FLOW_DISSECTOR;
-		struct bpf_prog_array *run_array;
-
-		run_array = rcu_dereference(init_net.bpf.run_array[type]);
-		if (!run_array)
-			run_array = rcu_dereference(net->bpf.run_array[type]);
-
-		if (run_array) {
-			struct bpf_flow_keys flow_keys;
-			struct bpf_flow_dissector ctx = {
-				.flow_keys = &flow_keys,
-				.data = data,
-				.data_end = data + hlen,
-			};
-			__be16 n_proto = proto;
-			struct bpf_prog *prog;
-			u32 result;
-
-			if (skb) {
-				ctx.skb = skb;
-				/* we can't use 'proto' in the skb case
-				 * because it might be set to skb->vlan_proto
-				 * which has been pulled from the data
-				 */
-				n_proto = skb->protocol;
-			}
+		DEBUG_NET_WARN_ON_ONCE(!net);
+		if (net) {
+			enum netns_bpf_attach_type type = NETNS_BPF_FLOW_DISSECTOR;
+			struct bpf_prog_array *run_array;
+
+			run_array = rcu_dereference(init_net.bpf.run_array[type]);
+			if (!run_array)
+				run_array = rcu_dereference(net->bpf.run_array[type]);
+
+			if (run_array) {
+				struct bpf_flow_keys flow_keys;
+				struct bpf_flow_dissector ctx = {
+					.flow_keys = &flow_keys,
+					.data = data,
+					.data_end = data + hlen,
+				};
+				__be16 n_proto = proto;
+				struct bpf_prog *prog;
+				u32 result;
+
+				if (skb) {
+					ctx.skb = skb;
+					/* we can't use 'proto' in the skb case
+					 * because it might be set to
+					 * skb->vlan_proto which has been pulled
+					 * from the data
+					 */
+					n_proto = skb->protocol;
+				}
 
-			prog = READ_ONCE(run_array->items[0].prog);
-			result = bpf_flow_dissect(prog, &ctx, n_proto, nhoff,
-						  hlen, flags);
-			if (result != BPF_FLOW_DISSECTOR_CONTINUE) {
-				__skb_flow_bpf_to_target(&flow_keys, flow_dissector,
-							 target_container);
-				rcu_read_unlock();
-				return result == BPF_OK;
+				prog = READ_ONCE(run_array->items[0].prog);
+				result = bpf_flow_dissect(prog, &ctx, n_proto,
+							  nhoff, hlen, flags);
+				if (result != BPF_FLOW_DISSECTOR_CONTINUE) {
+					__skb_flow_bpf_to_target(&flow_keys,
+								 flow_dissector,
+								 target_container);
+					rcu_read_unlock();
+					return result == BPF_OK;
+				}
 			}
 		}
-	}
 
-	rcu_read_unlock();
+		rcu_read_unlock();
+	}
 
 	if (dissector_uses_key(flow_dissector,
 			       FLOW_DISSECTOR_KEY_ETH_ADDRS)) {
-- 
2.54.0

[PATCH net-next v1 02/11] net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}

[Dave Seddon]

A small targeted optimisation to __skb_flow_dissect(). A new
flow_dissect_fast() dispatcher routes the standard flow_keys dissector
calls through per-shape straight-line extractors for the common
eth + IPv4/IPv6 + TCP/UDP packet shapes. On any miss, falls back to the
existing graph walk. Output is byte-identical to the slow path for the
shapes handled.

This is the parent of the fast-path series. Subsequent patches add
per-shape extensions for single VLAN, QinQ, PPPoE, MPLS, IP-in-IP and
GRE. Each patch ships its own per-shape sysctl in the new
``/proc/sys/net/flow_dissector/`` subtree; this patch establishes the
subtree and registers the first entry, ``eth_ip``.

The fast path is invoked strictly after the netns BPF flow dissector
hook: if a BPF program is attached and returns a verdict other than
BPF_FLOW_DISSECTOR_CONTINUE, __skb_flow_dissect() returns before
flow_dissect_fast() is reached. Attached BPF dissectors therefore
always take precedence and observe no behaviour change from this
series.

  - What this does. Adds an opt-in fast-path for eth + IPv4 (IHL=5,
    not fragmented) + TCP/UDP and eth + IPv6 (no extension headers) +
    TCP/UDP. When the per-shape sysctl is enabled and a packet
    matches, the helper fills struct flow_keys byte-identically to the
    slow path and returns. On any miss it defers to the existing graph
    walk. Only the two standard dissectors are eligible
    (flow_keys_dissector and flow_keys_dissector_symmetric); custom
    dissectors and slow-path-only flags defer unconditionally.

  - How it gates. A new
    DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key), externed in
    <net/flow_dissector.h>, toggled by
    /proc/sys/net/flow_dissector/eth_ip with proc_do_static_key. When
    the sysctl is 0 the dispatcher's per-shape case is a forward not-
    taken JMP — same slow-path cost as today. When 1, the helper runs
    inline.

  - Namespace. The sysctl subtree is /proc/sys/net/flow_dissector/
    (registered by a new late_initcall in net/core/flow_dissector.c
    via register_net_sysctl(&init_net, "net/flow_dissector", tbl) —
    same shape as net/mpls, net/bridge).

  - Forward declarations. flow_dissect_fast_ipv4 / _ipv6 are forward-
    declared at the top of the file (after includes). Subsequent
    patches add flow_dissect_fast_vlan etc. that call these helpers;
    anchoring the declarations at top of file makes the patch insertion
    order robust against kernel-version line-number drift.

  - Cost model. On a miss, dispatcher overhead is one ethertype switch
    (~5 cycles) plus the per-case static_branch test (~not-taken JMP
    when 0). On a hit, the straight-line body saves ~30 ns/skb of
    graph-walk cost on the common shape (Zen 2 / Cortex-A76; see the
    cover letter for the cross-uarch microbench table).

  - Output contract. Byte-identical with the slow path for the
    eligible shape. The IPv6 fast-path defers on any non-zero flow
    label: callers passing FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL --
    skb_get_hash(), i.e. RPS/RFS, fq, fq_codel, cake -- make the slow
    path stop at a non-zero label, and dissectors requesting the label
    key write it; deferring on the label itself keeps byte-identity
    for every combination without threading flags into the helpers,
    and zero-label IPv6 stays on the fast path. Fragmented IPv4
    packets defer. Custom dissectors defer, as do FLOW_DISSECTOR_F_*
    flags other than PARSE_1ST_FRAG and STOP_AT_FLOW_LABEL. Admitting
    STOP_AT_FLOW_LABEL is what makes the fast path reachable from
    skb_get_hash() at all -- the kernel's main dissection consumer.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst |  27 ++-
 include/net/flow_dissector.h             |   5 +
 net/core/flow_dissector.c                | 273 ++++++++++++++++++++++-
 3 files changed, 302 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index e586e17fc7a5..aac82169e019 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -460,7 +460,32 @@ historical importance.
 
 Default: 0
 
-2. /proc/sys/net/unix - Parameters for Unix domain sockets
+2. /proc/sys/net/flow_dissector - Flow dissector fast-path per-shape sysctls
+---------------------------------------------------------------------------
+
+The flow dissector is invoked on every packet that needs a flow hash
+(RPS, RFS, ECMP, sch_cake, cls_flower, ...). For common shapes
+(Eth+IPv4/IPv6+TCP/UDP, plus per-shape extensions for VLAN/QinQ,
+PPPoE, MPLS, IP-in-IP and GRE added by this series), an opt-in
+fast-path bypasses the slow-path graph walk and writes
+``struct flow_keys`` directly. Each shape is gated by its own static
+key + sysctl so operators enable only what their deployment uses.
+
+All defaults are 0 (off). Flipping any of these on takes effect
+immediately; no kernel rebuild needed.
+
+eth_ip
+~~~~~~
+
+Eth + IPv4 (IHL=5, no fragmentation) + TCP/UDP, and Eth + IPv6 (no
+extension headers) + TCP/UDP. Output is byte-identical to the slow
+path on the eligible shape; non-matching packets fall through to the
+slow path unchanged. The dispatcher costs one not-taken JMP per call
+when this is 0.
+
+Default: 0
+
+3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
 There is only one file in this directory.
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index ced79dc8e856..7d513a6cbb55 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -425,6 +425,11 @@ __be32 flow_get_u32_dst(const struct flow_keys *flow);
 extern struct flow_dissector flow_keys_dissector;
 extern struct flow_dissector flow_keys_basic_dissector;
 
+/* Per-shape fast-path static-branch keys, toggled via
+ * /proc/sys/net/flow_dissector/<shape>. All default off.
+ */
+extern struct static_key_false flow_dissector_eth_ip_key;
+
 /* struct flow_keys_digest:
  *
  * This structure is used to hold a digest of the full flow keys. This is a
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index b8c59209b460..88b1bd1cc4ca 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -37,6 +37,33 @@
 #endif
 #include <linux/bpf-netns.h>
 
+/* Per-shape fast-path gates, one static_branch per shape so operators
+ * enable only what their deployment carries. All default off.
+ */
+DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key);
+EXPORT_SYMBOL(flow_dissector_eth_ip_key);
+
+/* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
+#define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
+
+/* Fast-path helper forward declarations. */
+static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen);
+static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen);
+
+/* One of the two dissectors the fast-path eligibility check admits;
+ * defined here so flow_dissect_fast() below can reference it (its keys
+ * and init live near the bottom of the file).
+ */
+static struct flow_dissector flow_keys_dissector_symmetric __read_mostly;
+
 static void dissector_set_key(struct flow_dissector *flow_dissector,
 			      enum flow_dissector_key_id key_id)
 {
@@ -1043,6 +1070,223 @@ static bool is_pppoe_ses_hdr_valid(const struct pppoe_hdr *hdr)
 	return hdr->ver == 1 && hdr->type == 1 && hdr->code == 0;
 }
 
+/* The IPv4 fast-path assumes a 20-byte IPv4 header (IHL == 5). The
+ * runtime IHL check below is the dynamic invariant; this static
+ * assertion is the compile-time one, in the unlikely case struct
+ * iphdr ever grows.
+ */
+static_assert(sizeof(struct iphdr) == 20);
+
+/* Straight-line eth + IPv4 (IHL=5, not fragmented) + TCP|UDP. Returns
+ * true when @target_container has been filled byte-identically to the
+ * slow path; false on any miss (caller falls back to the slow path).
+ */
+static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	struct flow_dissector_key_addrs *key_addrs;
+	struct flow_dissector_key_basic *key_basic;
+	struct flow_dissector_key_ports *key_ports;
+	const struct iphdr *iph;
+	int thoff;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4))
+		return false;
+
+	iph = (const struct iphdr *)((const u8 *)data + nhoff);
+
+	if (unlikely(*(const u8 *)iph != FLOW_DIS_IPV4_VIHL_NOOPT))
+		return false;
+
+	if (unlikely(iph->frag_off & htons(IP_MF | IP_OFFSET)))
+		return false;
+
+	if (unlikely(iph->protocol != IPPROTO_TCP &&
+		     iph->protocol != IPPROTO_UDP))
+		return false;
+
+	thoff = nhoff + (int)sizeof(*iph);
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+		key_control->thoff = min_t(u16, thoff,
+					   skb ? skb->len : hlen);
+		key_control->flags = 0;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_BASIC)) {
+		key_basic = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_BASIC,
+						      target_container);
+		key_basic->n_proto = htons(ETH_P_IP);
+		key_basic->ip_proto = iph->protocol;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_IPV4_ADDRS)) {
+		key_addrs = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_IPV4_ADDRS,
+						      target_container);
+		memcpy(&key_addrs->v4addrs.src, &iph->saddr,
+		       sizeof(key_addrs->v4addrs.src));
+		memcpy(&key_addrs->v4addrs.dst, &iph->daddr,
+		       sizeof(key_addrs->v4addrs.dst));
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_PORTS)) {
+		const __be32 *ports = (const __be32 *)
+			((const u8 *)data + thoff);
+
+		key_ports = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_PORTS,
+						      target_container);
+		key_ports->ports = *ports;
+	}
+
+	return true;
+}
+
+/* Same as flow_dissect_fast_ipv4 but for a fixed 40-byte IPv6 header
+ * (no extension headers). The nexthdr check below holds the run-time
+ * invariant; the static_assert holds the compile-time one.
+ */
+static_assert(sizeof(struct ipv6hdr) == 40);
+
+static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	struct flow_dissector_key_addrs *key_addrs;
+	struct flow_dissector_key_basic *key_basic;
+	struct flow_dissector_key_ports *key_ports;
+	const struct ipv6hdr *iph;
+	int thoff;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4))
+		return false;
+
+	iph = (const struct ipv6hdr *)((const u8 *)data + nhoff);
+
+	if (unlikely((*(const u8 *)iph >> 4) != 6))
+		return false;
+
+	/* Any non-zero flow label defers, checked before everything else
+	 * (including the tunnel descents below). Callers passing
+	 * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL (skb_get_hash() -- so
+	 * RPS/RFS, fq, cake) make the slow path stop at a non-zero label
+	 * even on a tunnel outer, and dissectors requesting the label key
+	 * write it; both diverge from the plain parse. Deferring on the
+	 * label itself covers every combination without threading flags
+	 * into the helpers, and zero-label IPv6 (the overwhelmingly
+	 * common case) stays on the fast path.
+	 */
+	if (unlikely((iph->flow_lbl[0] & 0x0f) |
+		     iph->flow_lbl[1] | iph->flow_lbl[2]))
+		return false;
+
+	if (unlikely(iph->nexthdr != IPPROTO_TCP &&
+		     iph->nexthdr != IPPROTO_UDP))
+		return false;
+
+	thoff = nhoff + (int)sizeof(*iph);
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+		key_control->thoff = min_t(u16, thoff,
+					   skb ? skb->len : hlen);
+		key_control->flags = 0;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_BASIC)) {
+		key_basic = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_BASIC,
+						      target_container);
+		key_basic->n_proto = htons(ETH_P_IPV6);
+		key_basic->ip_proto = iph->nexthdr;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
+		key_addrs = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_IPV6_ADDRS,
+						      target_container);
+		memcpy(&key_addrs->v6addrs.src, &iph->saddr,
+		       sizeof(key_addrs->v6addrs.src));
+		memcpy(&key_addrs->v6addrs.dst, &iph->daddr,
+		       sizeof(key_addrs->v6addrs.dst));
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_PORTS)) {
+		const __be32 *ports = (const __be32 *)
+			((const u8 *)data + thoff);
+
+		key_ports = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_PORTS,
+						      target_container);
+		key_ports->ports = *ports;
+	}
+
+	return true;
+}
+
+/* Top-level dispatcher: eligibility check (only the two standard
+ * dissectors and flag subset) + per-proto switch with per-shape
+ * static_branch gating. Each case's branch is a forward not-taken JMP
+ * when its sysctl is 0 — matches the slow-path cost.
+ */
+static bool flow_dissect_fast(const struct sk_buff *skb,
+			      struct flow_dissector *flow_dissector,
+			      void *target_container,
+			      const void *data,
+			      __be16 proto, int nhoff, int hlen,
+			      unsigned int flags)
+{
+	if (flow_dissector != &flow_keys_dissector &&
+	    flow_dissector != &flow_keys_dissector_symmetric)
+		return false;
+
+	/* skb_get_hash() -- behind RPS/RFS, fq, fq_codel, cake -- passes
+	 * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL on every dissect, so the
+	 * flag must be admitted or the fast path never runs for the
+	 * kernel's main consumers. Its only semantic (stop early on a
+	 * non-zero IPv6 flow label) is subsumed by the IPv6 fast path
+	 * deferring on any non-zero label. Any other flag defers.
+	 */
+	if (flags & ~(unsigned int)(FLOW_DISSECTOR_F_PARSE_1ST_FRAG |
+				    FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL))
+		return false;
+
+	switch (proto) {
+	case htons(ETH_P_IP):
+		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
+			return false;
+		return flow_dissect_fast_ipv4(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	case htons(ETH_P_IPV6):
+		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
+			return false;
+		return flow_dissect_fast_ipv6(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	default:
+		return false;
+	}
+}
+
 /**
  * __skb_flow_dissect - extract the flow_keys struct and return it
  * @net: associated network namespace, derived from @skb if NULL
@@ -1183,6 +1427,13 @@ bool __skb_flow_dissect(const struct net *net,
 		rcu_read_unlock();
 	}
 
+	/* Opt-in fast-path; flow_dissect_fast() does the eligibility check
+	 * and per-shape gating.
+	 */
+	if (flow_dissect_fast(skb, flow_dissector, target_container,
+			      data, proto, nhoff, hlen, flags))
+		return true;
+
 	if (dissector_uses_key(flow_dissector,
 			       FLOW_DISSECTOR_KEY_ETH_ADDRS)) {
 		struct flow_dissector_key_eth_addrs *key_eth_addrs;
@@ -1879,8 +2130,6 @@ void make_flow_keys_digest(struct flow_keys_digest *digest,
 }
 EXPORT_SYMBOL(make_flow_keys_digest);
 
-static struct flow_dissector flow_keys_dissector_symmetric __read_mostly;
-
 u32 __skb_get_hash_symmetric_net(const struct net *net, const struct sk_buff *skb)
 {
 	struct flow_keys keys;
@@ -2116,3 +2365,23 @@ static int __init init_default_flow_dissectors(void)
 	return 0;
 }
 core_initcall(init_default_flow_dissectors);
+
+/* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */
+static struct ctl_table flow_dissector_sysctl_table[] = {
+	{
+		.procname	= "eth_ip",
+		.data		= &flow_dissector_eth_ip_key.key,
+		.maxlen		= sizeof(flow_dissector_eth_ip_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
+};
+
+static int __init flow_dissector_sysctl_init(void)
+{
+	if (!register_net_sysctl(&init_net, "net/flow_dissector",
+				 flow_dissector_sysctl_table))
+		return -ENOMEM;
+	return 0;
+}
+late_initcall(flow_dissector_sysctl_init);
-- 
2.54.0

[PATCH net-next v1 03/11] net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP

[Dave Seddon]

Extension of the parent series covering the VLAN-tagged shapes: a
single ETH_P_8021Q or ETH_P_8021AD tag, or two stacked tags (the
canonical QinQ S-tag + C-tag), in front of IPv4/IPv6 + TCP/UDP. Adds
flow_dissect_fast_vlan() that consumes the tag(s) and dispatches to
the existing flow_dissect_fast_ipv4 / _ipv6 helpers on the inner
ethertype.

  - What this covers. VLAN-tagged Eth (8021Q or 8021AD, depth 1 or
    2) + IPv4 or IPv6 (per parent's shape constraints) + TCP/UDP.
    Both hardware-stripped (TCI in skb metadata, inner ethertype in
    skb->protocol; only ever the outermost tag) and in-band (4-byte
    vlan_hdr at data + nhoff) tag forms.

  - Keys written. Outer tag: FLOW_DISSECTOR_KEY_VLAN (vlan_id /
    vlan_priority / vlan_tpid / vlan_eth_type); inner tag (depth 1):
    FLOW_DISSECTOR_KEY_CVLAN, mirroring the slow-path's
    MAX -> VLAN -> CVLAN state machine. Bumps
    FLOW_DISSECTOR_KEY_NUM_OF_VLANS when requested by the dissector.
    Byte-identical with the slow path. Past depth 2 the helper
    returns false (defers to the slow path).

  - Gates. Two static keys, each with its own sysctl:
    DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key)
    (/proc/sys/net/flow_dissector/vlan) gates entry at depth 0, and
    DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key)
    (/proc/sys/net/flow_dissector/qinq) gates the depth-1 tag.
    Defaults 0; off has no behaviour change beyond one extra
    not-taken JMP in the dispatcher switch.

  - Sibling-key handling. The two gates are coupled in their proc
    handlers: writing qinq=1 auto-enables the vlan key (QinQ extends
    VLAN; depth-0 entry is gated by vlan), and turning vlan off clears
    the qinq key (depth-2 cannot fire when the outer entry is off).
    Reads always reflect the underlying key state.

  - Cost. On a miss, one ethertype switch + the static_branch test.
    On a hit, ~3-5 ns added to the IPv4/IPv6 fast-path body per tag
    consumed, while the slow-path's per-tag handling (state-machine
    transition + proto_again rewind) is skipped -- net saving
    ~30 ns/skb on the single-VLAN shape and ~50 ns/skb on QinQ.

Verified byte-identical by the KUnit equivalence suite. Isolated
dissector cost on the allshapes microbench (all fast paths compiled
in, range across 7 measured microarchitectures): vlan -18.1%..-38.5%,
qinq -16.1%..-39.8% -- see the cover letter's performance summary.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst |  31 +++++
 include/net/flow_dissector.h             |   2 +
 net/core/flow_dissector.c                | 147 +++++++++++++++++++++++
 3 files changed, 180 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index aac82169e019..824f4b735caa 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -485,6 +485,37 @@ when this is 0.
 
 Default: 0
 
+vlan
+~~~~
+
+Single 802.1Q or 802.1AD tagged frames over Eth + IPv4/IPv6 +
+TCP/UDP. Handles both the hardware-stripped form (tag in skb
+metadata, inner ethertype in skb->protocol) and the in-band form
+(4-byte vlan_hdr at data + nhoff). Writes FLOW_DISSECTOR_KEY_VLAN
+and bumps FLOW_DISSECTOR_KEY_NUM_OF_VLANS when those keys are
+requested by the dissector. Byte-identical with the slow path for
+the eligible shape; defers on any miss (QinQ defers; see the qinq
+sysctl).
+
+Default: 0
+
+qinq
+~~~~
+
+Lifts the vlan fast-path's depth limit from 1 to 2 so two stacked
+802.1AD / 802.1Q tags (canonical QinQ S-tag + C-tag, or two stacked
+802.1Q tags) are also fast-pathed. Outer tag is written to
+FLOW_DISSECTOR_KEY_VLAN, inner to FLOW_DISSECTOR_KEY_CVLAN, mirroring
+the slow-path's MAX → VLAN → CVLAN state machine. Byte-identical
+with the slow path for the eligible shape.
+
+Auto-toggle: writing qinq=1 also enables vlan (QinQ extends VLAN;
+the depth-0 entry of the helper is gated by the vlan key). Turning
+vlan off (writing 0) also clears qinq (depth-2 cannot fire when
+the outer entry is off).
+
+Default: 0
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index 7d513a6cbb55..4f36f3889a28 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -429,6 +429,8 @@ extern struct flow_dissector flow_keys_basic_dissector;
  * /proc/sys/net/flow_dissector/<shape>. All default off.
  */
 extern struct static_key_false flow_dissector_eth_ip_key;
+extern struct static_key_false flow_dissector_vlan_key;
+extern struct static_key_false flow_dissector_qinq_key;
 
 /* struct flow_keys_digest:
  *
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 88b1bd1cc4ca..826314e28db0 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -42,6 +42,10 @@
  */
 DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key);
 EXPORT_SYMBOL(flow_dissector_eth_ip_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key);
+EXPORT_SYMBOL(flow_dissector_vlan_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key);
+EXPORT_SYMBOL(flow_dissector_qinq_key);
 
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
@@ -1242,6 +1246,93 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 	return true;
 }
 
+/* Strip up to two 802.1Q/802.1AD tags: outer tag ->
+ * FLOW_DISSECTOR_KEY_VLAN, inner -> _CVLAN, mirroring the slow path's
+ * MAX -> VLAN -> CVLAN state machine; more than two tags defer.
+ * Depth 0 is gated by the vlan key, depth >= 1 by the qinq key (the
+ * qinq sysctl also flips vlan on -- see the proc_handlers).
+ */
+static bool flow_dissect_fast_vlan(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   __be16 proto, int nhoff, int hlen,
+				   int vlan_depth)
+{
+	struct flow_dissector_key_num_of_vlans *key_nvs;
+	struct flow_dissector_key_vlan *key_vlan;
+	enum flow_dissector_key_id vlan_key;
+	__be16 inner_proto;
+	__be16 saved_tpid;
+	u16 tci_prio;
+	u16 tci_id;
+
+	if (vlan_depth >= 1 &&
+	    !static_branch_unlikely(&flow_dissector_qinq_key))
+		return false;
+	if (vlan_depth >= 2)
+		return false;
+
+	vlan_key = vlan_depth == 0 ?
+		   FLOW_DISSECTOR_KEY_VLAN :
+		   FLOW_DISSECTOR_KEY_CVLAN;
+	saved_tpid = proto;
+
+	/* HW-stripped tag is only ever the outermost. */
+	if (vlan_depth == 0 && skb && skb_vlan_tag_present(skb)) {
+		tci_id = skb_vlan_tag_get_id(skb);
+		tci_prio = skb_vlan_tag_get_prio(skb);
+		inner_proto = skb->protocol;
+	} else {
+		const struct vlan_hdr *vlan;
+
+		if (unlikely(hlen - nhoff < (int)sizeof(*vlan)))
+			return false;
+		vlan = (const struct vlan_hdr *)((const u8 *)data + nhoff);
+		tci_id = ntohs(vlan->h_vlan_TCI) & VLAN_VID_MASK;
+		tci_prio = (ntohs(vlan->h_vlan_TCI) &
+			    VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT;
+		inner_proto = vlan->h_vlan_encapsulated_proto;
+		nhoff += (int)sizeof(*vlan);
+	}
+
+	if (dissector_uses_key(flow_dissector, vlan_key)) {
+		key_vlan = skb_flow_dissector_target(flow_dissector,
+						     vlan_key,
+						     target_container);
+		key_vlan->vlan_id = tci_id;
+		key_vlan->vlan_priority = tci_prio;
+		key_vlan->vlan_tpid = saved_tpid;
+		key_vlan->vlan_eth_type = inner_proto;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_NUM_OF_VLANS)) {
+		key_nvs = skb_flow_dissector_target(flow_dissector,
+						    FLOW_DISSECTOR_KEY_NUM_OF_VLANS,
+						    target_container);
+		key_nvs->num_of_vlans++;
+	}
+
+	switch (inner_proto) {
+	case htons(ETH_P_IP):
+		return flow_dissect_fast_ipv4(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	case htons(ETH_P_IPV6):
+		return flow_dissect_fast_ipv6(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	case htons(ETH_P_8021Q):
+	case htons(ETH_P_8021AD):
+		return flow_dissect_fast_vlan(skb, flow_dissector,
+					      target_container, data,
+					      inner_proto, nhoff, hlen,
+					      vlan_depth + 1);
+	default:
+		return false;
+	}
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -1270,6 +1361,13 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 		return false;
 
 	switch (proto) {
+	case htons(ETH_P_8021Q):
+	case htons(ETH_P_8021AD):
+		if (!static_branch_unlikely(&flow_dissector_vlan_key))
+			return false;
+		return flow_dissect_fast_vlan(skb, flow_dissector,
+					      target_container, data,
+					      proto, nhoff, hlen, 0);
 	case htons(ETH_P_IP):
 		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
 			return false;
@@ -2367,6 +2465,41 @@ static int __init init_default_flow_dissectors(void)
 core_initcall(init_default_flow_dissectors);
 
 /* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */
+/* proc_handler for net.flow_dissector.vlan. Writes of 0 also clear
+ * flow_dissector_qinq_key — QinQ depth-2 is meaningless when the
+ * outer single-VLAN gate is off, so the slow path would have to
+ * handle every VLAN-tagged packet anyway.
+ */
+static int proc_set_vlan_key(const struct ctl_table *table, int write,
+			     void *buffer, size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_do_static_key(table, write, buffer, lenp, ppos);
+	if (ret == 0 && write &&
+	    !static_branch_unlikely(&flow_dissector_vlan_key) &&
+	    static_branch_unlikely(&flow_dissector_qinq_key))
+		static_branch_disable(&flow_dissector_qinq_key);
+	return ret;
+}
+
+/* proc_handler for net.flow_dissector.qinq. Writes of 1 also enable
+ * flow_dissector_vlan_key — QinQ extends the VLAN fast-path to depth
+ * 2; it cannot fire when the depth-0 entry is gated off.
+ */
+static int proc_set_qinq_key(const struct ctl_table *table, int write,
+			     void *buffer, size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_do_static_key(table, write, buffer, lenp, ppos);
+	if (ret == 0 && write &&
+	    static_branch_unlikely(&flow_dissector_qinq_key) &&
+	    !static_branch_unlikely(&flow_dissector_vlan_key))
+		static_branch_enable(&flow_dissector_vlan_key);
+	return ret;
+}
+
 static struct ctl_table flow_dissector_sysctl_table[] = {
 	{
 		.procname	= "eth_ip",
@@ -2375,6 +2508,20 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_do_static_key,
 	},
+	{
+		.procname	= "vlan",
+		.data		= &flow_dissector_vlan_key.key,
+		.maxlen		= sizeof(flow_dissector_vlan_key),
+		.mode		= 0644,
+		.proc_handler	= proc_set_vlan_key,
+	},
+	{
+		.procname	= "qinq",
+		.data		= &flow_dissector_qinq_key.key,
+		.maxlen		= sizeof(flow_dissector_qinq_key),
+		.mode		= 0644,
+		.proc_handler	= proc_set_qinq_key,
+	},
 };
 
 static int __init flow_dissector_sysctl_init(void)
-- 
2.54.0

[PATCH net-next v1 04/11] net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} + TCP/UDP

[Dave Seddon]

Adds flow_dissect_fast_pppoe() — a straight-line extractor for PPPoE
session frames (ETH_P_PPP_SES, EtherType 0x8864 per RFC 2516) carrying
IPv4 or IPv6 inside PPP. Strips the 6-byte pppoe_hdr + 2-byte PPP
protocol field, writes FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto,
type) when requested, and tail-calls into the eth_ip fast-path for
the inner IPv4 or IPv6 + TCP/UDP.

Byte-identical with the slow path's `case htons(ETH_P_PPP_SES)` block
for the eligible shape (RFC-2516-valid header + PPP_IP / PPP_IPV6 +
straight-line IP). On any miss — invalid header, PFC-compressed PPP
proto, PPP_MPLS_UC / PPP_MPLS_MC (defer until MPLS fast-path lands),
LCP / IPCP / other control frames — returns false so the dispatcher
falls through to the slow path. Output is exactly what the slow path
would produce.

Why this matters: PPPoE is the dominant L2 over millions of ISP last-
mile DSL/FTTH links, and the existing slow-path PPPoE block does a
__skb_header_pointer + per-field switch on every packet at the
graph-walk entry. The fast-path saves the walk overhead for the
common IPv4 / IPv6 inner case while keeping the slow path as the
escape valve for control / less-common inner protos.

Gated by:
  - new DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key)
  - new sysctl /proc/sys/net/flow_dissector/pppoe (default 0)
  - dispatcher case `htons(ETH_P_PPP_SES)` with
    `static_branch_unlikely(&flow_dissector_pppoe_key)` guard

When the sysctl is 0, the dispatcher hits a forward not-taken JMP —
same per-call cost as the vlan / qinq cases. The new dispatcher
case also doesn't widen the switch table (it adds one case to a
switch the compiler can already lower to a jump table on the
ETH_P_* constants).

Documentation/admin-guide/sysctl/net.rst grows a `pppoe` subsection
under the existing `/proc/sys/net/flow_dissector/` chapter, between
qinq and the next chapter, matching the existing per-shape layout.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst | 18 ++++++
 include/net/flow_dissector.h             |  1 +
 net/core/flow_dissector.c                | 76 ++++++++++++++++++++++++
 3 files changed, 95 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index 824f4b735caa..1096e6ddca39 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -516,6 +516,24 @@ the outer entry is off).
 
 Default: 0
 
+pppoe
+~~~~~
+
+PPPoE session frames (RFC 2516, EtherType ``ETH_P_PPP_SES`` 0x8864)
+carrying IPv4 or IPv6 inside PPP. The fast-path strips the 6-byte
+PPPoE session header + 2-byte PPP protocol field, writes
+FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto, type) when the
+dissector requests it, and tail-calls into the eth_ip fast-path for
+the inner IPv4 / IPv6 + TCP/UDP. Byte-identical with the slow path
+for the eligible shape.
+
+Defers to slow path for: PPP_MPLS_UC / PPP_MPLS_MC (until an MPLS
+fast-path lands), PFC-compressed PPP protocol fields, LCP/IPCP/etc.
+control frames, and any malformed PPPoE header (ver != 1, type != 1,
+code != 0).
+
+Default: 0
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index 4f36f3889a28..df85162959cc 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -431,6 +431,7 @@ extern struct flow_dissector flow_keys_basic_dissector;
 extern struct static_key_false flow_dissector_eth_ip_key;
 extern struct static_key_false flow_dissector_vlan_key;
 extern struct static_key_false flow_dissector_qinq_key;
+extern struct static_key_false flow_dissector_pppoe_key;
 
 /* struct flow_keys_digest:
  *
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 826314e28db0..21a3f5e0ea05 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -46,6 +46,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key);
 EXPORT_SYMBOL(flow_dissector_vlan_key);
 DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key);
 EXPORT_SYMBOL(flow_dissector_qinq_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key);
+EXPORT_SYMBOL(flow_dissector_pppoe_key);
 
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
@@ -1333,6 +1335,67 @@ static bool flow_dissect_fast_vlan(const struct sk_buff *skb,
 	}
 }
 
+/* PPPoE session (RFC 2516) + 2-byte PPP protocol: write
+ * FLOW_DISSECTOR_KEY_PPPOE if requested, then tail-call the eth_ip
+ * fast-path for PPP_IP / PPP_IPV6. Anything else (PFC-compressed
+ * protocol, LCP, MPLS) defers.
+ */
+static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,
+				    struct flow_dissector *flow_dissector,
+				    void *target_container,
+				    const void *data,
+				    int nhoff, int hlen)
+{
+	struct flow_dissector_key_pppoe *key_pppoe;
+	const struct {
+		struct pppoe_hdr hdr;
+		__be16 proto;
+	} *hdr;
+	__be16 inner_eth_proto;
+	u16 ppp_proto;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*hdr)))
+		return false;
+	hdr = (const void *)((const u8 *)data + nhoff);
+	if (!is_pppoe_ses_hdr_valid(&hdr->hdr))
+		return false;
+
+	ppp_proto = ntohs(hdr->proto);
+	switch (ppp_proto) {
+	case PPP_IP:
+		inner_eth_proto = htons(ETH_P_IP);
+		break;
+	case PPP_IPV6:
+		inner_eth_proto = htons(ETH_P_IPV6);
+		break;
+	default:
+		/* MPLS unicast/multicast + everything else (LCP, IPCP,
+		 * compressed PFC, ...) defers to the slow path.
+		 */
+		return false;
+	}
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_PPPOE)) {
+		key_pppoe = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_PPPOE,
+						      target_container);
+		key_pppoe->session_id = hdr->hdr.sid;
+		key_pppoe->ppp_proto = htons(ppp_proto);
+		key_pppoe->type = htons(ETH_P_PPP_SES);
+	}
+
+	nhoff += PPPOE_SES_HLEN;
+
+	if (inner_eth_proto == htons(ETH_P_IP))
+		return flow_dissect_fast_ipv4(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	return flow_dissect_fast_ipv6(skb, flow_dissector,
+				      target_container, data,
+				      nhoff, hlen);
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -1380,6 +1443,12 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 		return flow_dissect_fast_ipv6(skb, flow_dissector,
 					      target_container, data,
 					      nhoff, hlen);
+	case htons(ETH_P_PPP_SES):
+		if (!static_branch_unlikely(&flow_dissector_pppoe_key))
+			return false;
+		return flow_dissect_fast_pppoe(skb, flow_dissector,
+					       target_container, data,
+					       nhoff, hlen);
 	default:
 		return false;
 	}
@@ -2522,6 +2591,13 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_set_qinq_key,
 	},
+	{
+		.procname	= "pppoe",
+		.data		= &flow_dissector_pppoe_key.key,
+		.maxlen		= sizeof(flow_dissector_pppoe_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
 };
 
 static int __init flow_dissector_sysctl_init(void)
-- 
2.54.0

[PATCH net-next v1 05/11] net: flow_dissector: add fast-path for single MPLS label + IP

[Dave Seddon]

Adds flow_dissect_fast_mpls() — straight-line read of one MPLS label
entry (ETH_P_MPLS_UC 0x8847 / ETH_P_MPLS_MC 0x8848), extracting
label / TC / TTL / BoS and writing to FLOW_DISSECTOR_KEY_MPLS lse[0]
when the dissector requests it.

Byte-identical with the slow path's __skb_flow_dissect_mpls() called
with lse_index=0, bos=1: the slow path returns FLOW_DISSECT_RET_OUT_GOOD
when BoS is set without descending into the inner packet (MPLS flow
hashing is on the label stack, not the inner 5-tuple), so this
fast-path does the same — no IP tail-call.

Multi-label stacks (BoS=0 on the first label) defer to the slow
path. A depth-2+ variant can land later, mirroring the vlan ->
qinq staging used for the VLAN gates.

Why this matters: MPLS is the dominant SP backbone L2.5 and the
existing slow-path traverses an iteration loop per label even when
the consumer just wants the label-stack flow key. The fast-path
saves the iteration overhead for the common single-label case while
keeping the slow path as escape for multi-label and the
entropy-label case (which can only fire from label 2+ anyway).

Gated by:
  - new DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key)
  - new sysctl /proc/sys/net/flow_dissector/mpls (default 0)
  - dispatcher case htons(ETH_P_MPLS_UC) / htons(ETH_P_MPLS_MC) with
    static_branch_unlikely(&flow_dissector_mpls_key) guard

When the sysctl is 0, dispatcher cost is one not-taken JMP per
matching packet — same per-shape pattern as the other fast-path
patches.

Documentation/admin-guide/sysctl/net.rst grows an `mpls` subsection
under /proc/sys/net/flow_dissector/.

The helper also mirrors the slow path's out_good terminal writes: for
the standard flow_keys dissectors (which request no MPLS key), the
slow path returns OUT_GOOD after the first LSE with nhoff advanced
past it, so thoff, n_proto (the MPLS ethertype) and ip_proto (0) are
still written at the exit label. Skipping those writes is observable
in struct flow_keys and would break the byte-identical contract; the
in-tree KUnit equivalence test added later in this series catches
exactly this.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst | 19 ++++++
 include/net/flow_dissector.h             |  1 +
 net/core/flow_dissector.c                | 83 ++++++++++++++++++++++++
 3 files changed, 103 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index 1096e6ddca39..1ee5419ffaeb 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -534,6 +534,25 @@ code != 0).
 
 Default: 0
 
+mpls
+~~~~
+
+Single-label MPLS frames (EtherType ``ETH_P_MPLS_UC`` 0x8847 or
+``ETH_P_MPLS_MC`` 0x8848 with bottom-of-stack bit set on the first
+label). The fast-path reads the 4-byte label entry, extracts
+label / TC / TTL / BoS, and writes them to FLOW_DISSECTOR_KEY_MPLS
+lse[0] when requested. Byte-identical with the slow path's
+__skb_flow_dissect_mpls() for lse_index=0, bos=1: the slow path
+returns OUT_GOOD on BoS without descending into the inner packet
+(MPLS hashing is on the label stack, not inner 5-tuple), so this
+fast-path does the same.
+
+Multi-label stacks (BoS=0 on the first label) defer to slow path —
+extending to depth-2 or more is a follow-up, mirroring the
+vlan -> qinq staging used earlier in this series.
+
+Default: 0
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index df85162959cc..8df7821ba769 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -432,6 +432,7 @@ extern struct static_key_false flow_dissector_eth_ip_key;
 extern struct static_key_false flow_dissector_vlan_key;
 extern struct static_key_false flow_dissector_qinq_key;
 extern struct static_key_false flow_dissector_pppoe_key;
+extern struct static_key_false flow_dissector_mpls_key;
 
 /* struct flow_keys_digest:
  *
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 21a3f5e0ea05..0ec1a6a1bb0e 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -48,6 +48,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key);
 EXPORT_SYMBOL(flow_dissector_qinq_key);
 DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key);
 EXPORT_SYMBOL(flow_dissector_pppoe_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key);
+EXPORT_SYMBOL(flow_dissector_mpls_key);
 
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
@@ -1396,6 +1398,73 @@ static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,
 				      nhoff, hlen);
 }
 
+/* Single MPLS label (BoS=1): write FLOW_DISSECTOR_KEY_MPLS lse[0]
+ * when requested and stop -- the slow path returns OUT_GOOD on BoS
+ * without descending into the inner packet, so the fast path must
+ * not tail-call either. Multi-label stacks defer.
+ */
+static bool flow_dissect_fast_mpls(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   __be16 proto, int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	struct flow_dissector_key_basic *key_basic;
+	struct flow_dissector_key_mpls *key_mpls;
+	struct flow_dissector_mpls_lse *lse;
+	const struct mpls_label *hdr;
+	u32 entry, label, bos;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*hdr)))
+		return false;
+	hdr = (const struct mpls_label *)((const u8 *)data + nhoff);
+	entry = ntohl(hdr->entry);
+	bos = (entry & MPLS_LS_S_MASK) >> MPLS_LS_S_SHIFT;
+
+	/* Multi-label stack: defer. */
+	if (!bos)
+		return false;
+
+	/* Neither MPLS key requested: the slow path returns OUT_GOOD without
+	 * writing the key; skipping the write block matches it.
+	 */
+	label = (entry & MPLS_LS_LABEL_MASK) >> MPLS_LS_LABEL_SHIFT;
+
+	if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_MPLS)) {
+		key_mpls = skb_flow_dissector_target(flow_dissector,
+						     FLOW_DISSECTOR_KEY_MPLS,
+						     target_container);
+		lse = &key_mpls->ls[0];
+		lse->mpls_ttl = (entry & MPLS_LS_TTL_MASK) >> MPLS_LS_TTL_SHIFT;
+		lse->mpls_bos = bos;
+		lse->mpls_tc = (entry & MPLS_LS_TC_MASK) >> MPLS_LS_TC_SHIFT;
+		lse->mpls_label = label;
+		dissector_set_mpls_lse(key_mpls, 0);
+	}
+
+	/* Mirror the slow path's out_good terminal: nhoff advances past the
+	 * LSE, then thoff/basic are written from the final proto/ip_proto.
+	 */
+	nhoff += (int)sizeof(*hdr);
+	if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->thoff = min_t(u16, nhoff,
+					   skb ? skb->len : hlen);
+	}
+	if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_BASIC)) {
+		key_basic = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_BASIC,
+						      target_container);
+		key_basic->n_proto = proto;
+		key_basic->ip_proto = 0;
+	}
+
+	return true;
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -1449,6 +1518,13 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 		return flow_dissect_fast_pppoe(skb, flow_dissector,
 					       target_container, data,
 					       nhoff, hlen);
+	case htons(ETH_P_MPLS_UC):
+	case htons(ETH_P_MPLS_MC):
+		if (!static_branch_unlikely(&flow_dissector_mpls_key))
+			return false;
+		return flow_dissect_fast_mpls(skb, flow_dissector,
+					      target_container, data,
+					      proto, nhoff, hlen);
 	default:
 		return false;
 	}
@@ -2598,6 +2674,13 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_do_static_key,
 	},
+	{
+		.procname	= "mpls",
+		.data		= &flow_dissector_mpls_key.key,
+		.maxlen		= sizeof(flow_dissector_mpls_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
 };
 
 static int __init flow_dissector_sysctl_init(void)
-- 
2.54.0

[PATCH net-next v1 06/11] net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 / 6in4)

[Dave Seddon]

Adds flow_dissect_fast_ipip_inner() — a small recursion helper
invoked from the existing flow_dissect_fast_ipv4() and
flow_dissect_fast_ipv6() helpers when the outer IP's protocol
(IPv4) or nexthdr (IPv6) is IPPROTO_IPIP or IPPROTO_IPV6 and the
ipip gate is on. Tail-calls into the matching inner IP fast-path
then stamps FLOW_DIS_ENCAPSULATION on key_control->flags. This is
the series' first tunnel descent: the fast path parses through the
tunnel header and dissects the inner flow, the pattern the GRE and
UDP-tunnel patches later in the series reuse.

Byte-identical with the slow path's IPPROTO_IPIP and IPPROTO_IPV6
switch cases in __skb_flow_dissect(): the slow path also lets the
inner IP overwrite outer addrs/basic and OR's ENCAP into the
control flags. The ENCAP write must happen *after* the inner
returns because the existing fast-path helpers unconditionally
set key_control->flags = 0 on entry (correct for the top-level
call but clobbers the encap flag during recursion); the
ipip_inner helper re-establishes ENCAP after the inner returns.

Coverage:
  outer IPv4, protocol == IPIP   -> inner IPv4
  outer IPv4, protocol == IPV6   -> inner IPv6 (6in4)
  outer IPv6, nexthdr  == IPIP   -> inner IPv4 (4in6)
  outer IPv6, nexthdr  == IPV6   -> inner IPv6

FLOW_DISSECTOR_F_STOP_BEFORE_ENCAP and STOP_AT_ENCAP don't apply
because the top-level dispatcher already rejects anything beyond
F_PARSE_1ST_FRAG, so the fast-path entry never sees the encap-stop
flags - packets that request them go to the slow path unchanged.

Gated by:
  - new DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key)
  - new sysctl /proc/sys/net/flow_dissector/ipip (default 0)
  - static_branch_unlikely guard inside the v4/v6 helpers, hit only
    when the outer protocol/nexthdr is not TCP/UDP (i.e. the
    TCP/UDP fast-path was about to return false anyway)

Cost when off: the not-TCP/UDP fall-through grows by one
static_branch_unlikely check. The TCP/UDP hot path is
unchanged — the new branch is below the existing TCP/UDP fast-out.

The IPv6 helper's standalone not-TCP/UDP guard becomes unreachable
once the new block handles that case in full, so it is removed here.

Documentation/admin-guide/sysctl/net.rst grows an `ipip` subsection
under /proc/sys/net/flow_dissector/.

The IPv6-outer descent mirrors the slow path's outer writes (v6addrs,
addr_type) before recursing into the inner header. The slow path
fills these for the outer header and the inner pass then overwrites
only what it uses: an inner IPv4 overwrites just the first 8 bytes of
the addrs union, leaving outer-v6 bytes in the tail. Byte-identical
output means reproducing exactly that, residue included — the KUnit
equivalence test's 4in6 cases fail otherwise. (A non-zero outer flow
label defers before any descent — see the eth_ip patch's output
contract — so no label residue exists. The IPv4-outer descent needs
no mirroring: an inner IPv4 or IPv6 write always fully covers the
outer's 8 address bytes.)

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst |  19 ++++
 include/net/flow_dissector.h             |   1 +
 net/core/flow_dissector.c                | 117 ++++++++++++++++++++++-
 3 files changed, 134 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index 1ee5419ffaeb..15c8ace2c0a4 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -553,6 +553,25 @@ vlan -> qinq staging used earlier in this series.
 
 Default: 0
 
+ipip
+~~~~
+
+IPv4-in-IPv4 (``IPPROTO_IPIP`` 4), IPv6-in-IPv4 (``IPPROTO_IPV6`` 41
+inside an IPv4 outer), IPv4-in-IPv6 and IPv6-in-IPv6 tunnels. When
+the outer IP fast-path notices the outer protocol/nexthdr is one of
+these and this gate is on, it skips the outer header and tail-calls
+into the inner IPv4 / IPv6 fast-path. The flow_keys ends up with the
+inner 5-tuple (matching the slow path's behaviour, where the inner
+parse overwrites the outer addrs/basic), plus
+FLOW_DIS_ENCAPSULATION set on key_control->flags.
+
+Byte-identical with the slow path's IPPROTO_IPIP / IPPROTO_IPV6
+switch cases for the eligible shape. Defers on any miss — including
+fragmented outer, unusual outer IHL, or unsupported inner shapes
+that the inner IP fast-path itself would defer.
+
+Default: 0
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index 8df7821ba769..d9b4b461cd05 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -433,6 +433,7 @@ extern struct static_key_false flow_dissector_vlan_key;
 extern struct static_key_false flow_dissector_qinq_key;
 extern struct static_key_false flow_dissector_pppoe_key;
 extern struct static_key_false flow_dissector_mpls_key;
+extern struct static_key_false flow_dissector_ipip_key;
 
 /* struct flow_keys_digest:
  *
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 0ec1a6a1bb0e..abf4fdb0100b 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -50,6 +50,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key);
 EXPORT_SYMBOL(flow_dissector_pppoe_key);
 DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key);
 EXPORT_SYMBOL(flow_dissector_mpls_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key);
+EXPORT_SYMBOL(flow_dissector_ipip_key);
 
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
@@ -65,6 +67,12 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 				   void *target_container,
 				   const void *data,
 				   int nhoff, int hlen);
+static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
+					 struct flow_dissector *flow_dissector,
+					 void *target_container,
+					 const void *data,
+					 __be16 inner_eth_proto,
+					 int inner_nhoff, int hlen);
 
 /* One of the two dissectors the fast-path eligibility check admits;
  * defined here so flow_dissect_fast() below can reference it (its keys
@@ -1114,8 +1122,25 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
 		return false;
 
 	if (unlikely(iph->protocol != IPPROTO_TCP &&
-		     iph->protocol != IPPROTO_UDP))
+		     iph->protocol != IPPROTO_UDP)) {
+		/* IPIP / IPv6-in-IPv4 outer: defer to the inner fast-path and
+		 * stamp ENCAP. Outer addrs are not written -- the inner pass
+		 * overwrites them, as in the slow path.
+		 */
+		if (static_branch_unlikely(&flow_dissector_ipip_key)) {
+			if (iph->protocol == IPPROTO_IPIP)
+				return flow_dissect_fast_ipip_inner(skb,
+						flow_dissector, target_container,
+						data, htons(ETH_P_IP),
+						nhoff + (int)sizeof(*iph), hlen);
+			if (iph->protocol == IPPROTO_IPV6)
+				return flow_dissect_fast_ipip_inner(skb,
+						flow_dissector, target_container,
+						data, htons(ETH_P_IPV6),
+						nhoff + (int)sizeof(*iph), hlen);
+		}
 		return false;
+	}
 
 	thoff = nhoff + (int)sizeof(*iph);
 
@@ -1203,8 +1228,49 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 		return false;
 
 	if (unlikely(iph->nexthdr != IPPROTO_TCP &&
-		     iph->nexthdr != IPPROTO_UDP))
-		return false;
+		     iph->nexthdr != IPPROTO_UDP)) {
+		/* IPIP / IPV6-in-IPv6 tunnel outer: same pattern as the
+		 * IPv4 helper. The nexthdr_IPIP / nexthdr_IPV6 cases
+		 * defer to the inner IP fast-path and stamp ENCAP.
+		 */
+		bool ipip = static_branch_unlikely(&flow_dissector_ipip_key) &&
+			    (iph->nexthdr == IPPROTO_IPIP ||
+			     iph->nexthdr == IPPROTO_IPV6);
+
+		if (!ipip)
+			return false;
+
+		/* Mirror the slow path's outer-IPv6 writes before the
+		 * descent. The slow path fills v6addrs for the outer
+		 * header and the inner pass then overwrites only what
+		 * it uses — an inner IPv4 leaves the tail of the addrs
+		 * union holding outer-v6 bytes. Byte-identical means
+		 * reproducing exactly that, residue included. (The
+		 * outer flow label is always zero here — a non-zero
+		 * label deferred above — so there is no label residue.)
+		 */
+		if (dissector_uses_key(flow_dissector,
+				       FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
+			key_addrs = skb_flow_dissector_target(flow_dissector,
+							      FLOW_DISSECTOR_KEY_IPV6_ADDRS,
+							      target_container);
+			memcpy(&key_addrs->v6addrs.src, &iph->saddr,
+			       sizeof(key_addrs->v6addrs.src));
+			memcpy(&key_addrs->v6addrs.dst, &iph->daddr,
+			       sizeof(key_addrs->v6addrs.dst));
+			key_control = skb_flow_dissector_target(flow_dissector,
+								FLOW_DISSECTOR_KEY_CONTROL,
+								target_container);
+			key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+		}
+
+		return flow_dissect_fast_ipip_inner(skb,
+				flow_dissector, target_container,
+				data,
+				iph->nexthdr == IPPROTO_IPIP ?
+				htons(ETH_P_IP) : htons(ETH_P_IPV6),
+				nhoff + (int)sizeof(*iph), hlen);
+	}
 
 	thoff = nhoff + (int)sizeof(*iph);
 
@@ -1465,6 +1531,44 @@ static bool flow_dissect_fast_mpls(const struct sk_buff *skb,
 	return true;
 }
 
+/* IPIP / 4in6 / 6in4 inner descent: recurse into the inner IP
+ * fast-path, then stamp FLOW_DIS_ENCAPSULATION -- the inner helper
+ * zeros key_control->flags, so the ENCAP write must come after.
+ */
+static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
+					 struct flow_dissector *flow_dissector,
+					 void *target_container,
+					 const void *data,
+					 __be16 inner_eth_proto,
+					 int inner_nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	bool ok;
+
+	if (inner_eth_proto == htons(ETH_P_IP))
+		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
+					    target_container, data,
+					    inner_nhoff, hlen);
+	else if (inner_eth_proto == htons(ETH_P_IPV6))
+		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
+					    target_container, data,
+					    inner_nhoff, hlen);
+	else
+		return false;
+
+	if (!ok)
+		return false;
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->flags |= FLOW_DIS_ENCAPSULATION;
+	}
+	return true;
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -2681,6 +2785,13 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_do_static_key,
 	},
+	{
+		.procname	= "ipip",
+		.data		= &flow_dissector_ipip_key.key,
+		.maxlen		= sizeof(flow_dissector_ipip_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
 };
 
 static int __init flow_dissector_sysctl_init(void)
-- 
2.54.0

[PATCH net-next v1 07/11] net: flow_dissector: add byte-identical fast-path for plain GRE inner

[Dave Seddon]

Adds flow_dissect_fast_gre_inner() — invoked from inside the
existing flow_dissect_fast_ipv4() / flow_dissect_fast_ipv6() helpers
when the outer IP's protocol/nexthdr is IPPROTO_GRE and the GRE gate
is on.

Byte-identical with the slow path's __skb_flow_dissect_gre() for the
subset where:

  - GRE version == 0 (v1 = PPTP defers to slow path)
  - All GRE flags clear (no CSUM, KEY, SEQ, ROUTING)
  - protocol == ETH_P_IP or ETH_P_IPV6 (no TEB, PPP, MPLS-over-GRE)

In that subset, the GRE base header is exactly 4 bytes; slow path also
descends to inner IP (sets *p_proto = hdr->protocol and returns
PROTO_AGAIN) and stamps FLOW_DIS_ENCAPSULATION on key_control->flags.
The fast-path produces the same output: skips the 4-byte header,
tail-calls into the inner IP fast-path, re-establishes ENCAP on
key_control after the inner overwrites it (same pattern as the
IPIP family fast-path from the prior patch).

Why this matters: plain GRE (no key, no checksum) is the dominant
shape for cloud backbone tunnels (AWS GRE between regions, GCP
inter-region overlays) and corporate site-to-site VPNs that don't
need per-tunnel keying. The existing slow-path GRE handler does a
__skb_header_pointer + flag-walk per packet; the fast-path saves
that for the common case while keeping the slow path as the escape
valve for GRE-with-flags, v1/PPTP, TEB, and PPP-over-GRE.

Gated by:
  - new DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key)
  - new sysctl /proc/sys/net/flow_dissector/gre (default 0)
  - static_branch_unlikely guard inside the v4/v6 helpers, on the
    same not-TCP/UDP fall-through path as the IPIP gate (so the
    eth_ip hot path remains unchanged when this is off)

Cost when off: the not-TCP/UDP fall-through grows by one
static_branch_unlikely check on top of the IPIP check. The TCP/UDP
hot path is untouched.

GRE-with-KEY (used in some MPLS-over-GRE and cloud-overlay
deployments) is a natural follow-up: same descent shape with the
optional 4-byte key field read and a write to
FLOW_DISSECTOR_KEY_GRE_KEYID.

Documentation/admin-guide/sysctl/net.rst grows a `gre` subsection
under /proc/sys/net/flow_dissector/.

Classification note: the slow path already descends through plain
GRE unconditionally, so this fast path is byte-identical with
today's behaviour — a pure CPU saving. That is unlike the UDP-tunnel
descents later in the series (VXLAN/Geneve/GTP-U/FOU/GUE), where the
slow path stops at the outer UDP header today and the descent itself
is the new, gated behaviour.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst |  25 ++++++
 include/net/flow_dissector.h             |   1 +
 net/core/flow_dissector.c                | 101 +++++++++++++++++++++--
 3 files changed, 121 insertions(+), 6 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index 15c8ace2c0a4..dbc819740db0 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -572,6 +572,31 @@ that the inner IP fast-path itself would defer.
 
 Default: 0
 
+gre
+~~~
+
+GRE-encapsulated inner IP (``IPPROTO_GRE`` 47). The fast-path
+mirrors the slow path's __skb_flow_dissect_gre() descent for the
+common subset:
+
+- GRE version 0 (the v1 PPTP variant defers to slow path)
+- All GRE flags clear (no GRE_CSUM, GRE_KEY, GRE_SEQ, GRE_ROUTING —
+  i.e. plain 4-byte GRE base header)
+- protocol field is ``ETH_P_IP`` 0x0800 or ``ETH_P_IPV6`` 0x86DD
+  (no Transparent Ethernet Bridging, no PPP-over-GRE, no MPLS-
+  over-GRE)
+
+In that subset, slow path also descends to inner IP and stamps
+``key_control->flags |= FLOW_DIS_ENCAPSULATION``; the fast-path
+produces the same output.
+
+GRE-with-KEY (common in MPLS-over-GRE deployments and some cloud
+overlays) is a follow-up patch — same descent shape with an
+additional 4-byte key field read and a write to
+``FLOW_DISSECTOR_KEY_GRE_KEYID``.
+
+Default: 0
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index d9b4b461cd05..ba7226a42d19 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -434,6 +434,7 @@ extern struct static_key_false flow_dissector_qinq_key;
 extern struct static_key_false flow_dissector_pppoe_key;
 extern struct static_key_false flow_dissector_mpls_key;
 extern struct static_key_false flow_dissector_ipip_key;
+extern struct static_key_false flow_dissector_gre_key;
 
 /* struct flow_keys_digest:
  *
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index abf4fdb0100b..b79c80612dee 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -52,6 +52,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key);
 EXPORT_SYMBOL(flow_dissector_mpls_key);
 DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key);
 EXPORT_SYMBOL(flow_dissector_ipip_key);
+DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key);
+EXPORT_SYMBOL(flow_dissector_gre_key);
 
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
@@ -73,6 +75,11 @@ static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
 					 const void *data,
 					 __be16 inner_eth_proto,
 					 int inner_nhoff, int hlen);
+static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
+					struct flow_dissector *flow_dissector,
+					void *target_container,
+					const void *data,
+					int nhoff, int hlen);
 
 /* One of the two dissectors the fast-path eligibility check admits;
  * defined here so flow_dissect_fast() below can reference it (its keys
@@ -1139,6 +1146,11 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
 						data, htons(ETH_P_IPV6),
 						nhoff + (int)sizeof(*iph), hlen);
 		}
+		if (static_branch_unlikely(&flow_dissector_gre_key) &&
+		    iph->protocol == IPPROTO_GRE)
+			return flow_dissect_fast_gre_inner(skb, flow_dissector,
+					target_container, data,
+					nhoff + (int)sizeof(*iph), hlen);
 		return false;
 	}
 
@@ -1236,8 +1248,10 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 		bool ipip = static_branch_unlikely(&flow_dissector_ipip_key) &&
 			    (iph->nexthdr == IPPROTO_IPIP ||
 			     iph->nexthdr == IPPROTO_IPV6);
+		bool gre = static_branch_unlikely(&flow_dissector_gre_key) &&
+			   iph->nexthdr == IPPROTO_GRE;
 
-		if (!ipip)
+		if (!ipip && !gre)
 			return false;
 
 		/* Mirror the slow path's outer-IPv6 writes before the
@@ -1264,11 +1278,15 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 			key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
 		}
 
-		return flow_dissect_fast_ipip_inner(skb,
-				flow_dissector, target_container,
-				data,
-				iph->nexthdr == IPPROTO_IPIP ?
-				htons(ETH_P_IP) : htons(ETH_P_IPV6),
+		if (ipip)
+			return flow_dissect_fast_ipip_inner(skb,
+					flow_dissector, target_container,
+					data,
+					iph->nexthdr == IPPROTO_IPIP ?
+					htons(ETH_P_IP) : htons(ETH_P_IPV6),
+					nhoff + (int)sizeof(*iph), hlen);
+		return flow_dissect_fast_gre_inner(skb, flow_dissector,
+				target_container, data,
 				nhoff + (int)sizeof(*iph), hlen);
 	}
 
@@ -1569,6 +1587,70 @@ static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
 	return true;
 }
 
+/* Plain-GRE inner descent (version 0, no flags, inner IP): a 4-byte
+ * base header the slow path steps over with PROTO_AGAIN. Any flags,
+ * version 1 (PPTP) or a non-IP inner defers. ENCAP is stamped after
+ * the inner pass, which zeros key_control->flags.
+ */
+static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
+					struct flow_dissector *flow_dissector,
+					void *target_container,
+					const void *data,
+					int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	const struct gre_base_hdr *hdr;
+	__be16 inner_proto;
+	int inner_nhoff;
+	bool ok;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*hdr)))
+		return false;
+	hdr = (const struct gre_base_hdr *)((const u8 *)data + nhoff);
+
+	/* flags field carries the GRE control flags *and* the 3-bit
+	 * version in its low bits. A zero word means "no flags set
+	 * AND version 0" — the byte-identical fast subset.
+	 */
+	if (hdr->flags != 0)
+		return false;
+
+	switch (hdr->protocol) {
+	case htons(ETH_P_IP):
+		inner_proto = htons(ETH_P_IP);
+		break;
+	case htons(ETH_P_IPV6):
+		inner_proto = htons(ETH_P_IPV6);
+		break;
+	default:
+		return false;
+	}
+
+	inner_nhoff = nhoff + (int)sizeof(*hdr);
+
+	if (inner_proto == htons(ETH_P_IP))
+		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
+					    target_container, data,
+					    inner_nhoff, hlen);
+	else
+		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
+					    target_container, data,
+					    inner_nhoff, hlen);
+
+	if (!ok)
+		return false;
+
+	/* Re-establish ENCAP after the inner pass zeroed key_control->flags. */
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->flags |= FLOW_DIS_ENCAPSULATION;
+	}
+	return true;
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -2792,6 +2874,13 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_do_static_key,
 	},
+	{
+		.procname	= "gre",
+		.data		= &flow_dissector_gre_key.key,
+		.maxlen		= sizeof(flow_dissector_gre_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
 };
 
 static int __init flow_dissector_sysctl_init(void)
-- 
2.54.0

[PATCH net-next v1 08/11] net: flow_dissector: per-shape counters + /proc/net/flow_dissector_stats

[Dave Seddon]

The opt-in fast-path (this series) gates each byte-identical shape behind
its own net.flow_dissector.* sysctl. Operators need to see how traffic is
actually composed to decide which gates to enable — and that same signal
lets a policy layer (a userspace agent, or the auto controller
proposed in a separate RFC thread) enable/disable the gates
automatically instead of hand-tuning knobs.

Add per-cpu counters, keyed by a small enum flow_dissector_shape covering
the byte-identical shapes (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre):

  occurrences[]  a shape handled by the SLOW path. Measured while its
                 gate is off, this approximates the eligible-fraction
                 signal: how much traffic that shape's fast body would
                 have handled. The slow counts sit at protocol
                 recognition points, which are a superset of fast-path
                 eligibility (a VLAN+ARP frame counts as vlan; an
                 options-bearing IPv4 counts as eth_ip), so eligible%
                 is an upper bound -- the exact hit rate is
                 fast_hits / (occurrences + fast_hits) once a gate is
                 on. Deliberately so: matching the fast path's full
                 eligibility conditions in the slow path would mean
                 re-implementing the fast path's checks there.
  fast_hits[]    the fast body actually ran for that shape (gate on).
  dissects       total dissects — the denominator.

Because the fast path returns before the slow-path classification, exactly
one of occurrences/fast_hits is incremented per shaped packet depending on
gate state, so (occurrences + fast_hits) / dissects is a gate-invariant
eligible fraction. Placement mirrors the two paths precisely: the slow
counts sit at each protocol's dissection point -- for eth_ip that is the
out: exit label, taken only on a top-level, non-encapsulated, TCP/UDP
terminal, so a packet counted as VLAN/IPIP/GRE is never also counted as
eth_ip; the fast counts sit at each shape's success terminal, so a
fall-through to the slow path is never double-counted.

Expose the summed counters read-only at /proc/net/flow_dissector_stats
(documented in Documentation/admin-guide/sysctl/net.rst),
one line per shape with occurrences, fast_hits, computed eligible%, and
the current gate state. Counters and the file are global (init_net),
matching the global gates; the file is visible in the init netns only,
so containers do not see it. Per-netns is a noted follow-up.

Dissects fully handled by an attached netns BPF flow dissector program
are deliberately not counted: the dissects increment sits after the BPF
hook's early return, so the file counts only dissects that reach the
in-kernel dissector. That keeps fast_hits/dissects an undiluted
hit-rate for the code these gates control — the signal an operator (or
the auto controller proposed in a separate RFC thread) acts on.

The counters are unconditional by design, not oversight: their whole
purpose is observing traffic composition while the gates are OFF, so a
"stats gate" would blind exactly the signal an operator (or the
auto-enable controller proposed separately) needs before enabling
anything.

Cost: one unconditional this_cpu_inc per dissect (the denominator)
plus one this_cpu_inc at the matched shape's classification point,
summed only on read. Measured on a CPU-bound pktgen soak this is
+0.74% dissector time (+0.44 sigma -- within run-to-run noise); see
the cover letter. The gates themselves are unaffected: an off gate
stays a not-taken static branch.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/admin-guide/sysctl/net.rst |  39 ++++
 include/net/flow_dissector.h             |  17 ++
 net/core/flow_dissector.c                | 236 ++++++++++++++++++++---
 3 files changed, 270 insertions(+), 22 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index dbc819740db0..3b55b2528ba2 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -597,6 +597,45 @@ additional 4-byte key field read and a write to
 
 Default: 0
 
+The ``/proc/net/flow_dissector_stats`` observability file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Read-only. Reports, per byte-identical shape, the number of packets the
+slow path handled (``occurrences``), the number the fast body handled
+(``fast_hits``), the resulting eligible fraction, and the current gate
+state, plus a total dissect count. Because the fast path returns before
+the slow-path classification, exactly one of the two counters is
+incremented per shaped packet depending on gate state, so
+``(occurrences + fast_hits) / dissects`` is a gate-invariant measure of
+how much traffic each shape's fast body would handle -- the signal an
+operator uses to decide whether a gate is worth enabling. The slow-path
+counts sit at protocol recognition points, a superset of fast-path
+eligibility (a VLAN+ARP frame counts as vlan), so the eligible fraction
+is an upper bound; the exact hit rate once a gate is on is
+``fast_hits / (occurrences + fast_hits)``. The counters
+are per-cpu and summed on read; the per-packet cost is one increment on
+the already-hot classification path.
+
+An example snapshot, from a host that mostly sees plain Eth + IP traffic
+with some VLAN, before any gate has been enabled::
+
+  # cat /proc/net/flow_dissector_stats
+  shape         occurrences        fast_hits  eligible%  gate
+  eth_ip            1866909                0      75.71   off
+  vlan               598847                0      24.28   off
+  qinq                    0                0       0.00   off
+  pppoe                   0                0       0.00   off
+  mpls                    0                0       0.00   off
+  ipip                    0                0       0.00   off
+  gre                     0                0       0.00   off
+  dissects: 2465765
+
+Read top-down: 76% of dissects are plain Eth + IP and 24% are
+VLAN-tagged, so ``eth_ip`` and ``vlan`` are the shapes worth enabling on
+this host; the other shapes are absent. Every gate is still off, so all
+the traffic sits under ``occurrences``; enabling a gate moves that
+shape's packets to ``fast_hits``.
+
 3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index ba7226a42d19..53b16f7306c7 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -436,6 +436,23 @@ extern struct static_key_false flow_dissector_mpls_key;
 extern struct static_key_false flow_dissector_ipip_key;
 extern struct static_key_false flow_dissector_gre_key;
 
+/* Per-shape counters for /proc/net/flow_dissector_stats.
+ * occurrences[]: shape seen by the slow path (the eligible-fraction
+ * signal while its gate is off); fast_hits[]: fast path ran.
+ * Byte-identical shapes only; the behaviour-changing descents are
+ * never counted or auto-managed.
+ */
+enum flow_dissector_shape {
+	FLOW_DISSECTOR_SHAPE_ETH_IP,
+	FLOW_DISSECTOR_SHAPE_VLAN,
+	FLOW_DISSECTOR_SHAPE_QINQ,
+	FLOW_DISSECTOR_SHAPE_PPPOE,
+	FLOW_DISSECTOR_SHAPE_MPLS,
+	FLOW_DISSECTOR_SHAPE_IPIP,
+	FLOW_DISSECTOR_SHAPE_GRE,
+	FLOW_DISSECTOR_SHAPE__MAX,
+};
+
 /* struct flow_keys_digest:
  *
  * This structure is used to hold a digest of the full flow keys. This is a
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index b79c80612dee..2a4b9b2af834 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -36,6 +36,10 @@
 #include <net/netfilter/nf_conntrack_labels.h>
 #endif
 #include <linux/bpf-netns.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/string_choices.h>
+#include <linux/math64.h>
 
 /* Per-shape fast-path gates, one static_branch per shape so operators
  * enable only what their deployment carries. All default off.
@@ -55,6 +59,35 @@ EXPORT_SYMBOL(flow_dissector_ipip_key);
 DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key);
 EXPORT_SYMBOL(flow_dissector_gre_key);
 
+/* Per-cpu, summed on read via /proc/net/flow_dissector_stats.
+ * occurrences[]: shape handled by the slow path (with the gate off,
+ * the shape's eligible-fraction signal); fast_hits[]: fast body ran;
+ * dissects: denominator.
+ */
+struct flow_dissector_stats {
+	u64 occurrences[FLOW_DISSECTOR_SHAPE__MAX];
+	u64 fast_hits[FLOW_DISSECTOR_SHAPE__MAX];
+	u64 dissects;
+};
+
+static DEFINE_PER_CPU(struct flow_dissector_stats, flow_dissector_pcpu_stats);
+
+static inline void flow_dissector_count_slow(enum flow_dissector_shape shape)
+{
+	this_cpu_inc(flow_dissector_pcpu_stats.occurrences[shape]);
+}
+
+static inline void flow_dissector_count_fast(enum flow_dissector_shape shape)
+{
+	this_cpu_inc(flow_dissector_pcpu_stats.fast_hits[shape]);
+}
+
+/* Counting map: eth_ip counts in the dispatcher (only when the leaf
+ * did not set ENCAP), vlan/qinq/pppoe/mpls in their helpers, ipip/gre
+ * in the inner-descent helpers. The slow path counts where it
+ * recognises each shape, gate on or off.
+ */
+
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
 
@@ -1354,6 +1387,7 @@ static bool flow_dissect_fast_vlan(const struct sk_buff *skb,
 	__be16 saved_tpid;
 	u16 tci_prio;
 	u16 tci_id;
+	bool ok;
 
 	if (vlan_depth >= 1 &&
 	    !static_branch_unlikely(&flow_dissector_qinq_key))
@@ -1403,22 +1437,35 @@ static bool flow_dissect_fast_vlan(const struct sk_buff *skb,
 
 	switch (inner_proto) {
 	case htons(ETH_P_IP):
-		return flow_dissect_fast_ipv4(skb, flow_dissector,
-					      target_container, data,
-					      nhoff, hlen);
+		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen);
+		break;
 	case htons(ETH_P_IPV6):
-		return flow_dissect_fast_ipv6(skb, flow_dissector,
-					      target_container, data,
-					      nhoff, hlen);
+		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen);
+		break;
 	case htons(ETH_P_8021Q):
 	case htons(ETH_P_8021AD):
-		return flow_dissect_fast_vlan(skb, flow_dissector,
-					      target_container, data,
-					      inner_proto, nhoff, hlen,
-					      vlan_depth + 1);
+		ok = flow_dissect_fast_vlan(skb, flow_dissector,
+					    target_container, data,
+					    inner_proto, nhoff, hlen,
+					    vlan_depth + 1);
+		break;
 	default:
 		return false;
 	}
+
+	/* Count only on full success -- a miss defers and the slow path
+	 * counts the occurrence. Depth 0 counts vlan, depth >= 1 qinq; a
+	 * double-tagged hit counts both, as the slow path does.
+	 */
+	if (ok)
+		flow_dissector_count_fast(vlan_depth == 0 ?
+					  FLOW_DISSECTOR_SHAPE_VLAN :
+					  FLOW_DISSECTOR_SHAPE_QINQ);
+	return ok;
 }
 
 /* PPPoE session (RFC 2516) + 2-byte PPP protocol: write
@@ -1439,6 +1486,7 @@ static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,
 	} *hdr;
 	__be16 inner_eth_proto;
 	u16 ppp_proto;
+	bool ok;
 
 	if (unlikely(hlen - nhoff < (int)sizeof(*hdr)))
 		return false;
@@ -1474,12 +1522,17 @@ static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,
 	nhoff += PPPOE_SES_HLEN;
 
 	if (inner_eth_proto == htons(ETH_P_IP))
-		return flow_dissect_fast_ipv4(skb, flow_dissector,
-					      target_container, data,
-					      nhoff, hlen);
-	return flow_dissect_fast_ipv6(skb, flow_dissector,
-				      target_container, data,
-				      nhoff, hlen);
+		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen);
+	else
+		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen);
+
+	if (ok)
+		flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_PPPOE);
+	return ok;
 }
 
 /* Single MPLS label (BoS=1): write FLOW_DISSECTOR_KEY_MPLS lse[0]
@@ -1546,6 +1599,7 @@ static bool flow_dissect_fast_mpls(const struct sk_buff *skb,
 		key_basic->ip_proto = 0;
 	}
 
+	flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_MPLS);
 	return true;
 }
 
@@ -1584,6 +1638,7 @@ static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
 							target_container);
 		key_control->flags |= FLOW_DIS_ENCAPSULATION;
 	}
+	flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_IPIP);
 	return true;
 }
 
@@ -1648,9 +1703,27 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
 							target_container);
 		key_control->flags |= FLOW_DIS_ENCAPSULATION;
 	}
+	flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_GRE);
 	return true;
 }
 
+/* Did a fast-path leaf stamp FLOW_DIS_ENCAPSULATION? The dispatcher
+ * counts eth_ip only for plain top-level IP; descents count under
+ * their own shape (UDP-tunnel descents not at all).
+ */
+static bool flow_dissect_fast_is_encap(struct flow_dissector *flow_dissector,
+				       void *target_container)
+{
+	struct flow_dissector_key_control *key_control;
+
+	if (!dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_CONTROL))
+		return false;
+	key_control = skb_flow_dissector_target(flow_dissector,
+						FLOW_DISSECTOR_KEY_CONTROL,
+						target_container);
+	return key_control->flags & FLOW_DIS_ENCAPSULATION;
+}
+
 /* Top-level dispatcher: eligibility check (only the two standard
  * dissectors and flag subset) + per-proto switch with per-shape
  * static_branch gating. Each case's branch is a forward not-taken JMP
@@ -1689,15 +1762,24 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 	case htons(ETH_P_IP):
 		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
 			return false;
-		return flow_dissect_fast_ipv4(skb, flow_dissector,
-					      target_container, data,
-					      nhoff, hlen);
+		if (!flow_dissect_fast_ipv4(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen))
+			return false;
+		/* Plain top-level eth+IP only; descents count under their own shape. */
+		if (!flow_dissect_fast_is_encap(flow_dissector, target_container))
+			flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP);
+		return true;
 	case htons(ETH_P_IPV6):
 		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
 			return false;
-		return flow_dissect_fast_ipv6(skb, flow_dissector,
-					      target_container, data,
-					      nhoff, hlen);
+		if (!flow_dissect_fast_ipv6(skb, flow_dissector,
+					    target_container, data,
+					    nhoff, hlen))
+			return false;
+		if (!flow_dissect_fast_is_encap(flow_dissector, target_container))
+			flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP);
+		return true;
 	case htons(ETH_P_PPP_SES):
 		if (!static_branch_unlikely(&flow_dissector_pppoe_key))
 			return false;
@@ -1751,6 +1833,8 @@ bool __skb_flow_dissect(const struct net *net,
 	bool mpls_el = false;
 	int mpls_lse = 0;
 	int num_hdrs = 0;
+	int nhoff_init = 0;
+	bool eth_ip_top = false;
 	u8 ip_proto = 0;
 	bool ret;
 
@@ -1859,6 +1943,8 @@ bool __skb_flow_dissect(const struct net *net,
 	/* Opt-in fast-path; flow_dissect_fast() does the eligibility check
 	 * and per-shape gating.
 	 */
+	this_cpu_inc(flow_dissector_pcpu_stats.dissects);
+
 	if (flow_dissect_fast(skb, flow_dissector, target_container,
 			      data, proto, nhoff, hlen, flags))
 		return true;
@@ -1892,6 +1978,8 @@ bool __skb_flow_dissect(const struct net *net,
 		key_num_of_vlans->num_of_vlans = 0;
 	}
 
+	nhoff_init = nhoff;
+
 proto_again:
 	fdret = FLOW_DISSECT_RET_CONTINUE;
 
@@ -1906,6 +1994,10 @@ bool __skb_flow_dissect(const struct net *net,
 			break;
 		}
 
+		/* Top-level eth+IPv4: eth_ip shape candidate (confirmed at out:). */
+		if (nhoff == nhoff_init)
+			eth_ip_top = true;
+
 		nhoff += iph->ihl * 4;
 
 		ip_proto = iph->protocol;
@@ -1954,6 +2046,9 @@ bool __skb_flow_dissect(const struct net *net,
 			break;
 		}
 
+		if (nhoff == nhoff_init)
+			eth_ip_top = true;
+
 		ip_proto = iph->nexthdr;
 		nhoff += sizeof(struct ipv6hdr);
 
@@ -2027,8 +2122,12 @@ bool __skb_flow_dissect(const struct net *net,
 
 		if (dissector_vlan == FLOW_DISSECTOR_KEY_MAX) {
 			dissector_vlan = FLOW_DISSECTOR_KEY_VLAN;
+			/* First (outer) tag: the vlan fast-path shape. */
+			flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_VLAN);
 		} else if (dissector_vlan == FLOW_DISSECTOR_KEY_VLAN) {
 			dissector_vlan = FLOW_DISSECTOR_KEY_CVLAN;
+			/* Second tag: the qinq (double-tag) fast-path shape. */
+			flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_QINQ);
 		} else {
 			fdret = FLOW_DISSECT_RET_PROTO_AGAIN;
 			break;
@@ -2074,6 +2173,8 @@ bool __skb_flow_dissect(const struct net *net,
 			break;
 		}
 
+		flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_PPPOE);
+
 		/* PFC (compressed 1-byte protocol) frames are not processed.
 		 * A compressed protocol field has the least significant bit of
 		 * the most significant octet set, which will fail the following
@@ -2138,6 +2239,7 @@ bool __skb_flow_dissect(const struct net *net,
 
 	case htons(ETH_P_MPLS_UC):
 	case htons(ETH_P_MPLS_MC):
+		flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_MPLS);
 		fdret = __skb_flow_dissect_mpls(skb, flow_dissector,
 						target_container, data,
 						nhoff, hlen, mpls_lse,
@@ -2235,6 +2337,7 @@ bool __skb_flow_dissect(const struct net *net,
 			fdret = FLOW_DISSECT_RET_OUT_GOOD;
 			break;
 		}
+		flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_GRE);
 
 		fdret = __skb_flow_dissect_gre(skb, key_control, flow_dissector,
 					       target_container, data,
@@ -2297,6 +2400,10 @@ bool __skb_flow_dissect(const struct net *net,
 			fdret = FLOW_DISSECT_RET_OUT_GOOD;
 			break;
 		}
+		/* Counted below the STOP check: a stop-flagged dissect is
+		 * one the fast path could never have handled.
+		 */
+		flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_IPIP);
 
 		proto = htons(ETH_P_IP);
 
@@ -2388,6 +2495,15 @@ bool __skb_flow_dissect(const struct net *net,
 	key_basic->n_proto = proto;
 	key_basic->ip_proto = ip_proto;
 
+	/* eth_ip shape: top-level eth+IP, TCP/UDP, no encap -- counted here
+	 * because the fast path returns earlier; with the gate off this is
+	 * the shape's eligible-fraction signal.
+	 */
+	if (ret && eth_ip_top &&
+	    !(key_control->flags & FLOW_DIS_ENCAPSULATION) &&
+	    (ip_proto == IPPROTO_TCP || ip_proto == IPPROTO_UDP))
+		flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_ETH_IP);
+
 	return ret;
 
 out_bad:
@@ -2883,11 +2999,87 @@ static struct ctl_table flow_dissector_sysctl_table[] = {
 	},
 };
 
+static const char * const flow_dissector_shape_names[FLOW_DISSECTOR_SHAPE__MAX] = {
+	[FLOW_DISSECTOR_SHAPE_ETH_IP] = "eth_ip",
+	[FLOW_DISSECTOR_SHAPE_VLAN]   = "vlan",
+	[FLOW_DISSECTOR_SHAPE_QINQ]   = "qinq",
+	[FLOW_DISSECTOR_SHAPE_PPPOE]  = "pppoe",
+	[FLOW_DISSECTOR_SHAPE_MPLS]   = "mpls",
+	[FLOW_DISSECTOR_SHAPE_IPIP]   = "ipip",
+	[FLOW_DISSECTOR_SHAPE_GRE]    = "gre",
+};
+
+static bool flow_dissector_shape_gate(enum flow_dissector_shape shape)
+{
+	switch (shape) {
+	case FLOW_DISSECTOR_SHAPE_ETH_IP:
+		return static_key_enabled(&flow_dissector_eth_ip_key);
+	case FLOW_DISSECTOR_SHAPE_VLAN:
+		return static_key_enabled(&flow_dissector_vlan_key);
+	case FLOW_DISSECTOR_SHAPE_QINQ:
+		return static_key_enabled(&flow_dissector_qinq_key);
+	case FLOW_DISSECTOR_SHAPE_PPPOE:
+		return static_key_enabled(&flow_dissector_pppoe_key);
+	case FLOW_DISSECTOR_SHAPE_MPLS:
+		return static_key_enabled(&flow_dissector_mpls_key);
+	case FLOW_DISSECTOR_SHAPE_IPIP:
+		return static_key_enabled(&flow_dissector_ipip_key);
+	case FLOW_DISSECTOR_SHAPE_GRE:
+		return static_key_enabled(&flow_dissector_gre_key);
+	default:
+		return false;
+	}
+}
+
+/* /proc/net/flow_dissector_stats: per-shape occurrences (slow path),
+ * fast_hits, and eligible% = (occurrences + fast_hits) / dissects --
+ * the signal a policy layer thresholds against a per-shape
+ * break-even.
+ */
+static int flow_dissector_stats_show(struct seq_file *seq, void *v)
+{
+	u64 occ[FLOW_DISSECTOR_SHAPE__MAX] = {};
+	u64 fast[FLOW_DISSECTOR_SHAPE__MAX] = {};
+	u64 dissects = 0;
+	int cpu, i;
+
+	for_each_possible_cpu(cpu) {
+		const struct flow_dissector_stats *s =
+			per_cpu_ptr(&flow_dissector_pcpu_stats, cpu);
+
+		for (i = 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++) {
+			occ[i]  += READ_ONCE(s->occurrences[i]);
+			fast[i] += READ_ONCE(s->fast_hits[i]);
+		}
+		dissects += READ_ONCE(s->dissects);
+	}
+
+	seq_printf(seq, "%-8s %16s %16s %10s %5s\n",
+		   "shape", "occurrences", "fast_hits", "eligible%", "gate");
+	for (i = 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++) {
+		u64 total = occ[i] + fast[i];
+		u64 pct_x100 = dissects ? div64_u64(total * 10000, dissects) : 0;
+
+		seq_printf(seq, "%-8s %16llu %16llu %7llu.%02llu %5s\n",
+			   flow_dissector_shape_names[i], occ[i], fast[i],
+			   pct_x100 / 100, pct_x100 % 100,
+			   str_on_off(flow_dissector_shape_gate(i)));
+	}
+	seq_printf(seq, "dissects: %llu\n", dissects);
+	return 0;
+}
+
 static int __init flow_dissector_sysctl_init(void)
 {
 	if (!register_net_sysctl(&init_net, "net/flow_dissector",
 				 flow_dissector_sysctl_table))
 		return -ENOMEM;
+
+	/* Global (init_net) counters — matches the global gates; per-netns
+	 * is a noted follow-up. A missing proc file is not fatal.
+	 */
+	proc_create_single("flow_dissector_stats", 0444, init_net.proc_net,
+			   flow_dissector_stats_show);
 	return 0;
 }
 late_initcall(flow_dissector_sysctl_init);
-- 
2.54.0

[PATCH net-next v1 09/11] net: flow_dissector: bound fast-path tunnel recursion

[Dave Seddon]

The opt-in fast path descends into IP-in-IP and GRE tunnels by
recursion: flow_dissect_fast_ipv4()/ipv6() call
flow_dissect_fast_ipip_inner()/gre_inner(), which parse the inner IP
header and call back into flow_dissect_fast_ipv4()/ipv6(). Unlike the
slow path, which caps total header descents at MAX_FLOW_DISSECT_HDRS
via skb_flow_dissect_allowed(), the fast helpers had no such bound.

A crafted frame with a deeply nested tunnel chain
(IP-in-IP-in-IP-in-..., or GRE) therefore drives one C stack frame per
~20 bytes of linear header. hlen is skb_headlen(), so a large,
GRO-coalesced or locally crafted skb yields hundreds to thousands of
levels and can exhaust the kernel stack. The inner helpers also do
work after the recursive call (they stamp FLOW_DIS_ENCAPSULATION), so
the frames are not tail-call eliminable. It is additionally an output
divergence: past 15 headers the slow path stops and reports the
15th-level keys, while the fast path would recurse to the innermost,
breaking the byte-identical contract.

Thread a num_hdrs counter through the tunnel-reachable helpers and,
at each tunnel descent, mirror skb_flow_dissect_allowed(): once the
count exceeds MAX_FLOW_DISSECT_HDRS the helper returns false and the
dissect falls back to the slow path (which caps at the same limit).
For chains within the limit both paths descend fully and agree; beyond
it the fast path defers to the slow path's result, so the output stays
identical and the recursion is bounded.

The initial count at each fast-path entry is set at or above the
number of headers the slow path has already consumed (outer VLAN tags,
PPPoE, the outer IP), so the fast path never descends past the point
the slow path would have capped.

The added KUnit equivalence test's deep-nest case exercises IPIP and
GRE chains across the cap boundary; with the bound removed it fails at
16 levels, confirming it guards this fix.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 net/core/flow_dissector.c | 61 ++++++++++++++++++++++++---------------
 1 file changed, 37 insertions(+), 24 deletions(-)

diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 2a4b9b2af834..49b256ce5c1e 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -91,28 +91,31 @@ static inline void flow_dissector_count_fast(enum flow_dissector_shape shape)
 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
 #define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
 
-/* Fast-path helper forward declarations. */
+/* @num_hdrs: protocol headers consumed so far (Ethernet = 1, each VLAN
+ * tag, each outer IP header). Tunnel descents increment it and defer
+ * past MAX_FLOW_DISSECT_HDRS, so both paths stop at the same depth.
+ */
 static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
 				   struct flow_dissector *flow_dissector,
 				   void *target_container,
 				   const void *data,
-				   int nhoff, int hlen);
+				   int nhoff, int hlen, int num_hdrs);
 static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 				   struct flow_dissector *flow_dissector,
 				   void *target_container,
 				   const void *data,
-				   int nhoff, int hlen);
+				   int nhoff, int hlen, int num_hdrs);
 static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
 					 struct flow_dissector *flow_dissector,
 					 void *target_container,
 					 const void *data,
 					 __be16 inner_eth_proto,
-					 int inner_nhoff, int hlen);
+					 int inner_nhoff, int hlen, int num_hdrs);
 static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
 					struct flow_dissector *flow_dissector,
 					void *target_container,
 					const void *data,
-					int nhoff, int hlen);
+					int nhoff, int hlen, int num_hdrs);
 
 /* One of the two dissectors the fast-path eligibility check admits;
  * defined here so flow_dissect_fast() below can reference it (its keys
@@ -1141,7 +1144,7 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
 				   struct flow_dissector *flow_dissector,
 				   void *target_container,
 				   const void *data,
-				   int nhoff, int hlen)
+				   int nhoff, int hlen, int num_hdrs)
 {
 	struct flow_dissector_key_control *key_control;
 	struct flow_dissector_key_addrs *key_addrs;
@@ -1172,18 +1175,18 @@ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
 				return flow_dissect_fast_ipip_inner(skb,
 						flow_dissector, target_container,
 						data, htons(ETH_P_IP),
-						nhoff + (int)sizeof(*iph), hlen);
+						nhoff + (int)sizeof(*iph), hlen, num_hdrs);
 			if (iph->protocol == IPPROTO_IPV6)
 				return flow_dissect_fast_ipip_inner(skb,
 						flow_dissector, target_container,
 						data, htons(ETH_P_IPV6),
-						nhoff + (int)sizeof(*iph), hlen);
+						nhoff + (int)sizeof(*iph), hlen, num_hdrs);
 		}
 		if (static_branch_unlikely(&flow_dissector_gre_key) &&
 		    iph->protocol == IPPROTO_GRE)
 			return flow_dissect_fast_gre_inner(skb, flow_dissector,
 					target_container, data,
-					nhoff + (int)sizeof(*iph), hlen);
+					nhoff + (int)sizeof(*iph), hlen, num_hdrs);
 		return false;
 	}
 
@@ -1241,7 +1244,7 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 				   struct flow_dissector *flow_dissector,
 				   void *target_container,
 				   const void *data,
-				   int nhoff, int hlen)
+				   int nhoff, int hlen, int num_hdrs)
 {
 	struct flow_dissector_key_control *key_control;
 	struct flow_dissector_key_addrs *key_addrs;
@@ -1317,10 +1320,10 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
 					data,
 					iph->nexthdr == IPPROTO_IPIP ?
 					htons(ETH_P_IP) : htons(ETH_P_IPV6),
-					nhoff + (int)sizeof(*iph), hlen);
+					nhoff + (int)sizeof(*iph), hlen, num_hdrs);
 		return flow_dissect_fast_gre_inner(skb, flow_dissector,
 				target_container, data,
-				nhoff + (int)sizeof(*iph), hlen);
+				nhoff + (int)sizeof(*iph), hlen, num_hdrs);
 	}
 
 	thoff = nhoff + (int)sizeof(*iph);
@@ -1439,12 +1442,12 @@ static bool flow_dissect_fast_vlan(const struct sk_buff *skb,
 	case htons(ETH_P_IP):
 		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen);
+					    nhoff, hlen, vlan_depth + 2);
 		break;
 	case htons(ETH_P_IPV6):
 		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen);
+					    nhoff, hlen, vlan_depth + 2);
 		break;
 	case htons(ETH_P_8021Q):
 	case htons(ETH_P_8021AD):
@@ -1524,11 +1527,11 @@ static bool flow_dissect_fast_pppoe(const struct sk_buff *skb,
 	if (inner_eth_proto == htons(ETH_P_IP))
 		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen);
+					    nhoff, hlen, 2);
 	else
 		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen);
+					    nhoff, hlen, 2);
 
 	if (ok)
 		flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_PPPOE);
@@ -1612,19 +1615,25 @@ static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb,
 					 void *target_container,
 					 const void *data,
 					 __be16 inner_eth_proto,
-					 int inner_nhoff, int hlen)
+					 int inner_nhoff, int hlen, int num_hdrs)
 {
 	struct flow_dissector_key_control *key_control;
 	bool ok;
 
+	/* Mirror the slow path's MAX_FLOW_DISSECT_HDRS budget; past the cap
+	 * defer, so both paths stop at the same depth.
+	 */
+	if (++num_hdrs > MAX_FLOW_DISSECT_HDRS)
+		return false;
+
 	if (inner_eth_proto == htons(ETH_P_IP))
 		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
 					    target_container, data,
-					    inner_nhoff, hlen);
+					    inner_nhoff, hlen, num_hdrs);
 	else if (inner_eth_proto == htons(ETH_P_IPV6))
 		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
 					    target_container, data,
-					    inner_nhoff, hlen);
+					    inner_nhoff, hlen, num_hdrs);
 	else
 		return false;
 
@@ -1651,7 +1660,7 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
 					struct flow_dissector *flow_dissector,
 					void *target_container,
 					const void *data,
-					int nhoff, int hlen)
+					int nhoff, int hlen, int num_hdrs)
 {
 	struct flow_dissector_key_control *key_control;
 	const struct gre_base_hdr *hdr;
@@ -1659,6 +1668,10 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
 	int inner_nhoff;
 	bool ok;
 
+	/* Same MAX_FLOW_DISSECT_HDRS budget as flow_dissect_fast_ipip_inner(). */
+	if (++num_hdrs > MAX_FLOW_DISSECT_HDRS)
+		return false;
+
 	if (unlikely(hlen - nhoff < (int)sizeof(*hdr)))
 		return false;
 	hdr = (const struct gre_base_hdr *)((const u8 *)data + nhoff);
@@ -1686,11 +1699,11 @@ static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb,
 	if (inner_proto == htons(ETH_P_IP))
 		ok = flow_dissect_fast_ipv4(skb, flow_dissector,
 					    target_container, data,
-					    inner_nhoff, hlen);
+					    inner_nhoff, hlen, num_hdrs);
 	else
 		ok = flow_dissect_fast_ipv6(skb, flow_dissector,
 					    target_container, data,
-					    inner_nhoff, hlen);
+					    inner_nhoff, hlen, num_hdrs);
 
 	if (!ok)
 		return false;
@@ -1764,7 +1777,7 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 			return false;
 		if (!flow_dissect_fast_ipv4(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen))
+					    nhoff, hlen, 1))
 			return false;
 		/* Plain top-level eth+IP only; descents count under their own shape. */
 		if (!flow_dissect_fast_is_encap(flow_dissector, target_container))
@@ -1775,7 +1788,7 @@ static bool flow_dissect_fast(const struct sk_buff *skb,
 			return false;
 		if (!flow_dissect_fast_ipv6(skb, flow_dissector,
 					    target_container, data,
-					    nhoff, hlen))
+					    nhoff, hlen, 1))
 			return false;
 		if (!flow_dissect_fast_is_encap(flow_dissector, target_container))
 			flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP);
-- 
2.54.0

[PATCH net-next v1 10/11] net: flow_dissector: add KUnit fast/slow path equivalence tests

[Dave Seddon]

The per-shape fast paths contract to be byte-identical with the slow
path: for any packet, either struct flow_keys is filled with exactly
the bytes the slow path would have written, or the fast path returns
false and the slow path runs. Until now that contract was only
verified out of tree (a userspace A/B harness compiling the dissector
both ways). Enforce it in-tree so any future divergence between the
two implementations is a failing test, not a silent behaviour change.

Each corpus packet is dissected twice into zeroed containers - once
with every fast-path gate disabled and once with every gate enabled -
and the return values and full struct flow_keys must match exactly.
(A fast-path miss cannot leak partial writes to callers:
flow_dissect_fast() is only invoked from __skb_flow_dissect(), which
falls through to the full slow-path walk on the same container
whenever the fast path returns false.) The corpus covers the eligible
shapes (eth+IPv4/IPv6 x TCP/UDP, VLAN, QinQ, PPPoE session, single
MPLS, IPIP/4in6/6in4, plain GRE), deliberate fast-path misses (IP
options, fragments with and without FLOW_DISSECTOR_F_PARSE_1ST_FRAG,
IPv6 extension headers and non-zero flow labels, triple VLAN,
multi-label MPLS, non-IP PPP, flagged GRE, ICMP), a truncation sweep
cutting every packet at every byte boundary, and skb-mode cases for
the hardware-stripped VLAN tag entry conditions raw-data mode cannot
reach.

Every equivalence case runs both with and without
FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- the flag skb_get_hash()
passes on every dissect -- against both eligible dissectors, and the
gates-off test's positive control asserts the fast path actually
hits for that flagship flag shape (it caught the dispatcher
rejecting the flag outright, which had made the fast path
unreachable from RPS/RFS/fq/cake).

Beyond byte equality, every check also asserts
flow_hash_from_keys() equality (the hash is what consumers act on).
A gates-off pass over the whole corpus proves the fast path never
runs when disabled (observed via the per-shape counters), and a
seeded, reproducible fuzzer mutates corpus packets and re-checks
equivalence every iteration, as a guard for divergences the
hand-written corpus does not name.

The file-static symmetric dissector and the summed fast-hit counters
are exposed to the suite through accessors compiled only under
CONFIG_FLOW_DISSECTOR_KUNIT_TEST.

Run with:

  ./tools/testing/kunit/kunit.py run --arch=x86_64 \
      --kconfig_add CONFIG_NET=y \
      --kconfig_add CONFIG_FLOW_DISSECTOR_KUNIT_TEST=y \
      flow_dissector_fastpath

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 include/net/flow_dissector.h   |    6 +
 net/Kconfig                    |   12 +
 net/core/Makefile              |    1 +
 net/core/flow_dissector.c      |   30 +
 net/core/flow_dissector_test.c | 1053 ++++++++++++++++++++++++++++++++
 5 files changed, 1102 insertions(+)
 create mode 100644 net/core/flow_dissector_test.c

diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index 53b16f7306c7..b92aef8039f8 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -453,6 +453,12 @@ enum flow_dissector_shape {
 	FLOW_DISSECTOR_SHAPE__MAX,
 };
 
+#if IS_ENABLED(CONFIG_FLOW_DISSECTOR_KUNIT_TEST)
+/* Test-only accessors, defined in flow_dissector.c. */
+struct flow_dissector *flow_keys_dissector_symmetric_kunit(void);
+u64 flow_dissector_fast_hits_kunit(void);
+#endif
+
 /* struct flow_keys_digest:
  *
  * This structure is used to hold a digest of the full flow keys. This is a
diff --git a/net/Kconfig b/net/Kconfig
index e38477393551..d0ce8fc34429 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -540,4 +540,16 @@ config NET_TEST
 
 	  If unsure, say N.
 
+config FLOW_DISSECTOR_KUNIT_TEST
+	tristate "KUnit tests for flow dissector fast/slow path equivalence" if !KUNIT_ALL_TESTS
+	depends on KUNIT
+	default KUNIT_ALL_TESTS
+	help
+	  KUnit tests asserting the flow dissector's opt-in per-shape
+	  fast paths produce byte-identical struct flow_keys output to
+	  the slow path for every corpus packet, with the gates on and
+	  off.
+
+	  If unsure, say N.
+
 endif   # if NET
diff --git a/net/core/Makefile b/net/core/Makefile
index b3fdcb4e355f..b27c69580d2c 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -46,6 +46,7 @@ obj-$(CONFIG_BPF_SYSCALL) += sock_map.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o
 obj-$(CONFIG_OF)	+= of_net.o
 obj-$(CONFIG_NET_TEST) += net_test.o
+obj-$(CONFIG_FLOW_DISSECTOR_KUNIT_TEST) += flow_dissector_test.o
 obj-$(CONFIG_NET_DEVMEM) += devmem.o
 obj-$(CONFIG_DEBUG_NET) += lock_debug.o
 obj-$(CONFIG_FAIL_SKB_REALLOC) += skb_fault_injection.o
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 49b256ce5c1e..aaed4473c211 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -2909,6 +2909,36 @@ EXPORT_SYMBOL(flow_keys_dissector);
 struct flow_dissector flow_keys_basic_dissector __read_mostly;
 EXPORT_SYMBOL(flow_keys_basic_dissector);
 
+#if IS_ENABLED(CONFIG_FLOW_DISSECTOR_KUNIT_TEST)
+/* Test-only accessor: the symmetric dissector is file-static but is
+ * the second dissector the fast-path eligibility gate admits.
+ */
+struct flow_dissector *flow_keys_dissector_symmetric_kunit(void)
+{
+	return &flow_keys_dissector_symmetric;
+}
+EXPORT_SYMBOL_GPL(flow_keys_dissector_symmetric_kunit);
+
+/* Test-only accessor: summed fast-path hits, the observable for the
+ * suite's gates-off negative.
+ */
+u64 flow_dissector_fast_hits_kunit(void)
+{
+	u64 sum = 0;
+	int cpu, i;
+
+	for_each_possible_cpu(cpu) {
+		const struct flow_dissector_stats *s =
+			per_cpu_ptr(&flow_dissector_pcpu_stats, cpu);
+
+		for (i = 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++)
+			sum += READ_ONCE(s->fast_hits[i]);
+	}
+	return sum;
+}
+EXPORT_SYMBOL_GPL(flow_dissector_fast_hits_kunit);
+#endif
+
 static int __init init_default_flow_dissectors(void)
 {
 	skb_flow_dissector_init(&flow_keys_dissector,
diff --git a/net/core/flow_dissector_test.c b/net/core/flow_dissector_test.c
new file mode 100644
index 000000000000..e0014affda7f
--- /dev/null
+++ b/net/core/flow_dissector_test.c
@@ -0,0 +1,1053 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * KUnit equivalence tests for the flow dissector opt-in fast paths.
+ *
+ * Every fast-path shape contracts to be byte-identical with the slow
+ * path. Each corpus packet is dissected twice into zeroed containers
+ * -- gates off, then gates on -- and the return values, the full
+ * struct flow_keys and flow_hash_from_keys() must match exactly, for
+ * both eligible dissectors (flow_keys_dissector and the file-static
+ * symmetric one). (A fast-path miss cannot leak partial writes to
+ * callers: flow_dissect_fast() is only invoked from
+ * __skb_flow_dissect(), which continues into the full slow-path walk
+ * on the same container whenever the fast path returns false.)
+ */
+
+#include <kunit/test.h>
+
+#include <linux/etherdevice.h>
+#include <linux/if_ether.h>
+#include <linux/if_pppox.h>
+#include <linux/if_tunnel.h>
+#include <linux/if_vlan.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/mpls.h>
+#include <linux/netdevice.h>
+#include <linux/ppp_defs.h>
+#include <linux/prandom.h>
+#include <linux/skbuff.h>
+#include <linux/udp.h>
+#include <net/flow_dissector.h>
+#include <net/gre.h>
+#include <net/ip.h>
+#include <net/net_namespace.h>
+
+#define FD_TEST_BUF_LEN 512
+#define FD_FUZZ_ITERS 4000
+
+/* The byte-identical fast-path gates. */
+static struct static_key_false * const fd_fast_gates[] = {
+	&flow_dissector_eth_ip_key,
+	&flow_dissector_vlan_key,
+	&flow_dissector_qinq_key,
+	&flow_dissector_pppoe_key,
+	&flow_dissector_mpls_key,
+	&flow_dissector_ipip_key,
+	&flow_dissector_gre_key,
+};
+
+static void fd_fast_gates_set(bool on)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fd_fast_gates); i++) {
+		if (on)
+			static_branch_enable(fd_fast_gates[i]);
+		else
+			static_branch_disable(fd_fast_gates[i]);
+	}
+}
+
+/* --- packet builders (network-layer payloads; __skb_flow_dissect is
+ * called in raw-data mode with the L3 proto passed explicitly, so no
+ * Ethernet header is included) ---
+ */
+
+static int put_ports(u8 *buf)
+{
+	__be16 *p = (__be16 *)buf;
+
+	p[0] = htons(0x1234);
+	p[1] = htons(0x5678);
+	return 4;
+}
+
+static int put_ipv4(u8 *buf, u8 ip_proto, u8 ihl, __be16 frag_off)
+{
+	struct iphdr *ip = (struct iphdr *)buf;
+	int hdrlen = ihl * 4;
+
+	memset(ip, 0, hdrlen);
+	ip->version = 4;
+	ip->ihl = ihl;
+	ip->ttl = 64;
+	ip->tot_len = htons(hdrlen + 4);
+	ip->frag_off = frag_off;
+	ip->protocol = ip_proto;
+	ip->saddr = htonl(0xc0a80101);	/* 192.168.1.1 */
+	ip->daddr = htonl(0xc0a80202);	/* 192.168.2.2 */
+	return hdrlen;
+}
+
+static int put_ipv6(u8 *buf, u8 nexthdr, u32 flow_label)
+{
+	struct ipv6hdr *ip6 = (struct ipv6hdr *)buf;
+
+	memset(ip6, 0, sizeof(*ip6));
+	ip6->version = 6;
+	ip6->flow_lbl[0] = (flow_label >> 16) & 0x0f;
+	ip6->flow_lbl[1] = (flow_label >> 8) & 0xff;
+	ip6->flow_lbl[2] = flow_label & 0xff;
+	ip6->payload_len = htons(4);
+	ip6->nexthdr = nexthdr;
+	ip6->hop_limit = 64;
+	ip6->saddr.s6_addr32[0] = htonl(0x20010db8);
+	ip6->saddr.s6_addr32[3] = htonl(1);
+	ip6->daddr.s6_addr32[0] = htonl(0x20010db8);
+	ip6->daddr.s6_addr32[3] = htonl(2);
+	return sizeof(*ip6);
+}
+
+static int put_vlan(u8 *buf, u16 tci, __be16 inner_proto)
+{
+	struct vlan_hdr *vlan = (struct vlan_hdr *)buf;
+
+	vlan->h_vlan_TCI = htons(tci);
+	vlan->h_vlan_encapsulated_proto = inner_proto;
+	return sizeof(*vlan);
+}
+
+static int put_pppoe(u8 *buf, u16 ppp_proto, u16 payload_len)
+{
+	struct pppoe_hdr *hdr = (struct pppoe_hdr *)buf;
+	__be16 *proto = (__be16 *)(buf + sizeof(*hdr));
+
+	memset(hdr, 0, sizeof(*hdr));
+	hdr->ver = 1;
+	hdr->type = 1;
+	hdr->code = 0;
+	hdr->sid = htons(0x42);
+	hdr->length = htons(payload_len + 2);
+	*proto = htons(ppp_proto);
+	return sizeof(*hdr) + 2;
+}
+
+static int put_mpls(u8 *buf, u32 label, u32 bos)
+{
+	struct mpls_label *lse = (struct mpls_label *)buf;
+
+	lse->entry = htonl((label << MPLS_LS_LABEL_SHIFT) |
+			   (bos << MPLS_LS_S_SHIFT) |
+			   (64 << MPLS_LS_TTL_SHIFT));
+	return sizeof(*lse);
+}
+
+static int put_gre(u8 *buf, __be16 flags, __be16 protocol)
+{
+	struct gre_base_hdr *gre = (struct gre_base_hdr *)buf;
+
+	gre->flags = flags;
+	gre->protocol = protocol;
+	return sizeof(*gre);
+}
+
+/* --- composite builders, one per corpus entry --- */
+
+static int build_ipv4_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 5, 0);
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv4_udp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_UDP, 5, 0);
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv6_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_TCP, 0);
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv6_udp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_UDP, 0);
+
+	return n + put_ports(buf + n);
+}
+
+static int build_vlan_ipv4_tcp(u8 *buf)
+{
+	int n = put_vlan(buf, 100, htons(ETH_P_IP));
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_vlan_ipv6_udp(u8 *buf)
+{
+	int n = put_vlan(buf, 100, htons(ETH_P_IPV6));
+
+	n += put_ipv6(buf + n, IPPROTO_UDP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_qinq_ipv4_tcp(u8 *buf)
+{
+	int n = put_vlan(buf, 100, htons(ETH_P_8021Q));
+
+	n += put_vlan(buf + n, 200, htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_vlan3_ipv4_tcp(u8 *buf)
+{
+	int n = put_vlan(buf, 100, htons(ETH_P_8021Q));
+
+	n += put_vlan(buf + n, 200, htons(ETH_P_8021Q));
+	n += put_vlan(buf + n, 300, htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_pppoe_ipv4_tcp(u8 *buf)
+{
+	int n = put_pppoe(buf, PPP_IP, 20 + 4);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_pppoe_ipv6_tcp(u8 *buf)
+{
+	int n = put_pppoe(buf, PPP_IPV6, 40 + 4);
+
+	n += put_ipv6(buf + n, IPPROTO_TCP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_pppoe_lcp(u8 *buf)
+{
+	int n = put_pppoe(buf, PPP_LCP, 4);
+	u8 lcp[4] = { 0x01, 0x01, 0x00, 0x04 };
+
+	memcpy(buf + n, lcp, sizeof(lcp));
+	return n + sizeof(lcp);
+}
+
+static int build_mpls_ipv4(u8 *buf)
+{
+	int n = put_mpls(buf, 1000, 1);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_mpls_multi(u8 *buf)
+{
+	int n = put_mpls(buf, 1000, 0);	/* S=0: not bottom of stack */
+
+	n += put_mpls(buf + n, 2000, 1);
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_ipip(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_IPIP, 5, 0);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_6in4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_IPV6, 5, 0);
+
+	n += put_ipv6(buf + n, IPPROTO_TCP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_4in6(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_IPIP, 0);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_4in6_flowlabel(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_IPIP, 0xbeef);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_6in6(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_IPV6, 0);
+
+	n += put_ipv6(buf + n, IPPROTO_TCP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, 0, htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_ipv6(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, 0, htons(ETH_P_IPV6));
+	n += put_ipv6(buf + n, IPPROTO_TCP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre6_ipv6(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_GRE, 0);
+
+	n += put_gre(buf + n, 0, htons(ETH_P_IPV6));
+	n += put_ipv6(buf + n, IPPROTO_TCP, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre6_ipv4(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_GRE, 0);
+
+	n += put_gre(buf + n, 0, htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+/* --- corner-case builders --- */
+
+/* GRE variants that must defer (any non-zero flags/version word). */
+static int build_gre_seq_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, GRE_SEQ, htons(ETH_P_IP));
+	memset(buf + n, 0, 4);		/* sequence number */
+	n += 4;
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_key_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, GRE_KEY, htons(ETH_P_IP));
+	memset(buf + n, 0, 4);		/* key */
+	n += 4;
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_version1(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	/* GRE version 1 (PPTP) lives in the low bits of the flags word. */
+	n += put_gre(buf + n, htons(0x0001), htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_teb(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	/* Transparent Ethernet bridging inner: fast path defers. */
+	n += put_gre(buf + n, 0, htons(ETH_P_TEB));
+	return n + put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+}
+
+/* An IPv6 extension header (8 bytes, hdrlen field 0). */
+static int put_ext_hdr(u8 *buf, u8 next_proto)
+{
+	memset(buf, 0, 8);
+	buf[0] = next_proto;
+	buf[1] = 0;			/* hdr ext len: (0 + 1) * 8 = 8 bytes */
+	return 8;
+}
+
+static int build_ipv6_hopopts_dest_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_HOPOPTS, 0);
+
+	n += put_ext_hdr(buf + n, IPPROTO_DSTOPTS);
+	n += put_ext_hdr(buf + n, IPPROTO_TCP);
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv6_frag_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_FRAGMENT, 0);
+
+	memset(buf + n, 0, 8);
+	buf[n] = IPPROTO_TCP;		/* fragment header, offset 0 */
+	*(__be32 *)(buf + n + 4) = htonl(0x12345678);
+	n += 8;
+	return n + put_ports(buf + n);
+}
+
+/* TPID ordering not covered elsewhere: 8021Q outer, 8021AD inner. */
+static int build_vlan_q_then_ad_ipv4(u8 *buf)
+{
+	int n = put_vlan(buf, 100, htons(ETH_P_8021AD));
+
+	n += put_vlan(buf + n, 200, htons(ETH_P_IP));
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+/* IPv4 with a larger options block (IHL 8 => 12 bytes of options). */
+static int build_ipv4_options12_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 8, 0);
+
+	return n + put_ports(buf + n);
+}
+
+/* PPPoE with a PFC-compressed (odd MSB) PPP protocol field: defers. */
+static int build_pppoe_pfc(u8 *buf)
+{
+	int n = put_pppoe(buf, 0x2100, 20 + 4);
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+/* Mid-stream fragment: non-zero offset AND MF still set. */
+static int build_ipv4_frag_mid_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 5, htons(IP_MF | 5));
+
+	return n + put_ports(buf + n);
+}
+
+/* GRE optional-field combinations (order: csum+reserved, key, seq). */
+static int build_gre_csum_key_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, GRE_CSUM | GRE_KEY, htons(ETH_P_IP));
+	memset(buf + n, 0, 8);		/* checksum + reserved, key */
+	n += 8;
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_csum_key_seq_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, GRE_CSUM | GRE_KEY | GRE_SEQ, htons(ETH_P_IP));
+	memset(buf + n, 0, 12);		/* checksum + reserved, key, seq */
+	n += 12;
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+/* Priority-tagged frame (802.1Q VID 0, only PCP bits in the TCI). */
+static int build_vlan_prio_ipv4_tcp(u8 *buf)
+{
+	int n = put_vlan(buf, 5 << VLAN_PRIO_SHIFT, htons(ETH_P_IP));
+
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+/* IPv6 routing extension header before TCP. */
+static int build_ipv6_routing_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_ROUTING, 0);
+
+	n += put_ext_hdr(buf + n, IPPROTO_TCP);
+	return n + put_ports(buf + n);
+}
+
+/* Deeply nested IPIP / GRE chains for the recursion cap. */
+static int build_ipip_nest(u8 *buf, int levels)
+{
+	int n = 0, i;
+
+	for (i = 0; i < levels; i++)
+		n += put_ipv4(buf + n, IPPROTO_IPIP, 5, 0);
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_nest(u8 *buf, int levels)
+{
+	int n = 0, i;
+
+	for (i = 0; i < levels; i++) {
+		n += put_ipv4(buf + n, IPPROTO_GRE, 5, 0);
+		n += put_gre(buf + n, 0, htons(ETH_P_IP));
+	}
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_gre_csum_ipv4(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_GRE, 5, 0);
+
+	n += put_gre(buf + n, GRE_CSUM, htons(ETH_P_IP));
+	memset(buf + n, 0, 4);		/* checksum + reserved */
+	n += 4;
+	n += put_ipv4(buf + n, IPPROTO_TCP, 5, 0);
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv4_options_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 6, 0);	/* IHL=6: 4B options */
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv4_frag_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 5, htons(IP_MF));
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv4_frag_offset_tcp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_TCP, 5, htons(0x0010));
+
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv4_icmp(u8 *buf)
+{
+	int n = put_ipv4(buf, IPPROTO_ICMP, 5, 0);
+	u8 icmp[8] = { 0x08, 0x00, 0x00, 0x00, 0x12, 0x34, 0x00, 0x01 };
+
+	memcpy(buf + n, icmp, sizeof(icmp));
+	return n + sizeof(icmp);
+}
+
+static int build_ipv6_hopopts_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_HOPOPTS, 0);
+	u8 hbh[8] = { IPPROTO_TCP, 0, 0, 0, 0, 0, 0, 0 };
+
+	memcpy(buf + n, hbh, sizeof(hbh));
+	n += sizeof(hbh);
+	return n + put_ports(buf + n);
+}
+
+static int build_ipv6_flowlabel_tcp(u8 *buf)
+{
+	int n = put_ipv6(buf, IPPROTO_TCP, 0xbeef);
+
+	return n + put_ports(buf + n);
+}
+
+struct fd_fast_case {
+	const char *name;
+	int (*build)(u8 *buf);
+	__be16 proto;
+	unsigned int flags;
+};
+
+/* htons() is not constant-foldable here without cpu_to_be16 literals;
+ * use cpu_to_be16 for static initializers.
+ */
+static const struct fd_fast_case fd_fast_cases[] = {
+	/* eligible shapes: the fast path should take these */
+	{ "ipv4_tcp", build_ipv4_tcp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv4_udp", build_ipv4_udp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv6_tcp", build_ipv6_tcp, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "ipv6_udp", build_ipv6_udp, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "vlan_ipv4_tcp", build_vlan_ipv4_tcp, cpu_to_be16(ETH_P_8021Q), 0 },
+	{ "vlan_ipv6_udp", build_vlan_ipv6_udp, cpu_to_be16(ETH_P_8021Q), 0 },
+	{ "qinq_ipv4_tcp", build_qinq_ipv4_tcp, cpu_to_be16(ETH_P_8021AD), 0 },
+	{ "pppoe_ipv4_tcp", build_pppoe_ipv4_tcp, cpu_to_be16(ETH_P_PPP_SES), 0 },
+	{ "pppoe_ipv6_tcp", build_pppoe_ipv6_tcp, cpu_to_be16(ETH_P_PPP_SES), 0 },
+	{ "mpls_ipv4", build_mpls_ipv4, cpu_to_be16(ETH_P_MPLS_UC), 0 },
+	{ "ipip", build_ipip, cpu_to_be16(ETH_P_IP), 0 },
+	{ "6in4", build_6in4, cpu_to_be16(ETH_P_IP), 0 },
+	{ "4in6", build_4in6, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "4in6_flowlabel", build_4in6_flowlabel, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "6in6", build_6in6, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "gre_ipv4", build_gre_ipv4, cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre6_ipv4", build_gre6_ipv4, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "gre_ipv6", build_gre_ipv6, cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre6_ipv6", build_gre6_ipv6, cpu_to_be16(ETH_P_IPV6), 0 },
+	/* deliberate fast-path misses: must fall back with identical
+	 * output (and, per the zeroed containers, no partial writes)
+	 */
+	{ "ipv4_options_tcp", build_ipv4_options_tcp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv4_frag_mf", build_ipv4_frag_tcp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv4_frag_mf_1stfrag", build_ipv4_frag_tcp, cpu_to_be16(ETH_P_IP),
+	  FLOW_DISSECTOR_F_PARSE_1ST_FRAG },
+	{ "ipv4_frag_offset", build_ipv4_frag_offset_tcp,
+	  cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv4_icmp", build_ipv4_icmp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv6_hopopts_tcp", build_ipv6_hopopts_tcp,
+	  cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "ipv6_flowlabel_tcp", build_ipv6_flowlabel_tcp,
+	  cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "vlan3_ipv4_tcp", build_vlan3_ipv4_tcp, cpu_to_be16(ETH_P_8021Q), 0 },
+	{ "pppoe_lcp", build_pppoe_lcp, cpu_to_be16(ETH_P_PPP_SES), 0 },
+	{ "mpls_multi_label", build_mpls_multi, cpu_to_be16(ETH_P_MPLS_UC), 0 },
+	{ "gre_csum_ipv4", build_gre_csum_ipv4, cpu_to_be16(ETH_P_IP), 0 },
+	/* corner cases */
+	{ "gre_seq_ipv4", build_gre_seq_ipv4, cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre_key_ipv4", build_gre_key_ipv4, cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre_version1", build_gre_version1, cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre_teb", build_gre_teb, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv6_hopopts_dest_tcp", build_ipv6_hopopts_dest_tcp,
+	  cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "ipv6_frag_tcp", build_ipv6_frag_tcp, cpu_to_be16(ETH_P_IPV6), 0 },
+	{ "vlan_q_then_ad_ipv4", build_vlan_q_then_ad_ipv4,
+	  cpu_to_be16(ETH_P_8021Q), 0 },
+	{ "ipv4_options12_tcp", build_ipv4_options12_tcp,
+	  cpu_to_be16(ETH_P_IP), 0 },
+	{ "pppoe_pfc", build_pppoe_pfc, cpu_to_be16(ETH_P_PPP_SES), 0 },
+	{ "ipv4_frag_mid", build_ipv4_frag_mid_tcp, cpu_to_be16(ETH_P_IP), 0 },
+	{ "ipv4_frag_mid_1stfrag", build_ipv4_frag_mid_tcp,
+	  cpu_to_be16(ETH_P_IP), FLOW_DISSECTOR_F_PARSE_1ST_FRAG },
+	{ "gre_csum_key_ipv4", build_gre_csum_key_ipv4,
+	  cpu_to_be16(ETH_P_IP), 0 },
+	{ "gre_csum_key_seq_ipv4", build_gre_csum_key_seq_ipv4,
+	  cpu_to_be16(ETH_P_IP), 0 },
+	{ "vlan_prio_ipv4_tcp", build_vlan_prio_ipv4_tcp,
+	  cpu_to_be16(ETH_P_8021Q), 0 },
+	{ "ipv6_routing_tcp", build_ipv6_routing_tcp,
+	  cpu_to_be16(ETH_P_IPV6), 0 },
+};
+
+static void fd_fast_case_to_desc(const struct fd_fast_case *c, char *desc)
+{
+	strscpy(desc, c->name, KUNIT_PARAM_DESC_SIZE);
+}
+
+KUNIT_ARRAY_PARAM(fd_fast, fd_fast_cases, fd_fast_case_to_desc);
+
+/* Dissect with gates off then on; returns and output must match. */
+static void fd_check_one(struct kunit *test, struct flow_dissector *fd,
+			 const u8 *data, __be16 proto, int hlen,
+			 unsigned int flags)
+{
+	struct flow_keys keys_slow, keys_fast;
+	bool ret_slow, ret_fast;
+
+	fd_fast_gates_set(false);
+	memset(&keys_slow, 0, sizeof(keys_slow));
+	ret_slow = __skb_flow_dissect(&init_net, NULL, fd, &keys_slow,
+				      data, proto, 0, hlen, flags);
+
+	fd_fast_gates_set(true);
+	memset(&keys_fast, 0, sizeof(keys_fast));
+	ret_fast = __skb_flow_dissect(&init_net, NULL, fd, &keys_fast,
+				      data, proto, 0, hlen, flags);
+	fd_fast_gates_set(false);
+
+	KUNIT_EXPECT_EQ(test, ret_slow, ret_fast);
+	KUNIT_EXPECT_MEMEQ(test, &keys_slow, &keys_fast, sizeof(keys_slow));
+	/* flow_keys -> skb->hash is what consumers see; assert it too. */
+	KUNIT_EXPECT_EQ(test, flow_hash_from_keys(&keys_slow),
+			flow_hash_from_keys(&keys_fast));
+}
+
+/* Check against both eligible dissectors (different used_keys). */
+static void fd_fast_check_equiv(struct kunit *test, const u8 *data,
+				__be16 proto, int hlen, unsigned int flags)
+{
+	/* skb_get_hash() passes STOP_AT_FLOW_LABEL on every dissect; each
+	 * case must hold with and without it.
+	 */
+	unsigned int fl = flags | FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL;
+
+	fd_check_one(test, &flow_keys_dissector, data, proto, hlen, flags);
+	fd_check_one(test, flow_keys_dissector_symmetric_kunit(),
+		     data, proto, hlen, flags);
+	fd_check_one(test, &flow_keys_dissector, data, proto, hlen, fl);
+	fd_check_one(test, flow_keys_dissector_symmetric_kunit(),
+		     data, proto, hlen, fl);
+}
+
+static void fd_fast_equiv_test(struct kunit *test)
+{
+	const struct fd_fast_case *c = test->param_value;
+	u8 *buf;
+	int len;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+
+	len = c->build(buf);
+	KUNIT_ASSERT_LE(test, len, FD_TEST_BUF_LEN);
+
+	fd_fast_check_equiv(test, buf, c->proto, len, c->flags);
+}
+
+/* Truncation sweep: the fast path must never parse (or partially
+ * write) anything the slow path rejected at the same length.
+ */
+static void fd_fast_truncation_test(struct kunit *test)
+{
+	int i, len, hlen;
+	u8 *buf;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+
+	for (i = 0; i < ARRAY_SIZE(fd_fast_cases); i++) {
+		const struct fd_fast_case *c = &fd_fast_cases[i];
+
+		len = c->build(buf);
+		for (hlen = len - 1; hlen >= 0; hlen--)
+			fd_fast_check_equiv(test, buf, c->proto, hlen,
+					    c->flags);
+	}
+}
+
+/* skb-mode builder (for hw-accel VLAN entry conditions). */
+static struct sk_buff *fd_fast_build_skb(struct kunit *test,
+					 int (*build)(u8 *buf),
+					 __be16 protocol)
+{
+	struct sk_buff *skb;
+	int len;
+	u8 *buf;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+	len = build(buf);
+
+	skb = alloc_skb(FD_TEST_BUF_LEN + NET_SKB_PAD, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+	skb_reserve(skb, NET_SKB_PAD);
+	skb_put_data(skb, buf, len);
+	skb->protocol = protocol;
+	skb_reset_mac_header(skb);
+	skb_reset_network_header(skb);
+	return skb;
+}
+
+static void fd_check_one_skb(struct kunit *test, struct flow_dissector *fd,
+			     struct sk_buff *skb)
+{
+	struct flow_keys keys_slow, keys_fast;
+	bool ret_slow, ret_fast;
+
+	fd_fast_gates_set(false);
+	memset(&keys_slow, 0, sizeof(keys_slow));
+	ret_slow = __skb_flow_dissect(&init_net, skb, fd,
+				      &keys_slow, NULL, 0, 0, 0, 0);
+
+	fd_fast_gates_set(true);
+	memset(&keys_fast, 0, sizeof(keys_fast));
+	ret_fast = __skb_flow_dissect(&init_net, skb, fd,
+				      &keys_fast, NULL, 0, 0, 0, 0);
+	fd_fast_gates_set(false);
+
+	KUNIT_EXPECT_EQ(test, ret_slow, ret_fast);
+	KUNIT_EXPECT_MEMEQ(test, &keys_slow, &keys_fast, sizeof(keys_slow));
+	KUNIT_EXPECT_EQ(test, flow_hash_from_keys(&keys_slow),
+			flow_hash_from_keys(&keys_fast));
+}
+
+static void fd_fast_check_equiv_skb(struct kunit *test, struct sk_buff *skb)
+{
+	fd_check_one_skb(test, &flow_keys_dissector, skb);
+	fd_check_one_skb(test, flow_keys_dissector_symmetric_kunit(), skb);
+}
+
+static void fd_fast_skb_plain_test(struct kunit *test)
+{
+	struct sk_buff *skb;
+
+	skb = fd_fast_build_skb(test, build_ipv4_tcp, htons(ETH_P_IP));
+	fd_fast_check_equiv_skb(test, skb);
+	kfree_skb(skb);
+}
+
+static void fd_fast_skb_hwaccel_vlan_test(struct kunit *test)
+{
+	struct sk_buff *skb;
+
+	/* Plain payload; VLAN tag in skb metadata, as after hw stripping. */
+	skb = fd_fast_build_skb(test, build_ipv4_tcp, htons(ETH_P_IP));
+	__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), 100);
+	fd_fast_check_equiv_skb(test, skb);
+	kfree_skb(skb);
+}
+
+static void fd_fast_skb_hwaccel_vlan_qinq_test(struct kunit *test)
+{
+	struct sk_buff *skb;
+
+	/* Outer tag hw-stripped (8021AD), second tag in the payload. */
+	skb = fd_fast_build_skb(test, build_vlan_ipv4_tcp, htons(ETH_P_8021Q));
+	__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021AD), 100);
+	fd_fast_check_equiv_skb(test, skb);
+	kfree_skb(skb);
+}
+
+/* Nested tunnels: both paths must stop at MAX_FLOW_DISSECT_HDRS --
+ * shallow stacks descend, past the cap the fast path defers and
+ * matches the slow path's capped result.
+ */
+static void fd_fast_deep_nest_test(struct kunit *test)
+{
+	int levels, len;
+	u8 *buf;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+
+	for (levels = 1; levels <= 24; levels++) {
+		len = build_ipip_nest(buf, levels);
+		if (len > FD_TEST_BUF_LEN)
+			break;
+		fd_fast_check_equiv(test, buf, htons(ETH_P_IP), len, 0);
+	}
+	for (levels = 1; levels <= 16; levels++) {
+		len = build_gre_nest(buf, levels);
+		if (len > FD_TEST_BUF_LEN)
+			break;
+		fd_fast_check_equiv(test, buf, htons(ETH_P_IP), len, 0);
+	}
+}
+
+/* Non-linear skb: the fast path reads only the linear head and must
+ * defer; the slow path pulls from the frag. Output must match.
+ */
+static void fd_fast_nonlinear_skb_test(struct kunit *test)
+{
+	const int linear = 8;
+	struct sk_buff *skb;
+	struct page *page;
+	u8 *buf;
+	int len;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+	len = build_ipv4_tcp(buf);
+
+	skb = alloc_skb(NET_SKB_PAD + linear, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+	skb_reserve(skb, NET_SKB_PAD);
+	skb_put_data(skb, buf, linear);
+
+	page = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, page);
+	memcpy(page_address(page), buf + linear, len - linear);
+	skb_add_rx_frag(skb, 0, page, 0, len - linear, len - linear);
+
+	skb->protocol = htons(ETH_P_IP);
+	skb_reset_mac_header(skb);
+	skb_reset_network_header(skb);
+
+	fd_fast_check_equiv_skb(test, skb);
+	kfree_skb(skb);
+}
+
+/* An ineligible (custom) dissector: the fast path must never run,
+ * so gates-on output equals gates-off for the whole corpus.
+ */
+static void fd_fast_ineligible_dissector_test(struct kunit *test)
+{
+	static const struct flow_dissector_key keys[] = {
+		{ .key_id = FLOW_DISSECTOR_KEY_CONTROL,
+		  .offset = offsetof(struct flow_keys, control) },
+		{ .key_id = FLOW_DISSECTOR_KEY_BASIC,
+		  .offset = offsetof(struct flow_keys, basic) },
+		{ .key_id = FLOW_DISSECTOR_KEY_IPV4_ADDRS,
+		  .offset = offsetof(struct flow_keys, addrs.v4addrs) },
+		{ .key_id = FLOW_DISSECTOR_KEY_IPV6_ADDRS,
+		  .offset = offsetof(struct flow_keys, addrs.v6addrs) },
+		{ .key_id = FLOW_DISSECTOR_KEY_PORTS,
+		  .offset = offsetof(struct flow_keys, ports) },
+		{ .key_id = FLOW_DISSECTOR_KEY_ICMP,
+		  .offset = offsetof(struct flow_keys, icmp) },
+	};
+	struct flow_dissector custom;
+	int i, len;
+	u8 *buf;
+
+	skb_flow_dissector_init(&custom, keys, ARRAY_SIZE(keys));
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+
+	for (i = 0; i < ARRAY_SIZE(fd_fast_cases); i++) {
+		const struct fd_fast_case *c = &fd_fast_cases[i];
+
+		len = c->build(buf);
+		fd_check_one(test, &custom, buf, c->proto, len, c->flags);
+	}
+}
+
+/* skb-mode coverage for ipv6 / pppoe / mpls / gre. */
+static void fd_fast_skb_shapes_test(struct kunit *test)
+{
+	static const struct {
+		int (*build)(u8 *buf);
+		__be16 proto;
+	} cases[] = {
+		{ build_ipv6_tcp, cpu_to_be16(ETH_P_IPV6) },
+		{ build_pppoe_ipv4_tcp, cpu_to_be16(ETH_P_PPP_SES) },
+		{ build_mpls_ipv4, cpu_to_be16(ETH_P_MPLS_UC) },
+		{ build_gre_ipv4, cpu_to_be16(ETH_P_IP) },
+	};
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(cases); i++) {
+		struct sk_buff *skb;
+
+		skb = fd_fast_build_skb(test, cases[i].build, cases[i].proto);
+		fd_fast_check_equiv_skb(test, skb);
+		kfree_skb(skb);
+	}
+}
+
+/* Seeded, reproducible fuzzer: mutate corpus packets and lengths,
+ * assert equivalence each iteration -- the guard for divergences the
+ * hand-written corpus does not name.
+ */
+static void fd_fast_fuzz_test(struct kunit *test)
+{
+	struct rnd_state rnd;
+	u8 *buf;
+	int i;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+	prandom_seed_state(&rnd, 0x1a2b3c4d5e6f7788ULL);
+
+	for (i = 0; i < FD_FUZZ_ITERS; i++) {
+		const struct fd_fast_case *c;
+		unsigned int flags;
+		int len, j, nmut;
+
+		c = &fd_fast_cases[prandom_u32_state(&rnd) %
+				   ARRAY_SIZE(fd_fast_cases)];
+		len = c->build(buf);
+
+		/* Mutate a handful of header bytes to random values. */
+		nmut = prandom_u32_state(&rnd) % (len + 1);
+		for (j = 0; j < nmut; j++)
+			buf[prandom_u32_state(&rnd) % len] =
+				prandom_u32_state(&rnd);
+
+		/* Random truncation and random 1st-frag flag. */
+		len = prandom_u32_state(&rnd) % (len + 1);
+		flags = (prandom_u32_state(&rnd) & 1) ?
+			FLOW_DISSECTOR_F_PARSE_1ST_FRAG : 0;
+
+		fd_fast_check_equiv(test, buf, c->proto, len, flags);
+	}
+}
+
+/* Gates off: the fast path must never run (counters are the
+ * observable). Positive controls: an eligible shape must count with
+ * the gates on.
+ */
+static void fd_fast_gates_off_test(struct kunit *test)
+{
+	struct flow_keys keys;
+	u64 before;
+	int i, len;
+	u8 *buf;
+
+	buf = kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf);
+
+	fd_fast_gates_set(false);
+	before = flow_dissector_fast_hits_kunit();
+	for (i = 0; i < ARRAY_SIZE(fd_fast_cases); i++) {
+		len = fd_fast_cases[i].build(buf);
+		memset(&keys, 0, sizeof(keys));
+		__skb_flow_dissect(&init_net, NULL, &flow_keys_dissector,
+				   &keys, buf, fd_fast_cases[i].proto, 0,
+				   len, fd_fast_cases[i].flags);
+	}
+	KUNIT_EXPECT_EQ(test, flow_dissector_fast_hits_kunit(), before);
+
+	fd_fast_gates_set(true);
+	len = build_ipv4_tcp(buf);
+	memset(&keys, 0, sizeof(keys));
+	__skb_flow_dissect(&init_net, NULL, &flow_keys_dissector, &keys,
+			   buf, htons(ETH_P_IP), 0, len, 0);
+	KUNIT_EXPECT_GT(test, flow_dissector_fast_hits_kunit(), before);
+
+	/* Flagship consumer shape: skb_get_hash() passes STOP_AT_FLOW_LABEL;
+	 * the fast path must hit for it.
+	 */
+	before = flow_dissector_fast_hits_kunit();
+	memset(&keys, 0, sizeof(keys));
+	__skb_flow_dissect(&init_net, NULL, &flow_keys_dissector, &keys,
+			   buf, htons(ETH_P_IP), 0, len,
+			   FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL);
+	fd_fast_gates_set(false);
+	KUNIT_EXPECT_GT(test, flow_dissector_fast_hits_kunit(), before);
+}
+
+static void fd_all_gates_off(void)
+{
+	fd_fast_gates_set(false);
+}
+
+static int fd_fast_suite_init(struct kunit_suite *suite)
+{
+	fd_all_gates_off();
+	return 0;
+}
+
+static void fd_fast_suite_exit(struct kunit_suite *suite)
+{
+	fd_all_gates_off();
+}
+
+static struct kunit_case fd_fast_test_cases[] = {
+	KUNIT_CASE_PARAM(fd_fast_equiv_test, fd_fast_gen_params),
+	KUNIT_CASE(fd_fast_truncation_test),
+	KUNIT_CASE(fd_fast_deep_nest_test),
+	KUNIT_CASE(fd_fast_nonlinear_skb_test),
+	KUNIT_CASE(fd_fast_ineligible_dissector_test),
+	KUNIT_CASE(fd_fast_skb_plain_test),
+	KUNIT_CASE(fd_fast_skb_hwaccel_vlan_test),
+	KUNIT_CASE(fd_fast_skb_hwaccel_vlan_qinq_test),
+	KUNIT_CASE(fd_fast_skb_shapes_test),
+	KUNIT_CASE(fd_fast_fuzz_test),
+	KUNIT_CASE(fd_fast_gates_off_test),
+	{}
+};
+
+static struct kunit_suite fd_fast_suite = {
+	.name = "flow_dissector_fastpath",
+	.suite_init = fd_fast_suite_init,
+	.suite_exit = fd_fast_suite_exit,
+	.test_cases = fd_fast_test_cases,
+};
+
+kunit_test_suite(fd_fast_suite);
+
+MODULE_DESCRIPTION("KUnit fast/slow equivalence tests for the flow dissector");
+MODULE_LICENSE("GPL");
-- 
2.54.0

[PATCH net-next v1 11/11] Documentation: networking: add flow_dissector overview and fast-path guide

[Dave Seddon]

The kernel has no general flow_dissector documentation: Documentation/bpf/
prog_flow_dissector.rst covers only the BPF override mechanism (and was
itself in no toctree). Add Documentation/networking/flow_dissector.rst as a
standalone overview -- what the flow dissector is, struct flow_keys and the
consumers of skb->hash (RSS/RPS/RFS, ECMP/multipath, bonding/LAG,
tc-flower,
aRFS), the generic protocol-graph parser, and the relationship to the BPF
flow dissector -- and document the opt-in fast paths added by this series:
the byte-identical per-shape gates, the
break-even model for when to enable a gate (with the observable
/proc/net/flow_dissector_stats). The per-knob
reference stays in Documentation/admin-guide/sysctl/net.rst; this doc links
to it rather than duplicating it.

Also index the previously-orphaned
Documentation/bpf/prog_flow_dissector.rst and cross-link it with the
new overview (the bpf/index.rst hunk can be split out if the BPF
maintainers prefer).

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@gmail.com>
---
 Documentation/bpf/index.rst                 |   1 +
 Documentation/bpf/prog_flow_dissector.rst   |   2 +
 Documentation/networking/flow_dissector.rst | 133 ++++++++++++++++++++
 Documentation/networking/index.rst          |   1 +
 4 files changed, 137 insertions(+)
 create mode 100644 Documentation/networking/flow_dissector.rst

diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index 0d5c6f659266..6e26731fa06f 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst
@@ -23,6 +23,7 @@ that goes into great technical depth about the BPF Architecture.
    cpumasks
    fs_kfuncs
    programs
+   prog_flow_dissector
    maps
    bpf_prog_run
    classic_vs_extended.rst
diff --git a/Documentation/bpf/prog_flow_dissector.rst b/Documentation/bpf/prog_flow_dissector.rst
index f24270b8b034..f4c4648461ca 100644
--- a/Documentation/bpf/prog_flow_dissector.rst
+++ b/Documentation/bpf/prog_flow_dissector.rst
@@ -9,6 +9,8 @@ Overview
 
 Flow dissector is a routine that parses metadata out of the packets. It's
 used in the various places in the networking subsystem (RFS, flow hash, etc).
+See :doc:`/networking/flow_dissector` for an overview of the flow dissector
+and the built-in C implementation this program type overrides.
 
 BPF flow dissector is an attempt to reimplement C-based flow dissector logic
 in BPF to gain all the benefits of BPF verifier (namely, limits on the
diff --git a/Documentation/networking/flow_dissector.rst b/Documentation/networking/flow_dissector.rst
new file mode 100644
index 000000000000..014f6d124b4f
--- /dev/null
+++ b/Documentation/networking/flow_dissector.rst
@@ -0,0 +1,133 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============
+Flow Dissector
+==============
+
+Overview
+========
+
+The flow dissector parses a packet's headers into a ``struct flow_keys``: the
+addresses, ports, and other fields that identify the flow the packet belongs
+to. Its entry point is ``__skb_flow_dissect()`` in
+``net/core/flow_dissector.c``.
+
+The dissected keys feed ``skb->hash`` (via ``flow_hash_from_keys()``), which is
+the canonical flow identity consumed all over the stack, including:
+
+* receive-side steering: RSS, RPS, RFS and accelerated RFS (aRFS);
+* transmit-side spreading: ECMP / multipath route selection and bonding / LAG
+  ``xmit_hash_policy``;
+* classification and offload: tc-flower.
+
+A single dissection therefore feeds every one of these, so the dissector sits
+on a hot path in most networking workloads.
+
+Two standard dissectors are built in: ``flow_keys_dissector`` (the general one)
+and ``flow_keys_dissector_symmetric`` (same source and destination hashed
+symmetrically). In addition, subsystems such as tc-flower build **custom**
+dissector instances at run time that request only the keys they need. A custom
+instance carries its own ``used_keys`` bitmask and per-key offset table, so its
+field accesses are resolved dynamically at each dissect.
+
+struct flow_keys
+================
+
+``struct flow_keys`` (see ``include/net/flow_dissector.h``) collects the
+dissected metadata. Its main members are:
+
+* ``control`` -- address type and control flags (for example
+  ``FLOW_DIS_ENCAPSULATION`` when the keys came from inside a tunnel, and
+  ``FLOW_DIS_IS_FRAGMENT``);
+* ``basic`` -- L3 protocol (``n_proto``) and L4 protocol (``ip_proto``);
+* ``addrs`` -- source and destination addresses (IPv4 or IPv6);
+* ``ports`` -- source and destination ports;
+* ``vlan`` / ``cvlan`` -- 802.1Q / 802.1AD tags;
+* ``tags``, ``keyid``, ``icmp`` -- MPLS/flow-label tags, GRE/tunnel key, and
+  ICMP identifiers.
+
+The dissection path
+===================
+
+The built-in dissector is a generic protocol-graph parser: it walks the header
+chain in a loop, dispatching on the current L2/L3 ethertype and then the L4
+protocol, following encapsulation (IP-in-IP, GRE, ...) by re-entering the loop
+on the inner header. The number of headers it will descend through is bounded
+by ``MAX_FLOW_DISSECT_HDRS`` so that a crafted deeply nested packet cannot make
+it loop without limit.
+
+This generic parser is referred to below as the *slow path*: it is table- and
+loop-driven and handles every protocol the dissector knows about.
+
+The BPF flow dissector
+======================
+
+A per-network-namespace BPF program can be attached to replace the built-in
+dissector for the protocols it chooses to handle. When one is attached,
+``__skb_flow_dissect()`` runs it **first**; if the program returns a verdict
+other than ``BPF_FLOW_DISSECTOR_CONTINUE``, its result is used and the built-in
+dissector (including the fast paths below) is not run for that packet. A system
+running an attached BPF dissector therefore sees behaviour defined entirely by
+its program.
+
+See :doc:`/bpf/prog_flow_dissector` for the BPF program type, its API, and its
+reference implementation.
+
+Opt-in fast paths
+=================
+
+For the packet shapes that dominate real deployments, the built-in dissector
+also provides an opt-in, straight-line *fast path* that produces the same
+result as the slow-path graph walk with far fewer instructions. Restricting the
+fast path to the two standard dissectors turns their per-key offsets into
+compile-time constants, so the parse becomes branch-light straight-line code;
+the isolated per-dissect cost of the eligible shapes drops by roughly half,
+with the largest wins on in-order cores.
+
+Each shape has its own gate -- a ``static_branch`` exposed as a sysctl under
+``/proc/sys/net/flow_dissector/`` -- and **every gate is off by default**. When
+a gate is off, the only added cost is one not-taken branch per dissect, and the
+dissector's output is exactly what it was before. The fast path runs only
+*after* the BPF hook described above, so an attached BPF program always takes
+precedence.
+
+The fast paths are **byte-identical by contract**: for any packet, a fast path
+either writes exactly the ``flow_keys`` the slow path would have written, or it
+returns false and the slow path runs. There are only these two outcomes, and
+the equivalence is enforced in-tree by a KUnit test suite
+(``CONFIG_FLOW_DISSECTOR_KUNIT_TEST``). Enabling a byte-identical shape gate
+never changes any consumer's hash; it only makes the dissection cheaper.
+
+The eligible byte-identical shapes are plain Ethernet + IPv4/IPv6 + TCP/UDP,
+single and stacked VLAN (QinQ), PPPoE sessions, a single MPLS label, the
+IP-in-IP family, and plain GRE. The individual knobs are documented in
+:doc:`/admin-guide/sysctl/net` under ``/proc/sys/net/flow_dissector``.
+
+When to enable
+==============
+
+Turning a byte-identical shape gate on is worthwhile when enough traffic matches
+that shape to repay the small cost a non-matching packet pays while the gate is
+on. Concretely, if a matching packet saves ``S`` and a non-matching packet costs
+``C`` (both measured per shape and per micro-architecture), enabling the gate is
+net-positive once the fraction of traffic matching the shape exceeds the
+break-even ``p_be = C / (S + C)``.
+
+That fraction is observable: the read-only file
+``/proc/net/flow_dissector_stats`` reports, per shape, how many packets the slow
+path handled, how many the fast body handled, and the resulting eligible
+fraction, so ``(occurrences + fast_hits) / dissects`` is a gate-invariant
+measure of how much traffic each shape would accelerate.
+
+A separate RFC proposes an optional ``auto`` mode built on these counters: the
+kernel would sample them over a packet-count window and flip the
+byte-identical gates itself, with hysteresis and a flip-rate cap, so the
+operator does not have to tune each shape by hand. That controller is not part
+of this series.
+
+References
+==========
+
+* :doc:`/admin-guide/sysctl/net` -- the ``/proc/sys/net/flow_dissector/``
+  per-shape knob reference and the ``/proc/net/flow_dissector_stats`` file.
+* :doc:`/bpf/prog_flow_dissector` -- the BPF flow dissector program type.
diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
index 44a422ad3b05..65368050ebab 100644
--- a/Documentation/networking/index.rst
+++ b/Documentation/networking/index.rst
@@ -52,6 +52,7 @@ Contents:
    eql
    fib_trie
    filter
+   flow_dissector
    generic-hdlc
    generic_netlink
    ../netlink/specs/index
-- 
2.54.0

Re: [PATCH net-next v1 00/11] net: flow_dissector: opt-in byte-identical fast paths for common shapes

[Willem de Bruijn]

Dave Seddon wrote:
> G'day,
> 
> __skb_flow_dissect() parses packet headers with a generic protocol-graph
> parser. The graph walk is what makes the dissector so flexible, but most
> machines spend that flexibility on the same traffic all day long: plain
> eth + IPv4/IPv6 + TCP/UDP, perhaps with a VLAN tag or a tunnel header in
> front. The intuition behind this series is that the common shapes can be
> dissected by straight-line code instead, without giving up the graph
> walk for everything else.
> 
> Concretely: the series adds an opt-in, static-branch-gated fast path per
> packet shape, covering the common cases and the IP-in-IP/GRE
> encapsulations. Each shape has its own gate under
> /proc/sys/net/flow_dissector/, every gate is disabled by default, and
> when a gate is off its added cost is one not-taken branch per dissect.
> The gates are manual in this series; a follow-up RFC will propose an
> optional auto mode that flips them from the measured traffic mix (via
> the patch-8 counters), so nobody has to hand-tune seven knobs
> (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre). Nothing
> here depends on it -- it is where this is heading, not a prerequisite.

> Relationship to the BPF flow dissector
> ======================================
> The fast path is invoked strictly after the netns BPF flow dissector
> hook in __skb_flow_dissect(): if a program is attached and returns any
> verdict other than BPF_FLOW_DISSECTOR_CONTINUE, the function returns
> before flow_dissect_fast() is reached. Attached BPF dissectors therefore
> always take precedence, and a system running one sees bit-for-bit
> unchanged behaviour from this series (patch 1 additionally makes its
> program lookup cheaper for everyone else). Dissects fully handled by a
> BPF program are deliberately not counted in
> /proc/net/flow_dissector_stats (patch 8).
> 
> The gates are global (a static key patches code shared by every netns);
> per-netns dissector policy already has a mechanism -- the netns BPF
> flow dissector -- and it keeps full precedence here.

This series adds a lot of code: +2500 LoC.

If you want a linear fast path, the BPF dissector offers that.

I don't think the purported benefit justifies the significant new
code, I'm afraid.

The code is duplicative of existing paths, so there may be additional
maintenance cost keeping the two consistent.

And which paths justify a fast paths and which do not is highly
subjective. Why is GRE included, for instance?