[PATCH net-next v1 00/11] net: flow_dissector: opt-in byte-identical fast paths for common shapes

[Dave Seddon <dave.seddon.ca@gmail.com>]

G'day,

__skb_flow_dissect() parses packet headers with a generic protocol-graph
parser. The graph walk is what makes the dissector so flexible, but most
machines spend that flexibility on the same traffic all day long: plain
eth + IPv4/IPv6 + TCP/UDP, perhaps with a VLAN tag or a tunnel header in
front. The intuition behind this series is that the common shapes can be
dissected by straight-line code instead, without giving up the graph
walk for everything else.

Concretely: the series adds an opt-in, static-branch-gated fast path per
packet shape, covering the common cases and the IP-in-IP/GRE
encapsulations. Each shape has its own gate under
/proc/sys/net/flow_dissector/, every gate is disabled by default, and
when a gate is off its added cost is one not-taken branch per dissect.
The gates are manual in this series; a follow-up RFC will propose an
optional auto mode that flips them from the measured traffic mix (via
the patch-8 counters), so nobody has to hand-tune seven knobs
(eth_ip, vlan, qinq, pppoe, mpls, ipip, gre). Nothing
here depends on it -- it is where this is heading, not a prerequisite.

The series is byte-identical by contract: for any packet, the fast path
either writes exactly the flow_keys bytes the slow path would have
written, or it returns false and the slow path runs. That contract is
enforced in-tree by a KUnit equivalence suite (patch 10).

Patch 1 stands slightly apart: it is a small win for the existing path --
today every dissect takes rcu_read_lock() and does two dependent
rcu_dereference() loads to look for a netns BPF flow dissector program
even when none is attached anywhere; a static key (mirroring
bpf_sk_lookup_enabled) skips all of it in the common no-program case.

Patches 2-7 are the byte-identical common shapes (eth+IPv4/IPv6 x
TCP/UDP, VLAN/QinQ, PPPoE, single MPLS, IP-in-IP, plain GRE). Patch 8
adds per-shape counters (/proc/net/flow_dissector_stats); patch 9 bounds
the fast-path tunnel recursion at MAX_FLOW_DISSECT_HDRS.

A companion series (posted separately) builds on this one to add opt-in
inner-flow descent for the UDP tunnels (VXLAN, Geneve, GTP-U, FOU/GUE)
-- that one changes hashing when enabled, so it is a separate policy
discussion; this series is purely a transparent optimisation.

Why not just optimise the existing loop
========================================
The slow path is a generic graph walk by design: one loop, one
proto/nhoff state machine, re-entered per header, able to express every
protocol relationship the dissector knows. That generality is the cost --
per-header dispatch, state saves, and branchy key-writing that the
compiler cannot flatten because the shape is only known at runtime. A
per-shape straight-line function knows the whole shape at compile time,
so the loads, checks and key writes schedule as one block. The two
approaches are complementary: the loop stays the single source of
semantic truth for every shape, and a handful of high-volume shapes get
a verified shortcut.

Performance summary
===================
Measured across 3 ISAs and 8 microarchitectures in total, with
byte-identical verification per shape (userspace A/B harness compiling
the dissector both ways from the same source). Two instruments, each
run on 7 of the 8 microarchitectures: "allshapes" compiles every fast
path in (the realistic icache footprint) and measures each shape in
isolation, on x86 Zen1/Zen2/Skylake, ARM Cortex-A53/A72/A76 and RISC-V
X60; "isolated A/B" compiles one shape's path against the unmodified
dissector (the upper bound for that shape), on the same set with
Haswell in place of Zen 1.

Reduction in dissector CPU cost, allshapes microbench. Each row is the
range across its 7 microarchitectures, worst to best:

    shape     worst    best
    ------   ------   ------
    eth_ip    -4.7%   -31.6%
    vlan     -18.1%   -38.5%
    qinq     -16.1%   -39.8%
    pppoe     -8.6%   -28.0%
    mpls      -6.7%   -15.0%
    ipip     -36.8%   -47.0%   largest: slow path re-parses inner IP
    gre        (byte-identical descent family, tracks ipip; verified
                by the KUnit suite rather than separately benchmarked)

Isolated A/B, eth + IPv4 + TCP (same source compiled two ways),
nanoseconds per packet, lower is better:

    microarchitecture                      slow    fast   delta
    ------------------------------------  ------  ------  -----
    x86 Zen 2 (Threadripper PRO 3945WX)    12.4     6.6    -47%
    x86 Skylake (Core i9-10885H)           10.6     5.6    -47%
    x86 Haswell (Celeron 2955U)            35.9    18.0    -50%
    ARM Cortex-A53                        119.0    61.3    -48%
    ARM Cortex-A72                         42.4    19.1    -55%
    ARM Cortex-A76                         19.4     9.3    -52%
    RISC-V SpacemiT X60                    98.2    49.3    -50%

The isolated dissect costs 47-55% less per packet, consistent across
the three ISAs and the seven microarchitectures measured with that
instrument; the relative reduction is largest on in-order cores,
indicating the saving is fewer instructions rather than a
microarchitectural artifact. IPv6 uses the same gate and is
byte-identical (x86 -39%, RISC-V -25%).

flow_keys -> skb->hash via skb_get_hash() is the flow identity behind
RPS/RFS/aRFS, the fq / fq_codel / cake fairness queues, bonding/LAG
transmit hashing and more; those callers pass no restricting flags, so
each of them gets the per-dissect saving at once. Dissector callers
that pass STOP flags (ECMP multipath, xfrm policy lookup) or run their
own dissector instances (tc-flower) deliberately stay on the slow
path -- unchanged, not regressed.

Counters (patch 8) are the one part with a cost in the all-gates-off
default: one this_cpu_inc per dissect plus one per matched shape.
Measured on a CPU-bound pktgen soak that is +0.74% dissector time
(+0.44 sigma -- within run-to-run noise). The gates themselves stay
free: off means one not-taken static branch.

Relationship to the BPF flow dissector
======================================
The fast path is invoked strictly after the netns BPF flow dissector
hook in __skb_flow_dissect(): if a program is attached and returns any
verdict other than BPF_FLOW_DISSECTOR_CONTINUE, the function returns
before flow_dissect_fast() is reached. Attached BPF dissectors therefore
always take precedence, and a system running one sees bit-for-bit
unchanged behaviour from this series (patch 1 additionally makes its
program lookup cheaper for everyone else). Dissects fully handled by a
BPF program are deliberately not counted in
/proc/net/flow_dissector_stats (patch 8).

The gates are global (a static key patches code shared by every netns);
per-netns dissector policy already has a mechanism -- the netns BPF
flow dissector -- and it keeps full precedence here.

Maintenance burden: why two paths won't diverge
===============================================
The slow path remains the single source of semantic truth and the fast
path is verified against it, not vice versa. Three layers enforce that:

  1. The eligibility gate: only the two standard dissectors and a flag
     subset ever enter flow_dissect_fast(); custom tc-flower dissector
     instances are structurally unaffected.
  2. Every shape is a default-off static branch: a kernel with gates off
     executes the not-taken-branch slow path only.
  3. The KUnit equivalence suite (patch 10) dissects a corpus of
     eligible shapes, deliberate fast-path misses, truncations at every
     byte boundary, and skb-mode entry conditions through both paths
     and memcmp()s the resulting struct flow_keys (and their
     flow_hash_from_keys()). Future divergence is a failing test, not a
     silent behaviour change.

The test earns its place: during development it caught real divergences
the out-of-tree A/B harness had missed -- the MPLS fast path skipping the
slow path's out_good terminal writes, the 4in6 descent not reproducing
the outer-IPv6 residue, and -- caught by a hardware boot smoke with
counter positive-controls, then pinned by a regression test -- the
dispatcher rejecting the STOP_AT_FLOW_LABEL flag that skb_get_hash()
passes on every dissect, which had made the fast path unreachable from
RPS/RFS/fq/cake entirely. All fixed in this posting.

Robustness and safety limits
============================
The fast path descends into IP-in-IP and GRE by recursion. The slow
path bounds header descents at MAX_FLOW_DISSECT_HDRS; patch 9 gives the
fast path the same bound, so a crafted nested chain falls back to the
slow path at the same depth rather than driving one C stack frame per
~20 bytes of linear header. The KUnit suite covers the bound boundary,
the non-linear-skb fallback, a gates-off negative (the fast path must
never run when disabled, observed via the counters), and a seeded
fuzzer as a general guard.


Patch 11 adds Documentation/networking/flow_dissector.rst -- there was no
general flow_dissector overview in the tree. It explains what the dissector
is and who consumes its output, then documents this series' fast paths: the
byte-identical contract and the break-even model for when a gate is worth
enabling. The per-knob reference stays in
Documentation/admin-guide/sysctl/net.rst; the new doc links to it.

On tooling: this series was developed with substantial LLM assistance
(Claude, Anthropic) -- identifying the optimisation opportunity,
drafting the patches, and building and running the test matrix. Every
patch carries an Assisted-by: trailer in the
Documentation/process/coding-assistants.rst format. All of it was
human-reviewed and human-measured: the performance numbers come from
months of runs across the hardware above, and the byte-identical
contract is enforced mechanically by the in-tree KUnit suite rather
than by trust in either the tool or the author.

This series grew out of work on Tom Herbert's XDP2 parser project:
porting the kernel flow dissector into XDP2's userspace benchmarking
harness, to compare parsers on equal footing, is what made the
per-shape cost of the generic loop visible -- this opt-in fast path is
the kernel-side result of that analysis.

Testing: 3 ISAs / 8 uarches (x86 Zen1/Zen2/Skylake/Haswell, ARM
Cortex-A53/A72/A76, RISC-V X60), in-tree KUnit (patch 10: 55 tests,
also run under KASAN+UBSAN; per-patch W=1, sparse and smatch clean;
CONFIG_NET_FOU n/m/y config matrix for the companion series) and
out-of-tree A/B. Supporting data (flow-distribution study,
per-shape/per-arch matrix, break-even derivation) lives in the
author's XDP2 fork: https://github.com/randomizedcoder/xdp2 , under
perf-results/ .

Dave Seddon (11):
  net: flow_dissector: gate BPF program lookup behind a static key
  net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}
  net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP
  net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} +
    TCP/UDP
  net: flow_dissector: add fast-path for single MPLS label + IP
  net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 /
    6in4)
  net: flow_dissector: add byte-identical fast-path for plain GRE inner
  net: flow_dissector: per-shape counters +
    /proc/net/flow_dissector_stats
  net: flow_dissector: bound fast-path tunnel recursion
  net: flow_dissector: add KUnit fast/slow path equivalence tests
  Documentation: networking: add flow_dissector overview and fast-path
    guide

 Documentation/admin-guide/sysctl/net.rst    |  178 ++-
 Documentation/bpf/index.rst                 |    1 +
 Documentation/bpf/prog_flow_dissector.rst   |    2 +
 Documentation/networking/flow_dissector.rst |  133 +++
 Documentation/networking/index.rst          |    1 +
 include/linux/skbuff.h                      |    1 +
 include/net/flow_dissector.h                |   34 +
 kernel/bpf/net_namespace.c                  |   17 +-
 net/Kconfig                                 |   12 +
 net/core/Makefile                           |    1 +
 net/core/flow_dissector.c                   | 1116 ++++++++++++++++++-
 net/core/flow_dissector_test.c              | 1053 +++++++++++++++++
 12 files changed, 2500 insertions(+), 49 deletions(-)
 create mode 100644 Documentation/networking/flow_dissector.rst
 create mode 100644 net/core/flow_dissector_test.c


base-commit: f6f3b36c15ed44de1fbb44e645e4fae8c4a4453e
-- 
2.54.0