From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6153930AD1A for ; Thu, 16 Jul 2026 00:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162645; cv=none; b=Xopdclifbi65nPc0ITrlPJdrBOcBnCrbXJqfjj+hrlaoIIi/NZU+5IhNf9meL39hzsCTDZhslxhfqQ2IBlLUgTWdZ6k+6JcSVdU9oLhV2SO6VWa0eekUKhhOWtVYN8Sv+9ehF/hKvCgF86iRk3u/1y+8k7nWoXZeF8ns9MiAu+Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162645; c=relaxed/simple; bh=2G6jX/fDUGhuh78H4k/QnWnytFyS22b5IyQpplABToo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QCaSiFyho5dE81k1Qu5ZRyfXH8JqfMhc+D2WHZBkowVuscCON0H1Ms71zjReKCGWXR2pp3MqN9NZP5zlaIV1RNlGiAtNMHdV3BhJRIMxuQP8E9EpF2DmqNl+sTbWv3E/BCuf6gXKXCGu7rILUgvtMxZ+XVLyM6PnjA1++f5lCPg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nAnVNOM+; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nAnVNOM+" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-ca7c1176317so4451488a12.1 for ; Wed, 15 Jul 2026 17:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162643; x=1784767443; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=ielH78VxMdnXNQvVLt3KPq+JAgT1fEDU0jLTYAml4gA=; b=nAnVNOM+RvQVdrqqethTeWkRUg8OBrmX+mapucP2tgEM15iDH7MMMXadR0O9iYLKmS 6S8QmgOdIH+u2jfw1NSRvgJjRXXfDKyqraIGf3yD+SYJeomhuxRGKf0Ho5A3Qd7u36Y5 yZBz5WyiOoT5QHx7kIU7NgepaarzdB8WPPaSIFgBIW449EBrOrsjlna7mFR/zzI6ezqF bRe19Zt3iA1j7TamGuAtJmrWxWE2ig1MGhMP4N1bU0o4xnfPdng4XicZhGKygLzicDe+ I1fhjPm+JMqSBj7tJRLvjiuhmrLnVqZrHXw1rP3//rT94gfGu3B4yjxC7piJucs2PH3V NOmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162643; x=1784767443; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ielH78VxMdnXNQvVLt3KPq+JAgT1fEDU0jLTYAml4gA=; b=i35+uzynvJRWDhJpyIPgPE3j4SUVGHxQTionpYOEByd3jAvB4tfm2jxCnExFoY0pj/ VSRRvJ99ADImOnRODZr1c4RsHsFkeUYiJ7Gu8qETXEOHMTgPR5qaPETBODxwsdA3ocfd l2IxzZT+6XGFy0XNC8d9CBM0VkWWf6nbLnwZGCQAaJD3pzodipm30ms91sCo8HMYM2KX Fp85MimaPF+s1qOXSvSpJm7djlICGuow9+jDYS1InuhL47eF4ojDve70bHGlKqDjmdWm xW5hc66qnJIe/8sCC2GLfKimgGLCGUmK4bv6Dix6E5DRs7QJ/yk4cRzuE03+noeDs7K7 jj1w== X-Gm-Message-State: AOJu0YzZvpMhnLFWVsI6nxzesnYT+hd2l+UVzcRQnhUVIHEFDIDmawQL 81pnmy9YaY9CJxU1Lxy2MJA11NlZfkMJNHM+nWdaKLSHTjcfmSQCGM8jK2BV7A== X-Gm-Gg: AfdE7cnq41mOGbT+S84z7aHZoQZYiyM0qc8XHxp/UUWmk6UMNo2e1yLM4qKPTjHtWCj n+s/ublrWcVPRQqI2lUd8dHwk7wofVzEyL8nTWd7dQ28cKTWwuKq3NX9A1pbWJhGhIcFgZpzDXu trRfBhKOp0yAwETLI1+BZYj/0ErdSsZfufa4hizla/RUL3T+HoRmyrw5JQWdJqpkls2Tyjm/gno R7+GG0wQhccou0yolyp5xRMou05bdGPxR0AAUM0Zo5JujqJmlFWca5IowLlo8eIuy8xaVLgiEJu npUSkbQaiteQNwUd3aOrlxNNglq1+Di5k7C1NwqjTzMgMRG86yEQEs9+u7yC2GlwodN41c9uY0x M3IBv/9iQYvJNyBncIhuflB6sz2eOOKjY0xP0FNxuPSTREnHRRzP/iDeh+pAOLyRbexvmk7fd9h Hd05tJEVVKlRXMrx62l9sYoovv/SkJbfMDvNfBsDyPaO/7IWR8h3h6tT7Ed8xYoXYr5NAYkHFHR dxAD1862g== X-Received: by 2002:a05:6a21:3a42:b0:3c3:8c91:5da0 with SMTP id adf61e73a8af0-3c38c916004mr628308637.15.1784162642618; Wed, 15 Jul 2026 17:44:02 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:02 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 02/11] net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP} Date: Wed, 15 Jul 2026 17:43:48 -0700 Message-ID: <20260716004357.3652679-3-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A small targeted optimisation to __skb_flow_dissect(). A new flow_dissect_fast() dispatcher routes the standard flow_keys dissector calls through per-shape straight-line extractors for the common eth + IPv4/IPv6 + TCP/UDP packet shapes. On any miss, falls back to the existing graph walk. Output is byte-identical to the slow path for the shapes handled. This is the parent of the fast-path series. Subsequent patches add per-shape extensions for single VLAN, QinQ, PPPoE, MPLS, IP-in-IP and GRE. Each patch ships its own per-shape sysctl in the new ``/proc/sys/net/flow_dissector/`` subtree; this patch establishes the subtree and registers the first entry, ``eth_ip``. The fast path is invoked strictly after the netns BPF flow dissector hook: if a BPF program is attached and returns a verdict other than BPF_FLOW_DISSECTOR_CONTINUE, __skb_flow_dissect() returns before flow_dissect_fast() is reached. Attached BPF dissectors therefore always take precedence and observe no behaviour change from this series. - What this does. Adds an opt-in fast-path for eth + IPv4 (IHL=5, not fragmented) + TCP/UDP and eth + IPv6 (no extension headers) + TCP/UDP. When the per-shape sysctl is enabled and a packet matches, the helper fills struct flow_keys byte-identically to the slow path and returns. On any miss it defers to the existing graph walk. Only the two standard dissectors are eligible (flow_keys_dissector and flow_keys_dissector_symmetric); custom dissectors and slow-path-only flags defer unconditionally. - How it gates. A new DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key), externed in , toggled by /proc/sys/net/flow_dissector/eth_ip with proc_do_static_key. When the sysctl is 0 the dispatcher's per-shape case is a forward not- taken JMP — same slow-path cost as today. When 1, the helper runs inline. - Namespace. The sysctl subtree is /proc/sys/net/flow_dissector/ (registered by a new late_initcall in net/core/flow_dissector.c via register_net_sysctl(&init_net, "net/flow_dissector", tbl) — same shape as net/mpls, net/bridge). - Forward declarations. flow_dissect_fast_ipv4 / _ipv6 are forward- declared at the top of the file (after includes). Subsequent patches add flow_dissect_fast_vlan etc. that call these helpers; anchoring the declarations at top of file makes the patch insertion order robust against kernel-version line-number drift. - Cost model. On a miss, dispatcher overhead is one ethertype switch (~5 cycles) plus the per-case static_branch test (~not-taken JMP when 0). On a hit, the straight-line body saves ~30 ns/skb of graph-walk cost on the common shape (Zen 2 / Cortex-A76; see the cover letter for the cross-uarch microbench table). - Output contract. Byte-identical with the slow path for the eligible shape. The IPv6 fast-path defers on any non-zero flow label: callers passing FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- skb_get_hash(), i.e. RPS/RFS, fq, fq_codel, cake -- make the slow path stop at a non-zero label, and dissectors requesting the label key write it; deferring on the label itself keeps byte-identity for every combination without threading flags into the helpers, and zero-label IPv6 stays on the fast path. Fragmented IPv4 packets defer. Custom dissectors defer, as do FLOW_DISSECTOR_F_* flags other than PARSE_1ST_FRAG and STOP_AT_FLOW_LABEL. Admitting STOP_AT_FLOW_LABEL is what makes the fast path reachable from skb_get_hash() at all -- the kernel's main dissection consumer. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 27 ++- include/net/flow_dissector.h | 5 + net/core/flow_dissector.c | 273 ++++++++++++++++++++++- 3 files changed, 302 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst index e586e17fc7a5..aac82169e019 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -460,7 +460,32 @@ historical importance. Default: 0 -2. /proc/sys/net/unix - Parameters for Unix domain sockets +2. /proc/sys/net/flow_dissector - Flow dissector fast-path per-shape sysctls +--------------------------------------------------------------------------- + +The flow dissector is invoked on every packet that needs a flow hash +(RPS, RFS, ECMP, sch_cake, cls_flower, ...). For common shapes +(Eth+IPv4/IPv6+TCP/UDP, plus per-shape extensions for VLAN/QinQ, +PPPoE, MPLS, IP-in-IP and GRE added by this series), an opt-in +fast-path bypasses the slow-path graph walk and writes +``struct flow_keys`` directly. Each shape is gated by its own static +key + sysctl so operators enable only what their deployment uses. + +All defaults are 0 (off). Flipping any of these on takes effect +immediately; no kernel rebuild needed. + +eth_ip +~~~~~~ + +Eth + IPv4 (IHL=5, no fragmentation) + TCP/UDP, and Eth + IPv6 (no +extension headers) + TCP/UDP. Output is byte-identical to the slow +path on the eligible shape; non-matching packets fall through to the +slow path unchanged. The dispatcher costs one not-taken JMP per call +when this is 0. + +Default: 0 + +3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- There is only one file in this directory. diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index ced79dc8e856..7d513a6cbb55 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -425,6 +425,11 @@ __be32 flow_get_u32_dst(const struct flow_keys *flow); extern struct flow_dissector flow_keys_dissector; extern struct flow_dissector flow_keys_basic_dissector; +/* Per-shape fast-path static-branch keys, toggled via + * /proc/sys/net/flow_dissector/. All default off. + */ +extern struct static_key_false flow_dissector_eth_ip_key; + /* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index b8c59209b460..88b1bd1cc4ca 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -37,6 +37,33 @@ #endif #include +/* Per-shape fast-path gates, one static_branch per shape so operators + * enable only what their deployment carries. All default off. + */ +DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key); +EXPORT_SYMBOL(flow_dissector_eth_ip_key); + +/* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ +#define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 + +/* Fast-path helper forward declarations. */ +static bool flow_dissect_fast_ipv4(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen); +static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen); + +/* One of the two dissectors the fast-path eligibility check admits; + * defined here so flow_dissect_fast() below can reference it (its keys + * and init live near the bottom of the file). + */ +static struct flow_dissector flow_keys_dissector_symmetric __read_mostly; + static void dissector_set_key(struct flow_dissector *flow_dissector, enum flow_dissector_key_id key_id) { @@ -1043,6 +1070,223 @@ static bool is_pppoe_ses_hdr_valid(const struct pppoe_hdr *hdr) return hdr->ver == 1 && hdr->type == 1 && hdr->code == 0; } +/* The IPv4 fast-path assumes a 20-byte IPv4 header (IHL == 5). The + * runtime IHL check below is the dynamic invariant; this static + * assertion is the compile-time one, in the unlikely case struct + * iphdr ever grows. + */ +static_assert(sizeof(struct iphdr) == 20); + +/* Straight-line eth + IPv4 (IHL=5, not fragmented) + TCP|UDP. Returns + * true when @target_container has been filled byte-identically to the + * slow path; false on any miss (caller falls back to the slow path). + */ +static bool flow_dissect_fast_ipv4(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + struct flow_dissector_key_addrs *key_addrs; + struct flow_dissector_key_basic *key_basic; + struct flow_dissector_key_ports *key_ports; + const struct iphdr *iph; + int thoff; + + if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4)) + return false; + + iph = (const struct iphdr *)((const u8 *)data + nhoff); + + if (unlikely(*(const u8 *)iph != FLOW_DIS_IPV4_VIHL_NOOPT)) + return false; + + if (unlikely(iph->frag_off & htons(IP_MF | IP_OFFSET))) + return false; + + if (unlikely(iph->protocol != IPPROTO_TCP && + iph->protocol != IPPROTO_UDP)) + return false; + + thoff = nhoff + (int)sizeof(*iph); + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS; + key_control->thoff = min_t(u16, thoff, + skb ? skb->len : hlen); + key_control->flags = 0; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC)) { + key_basic = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC, + target_container); + key_basic->n_proto = htons(ETH_P_IP); + key_basic->ip_proto = iph->protocol; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_IPV4_ADDRS)) { + key_addrs = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_IPV4_ADDRS, + target_container); + memcpy(&key_addrs->v4addrs.src, &iph->saddr, + sizeof(key_addrs->v4addrs.src)); + memcpy(&key_addrs->v4addrs.dst, &iph->daddr, + sizeof(key_addrs->v4addrs.dst)); + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS)) { + const __be32 *ports = (const __be32 *) + ((const u8 *)data + thoff); + + key_ports = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS, + target_container); + key_ports->ports = *ports; + } + + return true; +} + +/* Same as flow_dissect_fast_ipv4 but for a fixed 40-byte IPv6 header + * (no extension headers). The nexthdr check below holds the run-time + * invariant; the static_assert holds the compile-time one. + */ +static_assert(sizeof(struct ipv6hdr) == 40); + +static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + struct flow_dissector_key_addrs *key_addrs; + struct flow_dissector_key_basic *key_basic; + struct flow_dissector_key_ports *key_ports; + const struct ipv6hdr *iph; + int thoff; + + if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4)) + return false; + + iph = (const struct ipv6hdr *)((const u8 *)data + nhoff); + + if (unlikely((*(const u8 *)iph >> 4) != 6)) + return false; + + /* Any non-zero flow label defers, checked before everything else + * (including the tunnel descents below). Callers passing + * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL (skb_get_hash() -- so + * RPS/RFS, fq, cake) make the slow path stop at a non-zero label + * even on a tunnel outer, and dissectors requesting the label key + * write it; both diverge from the plain parse. Deferring on the + * label itself covers every combination without threading flags + * into the helpers, and zero-label IPv6 (the overwhelmingly + * common case) stays on the fast path. + */ + if (unlikely((iph->flow_lbl[0] & 0x0f) | + iph->flow_lbl[1] | iph->flow_lbl[2])) + return false; + + if (unlikely(iph->nexthdr != IPPROTO_TCP && + iph->nexthdr != IPPROTO_UDP)) + return false; + + thoff = nhoff + (int)sizeof(*iph); + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS; + key_control->thoff = min_t(u16, thoff, + skb ? skb->len : hlen); + key_control->flags = 0; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC)) { + key_basic = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC, + target_container); + key_basic->n_proto = htons(ETH_P_IPV6); + key_basic->ip_proto = iph->nexthdr; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS)) { + key_addrs = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS, + target_container); + memcpy(&key_addrs->v6addrs.src, &iph->saddr, + sizeof(key_addrs->v6addrs.src)); + memcpy(&key_addrs->v6addrs.dst, &iph->daddr, + sizeof(key_addrs->v6addrs.dst)); + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS)) { + const __be32 *ports = (const __be32 *) + ((const u8 *)data + thoff); + + key_ports = skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS, + target_container); + key_ports->ports = *ports; + } + + return true; +} + +/* Top-level dispatcher: eligibility check (only the two standard + * dissectors and flag subset) + per-proto switch with per-shape + * static_branch gating. Each case's branch is a forward not-taken JMP + * when its sysctl is 0 — matches the slow-path cost. + */ +static bool flow_dissect_fast(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 proto, int nhoff, int hlen, + unsigned int flags) +{ + if (flow_dissector != &flow_keys_dissector && + flow_dissector != &flow_keys_dissector_symmetric) + return false; + + /* skb_get_hash() -- behind RPS/RFS, fq, fq_codel, cake -- passes + * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL on every dissect, so the + * flag must be admitted or the fast path never runs for the + * kernel's main consumers. Its only semantic (stop early on a + * non-zero IPv6 flow label) is subsumed by the IPv6 fast path + * deferring on any non-zero label. Any other flag defers. + */ + if (flags & ~(unsigned int)(FLOW_DISSECTOR_F_PARSE_1ST_FRAG | + FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL)) + return false; + + switch (proto) { + case htons(ETH_P_IP): + if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) + return false; + return flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + case htons(ETH_P_IPV6): + if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) + return false; + return flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); + default: + return false; + } +} + /** * __skb_flow_dissect - extract the flow_keys struct and return it * @net: associated network namespace, derived from @skb if NULL @@ -1183,6 +1427,13 @@ bool __skb_flow_dissect(const struct net *net, rcu_read_unlock(); } + /* Opt-in fast-path; flow_dissect_fast() does the eligibility check + * and per-shape gating. + */ + if (flow_dissect_fast(skb, flow_dissector, target_container, + data, proto, nhoff, hlen, flags)) + return true; + if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_ETH_ADDRS)) { struct flow_dissector_key_eth_addrs *key_eth_addrs; @@ -1879,8 +2130,6 @@ void make_flow_keys_digest(struct flow_keys_digest *digest, } EXPORT_SYMBOL(make_flow_keys_digest); -static struct flow_dissector flow_keys_dissector_symmetric __read_mostly; - u32 __skb_get_hash_symmetric_net(const struct net *net, const struct sk_buff *skb) { struct flow_keys keys; @@ -2116,3 +2365,23 @@ static int __init init_default_flow_dissectors(void) return 0; } core_initcall(init_default_flow_dissectors); + +/* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */ +static struct ctl_table flow_dissector_sysctl_table[] = { + { + .procname = "eth_ip", + .data = &flow_dissector_eth_ip_key.key, + .maxlen = sizeof(flow_dissector_eth_ip_key), + .mode = 0644, + .proc_handler = proc_do_static_key, + }, +}; + +static int __init flow_dissector_sysctl_init(void) +{ + if (!register_net_sysctl(&init_net, "net/flow_dissector", + flow_dissector_sysctl_table)) + return -ENOMEM; + return 0; +} +late_initcall(flow_dissector_sysctl_init); -- 2.54.0