From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FB282F12AC for ; Thu, 9 Apr 2026 22:16:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775773017; cv=none; b=rDGAEUs+4DtTdwXDfMoG6qvnSdN4n7J3lghP6enZQCpA9m7VT7RXnarjVgBlZxuaImKD78BC9BK4BAViu2LDEt4RuqwMHU19DU2O3xQOdpWE47pRp+shmbIWnfyeRtfBlWUNvK5T4gERVbLnWrBnA5mvuUryGQt43Op+CEuW55Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775773017; c=relaxed/simple; bh=aXGljwR8wxR+ivNAI7rAzHeJHd0uentiXbGo58KW+6s=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=aA69kQlBUv+0cLXGUGtL2opL6/3ufhLREHGffhqP1gZe2I0VqE0rEZ61A4xR7AnFkhGrzFj0mPvUiJQ8PEEMmQpYEqOYJShqdgHKcqqdzZu2dYRu2lXZZKyESrVeOiRqh9cYoBOUmC35ebilibRdeTwhdVPi5SaMCnnR4QtJ4zE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Dh3UIER3; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=cAVY36R2; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Dh3UIER3; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=cAVY36R2; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Dh3UIER3"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="cAVY36R2"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Dh3UIER3"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="cAVY36R2" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5B7085BCEF; Thu, 9 Apr 2026 22:16:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1775773014; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=M317jivCkfvs0eEBD6JZyDGdl/WOEa3dpIHNc1mmByY=; b=Dh3UIER3obY/PMw0+GYs/ZP+4okhZgeI7fPyR+7n+XLFOUPAJ5j1HUXpSe5siq0y4IzFjP NDW4a935A/qUqjqv33nkCm2+PCbOvJH/GRSrt5B3BofDA1Ka/5Bx3iMwObV0Z8DbS+No7y 4icrJe1OzOO1KfhUjVLXKw4xZ7cKj64= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1775773014; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=M317jivCkfvs0eEBD6JZyDGdl/WOEa3dpIHNc1mmByY=; b=cAVY36R22tHdc6wQeEyo/XdlqTUWabIY9XVSOYuL8/jkMph1Q+iwELAnnYV/DIB7caU7sX XLDgQpmjIzKS4IAw== Authentication-Results: smtp-out2.suse.de; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=Dh3UIER3; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=cAVY36R2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1775773014; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=M317jivCkfvs0eEBD6JZyDGdl/WOEa3dpIHNc1mmByY=; b=Dh3UIER3obY/PMw0+GYs/ZP+4okhZgeI7fPyR+7n+XLFOUPAJ5j1HUXpSe5siq0y4IzFjP NDW4a935A/qUqjqv33nkCm2+PCbOvJH/GRSrt5B3BofDA1Ka/5Bx3iMwObV0Z8DbS+No7y 4icrJe1OzOO1KfhUjVLXKw4xZ7cKj64= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1775773014; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=M317jivCkfvs0eEBD6JZyDGdl/WOEa3dpIHNc1mmByY=; b=cAVY36R22tHdc6wQeEyo/XdlqTUWabIY9XVSOYuL8/jkMph1Q+iwELAnnYV/DIB7caU7sX XLDgQpmjIzKS4IAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 0A7D44A0B3; Thu, 9 Apr 2026 22:16:53 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id jW12N1Ul2GmhPgAAD6G6ig (envelope-from ); Thu, 09 Apr 2026 22:16:53 +0000 From: Gabriel Krisman Bertazi To: willemdebruijn.kernel@gmail.com, davem@davemloft.net, dsahern@kernel.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kuniyu@google.com Cc: horms@kernel.org, netdev@vger.kernel.org, Gabriel Krisman Bertazi Subject: [PATCH] udp: Force compute_score to always inline Date: Thu, 9 Apr 2026 18:15:32 -0400 Message-ID: <20260409221532.69090-1-krisman@suse.de> X-Mailer: git-send-email 2.52.0 Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Action: no action X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-1.51 / 50.00]; BAYES_HAM(-3.00)[100.00%]; SUSPICIOUS_RECIPS(1.50)[]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; FREEMAIL_TO(0.00)[gmail.com,davemloft.net,kernel.org,google.com,redhat.com]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.de:dkim,suse.de:mid,suse.de:email]; ARC_NA(0.00)[]; FUZZY_RATELIMITED(0.00)[rspamd.com]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TAGGED_RCPT(0.00)[]; RECEIVED_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:106:10:150:64:167:received]; RCPT_COUNT_SEVEN(0.00)[10]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.de:+]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FREEMAIL_ENVRCPT(0.00)[gmail.com] X-Rspamd-Queue-Id: 5B7085BCEF X-Spam-Flag: NO X-Spam-Score: -1.51 X-Spam-Level: Back in 2024 I reported a 7-12% regression on an iperf3 UDP loopback thoughput test that we traced to the extra overhead of calling compute_score on two places, introduced by commit f0ea27e7bfe1 ("udp: re-score reuseport groups when connected sockets are present"). At the time, I pointed out the overhead was caused by the multiple calls, associated with cpu-specific mitigations, and merged commit 50aee97d1511 ("udp: Avoid call to compute_score on multiple sites") to jump back explicitly, to force the rescore call in a single place. Recently though, we got another regression report against a newer distro version, which a team colleague traced back to the same root-cause. Turns out that once we updated to gcc-13, the compiler got smart enough to unroll the loop, undoing my previous mitigation. Let's bite the bullet and __always_inline compute_score on both ipv4 and ipv6 to prevent gcc from de-optimizing it again in the future. These functions are only called in two places each, udpX_lib_lookup1 and udpX_lib_lookup2, so the extra size shouldn't be a problem and it is hot enough to be very visible in profilings. In fact, with gcc13, forcing the inline will prevent gcc from unrolling the fix from commit 50aee97d1511, so we don't end up increasing udpX_lib_lookup2 at all. I haven't recollected the results myself, as I don't have access to the machine at the moment. But the same colleague reported 4.67% inprovement with this patch in the loopback benchmark, solving the regression report within noise margins. Fixes: 50aee97d1511 ("udp: Avoid call to compute_score on multiple sites") Signed-off-by: Gabriel Krisman Bertazi --- net/ipv4/udp.c | 8 ++++---- net/ipv6/udp.c | 9 +++++---- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 6c6b68a66dcd..e591e2ab0d7d 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -365,10 +365,10 @@ int udp_v4_get_port(struct sock *sk, unsigned short snum) return udp_lib_get_port(sk, snum, hash2_nulladdr); } -static int compute_score(struct sock *sk, const struct net *net, - __be32 saddr, __be16 sport, - __be32 daddr, unsigned short hnum, - int dif, int sdif) +static __always_inline int +compute_score(struct sock *sk, const struct net *net, + __be32 saddr, __be16 sport, __be32 daddr, + unsigned short hnum, int dif, int sdif) { int score; struct inet_sock *inet; diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 010b909275dd..889d229aad61 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -127,10 +127,11 @@ void udp_v6_rehash(struct sock *sk) udp_lib_rehash(sk, new_hash, new_hash4); } -static int compute_score(struct sock *sk, const struct net *net, - const struct in6_addr *saddr, __be16 sport, - const struct in6_addr *daddr, unsigned short hnum, - int dif, int sdif) +static __always_inline int +compute_score(struct sock *sk, const struct net *net, + const struct in6_addr *saddr, __be16 sport, + const struct in6_addr *daddr, unsigned short hnum, + int dif, int sdif) { int bound_dev_if, score; struct inet_sock *inet; -- 2.52.0