From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE90C7EE43 for ; Mon, 12 Jun 2023 11:01:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237530AbjFLLB0 (ORCPT ); Mon, 12 Jun 2023 07:01:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236810AbjFLLBB (ORCPT ); Mon, 12 Jun 2023 07:01:01 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0FBA468C for ; Mon, 12 Jun 2023 03:48:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5A6D462499 for ; Mon, 12 Jun 2023 10:48:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57BCDC433EF; Mon, 12 Jun 2023 10:48:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1686566882; bh=l9QkDw0Cwr3weMyaJYInXMHO17iSM7yO40tVekw8W+k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tHinHMcTOdB1XQKny89nyXjQ4nCHX64J2lSON4MazDdAcDZoj1mC/SYK5trh3L7xT mKP4ViTRUCCFg86ynhpvlfkNN+u6sBycTf/JKllAgrG1yhXUFzBWfnN0Eqnk+esJbB +p0UtgloLwTuefXl9sK2CWJKcfzO5/uN1TMxGpIg= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Eric Dumazet , Simon Horman , Kuniyuki Iwashima , "David S. Miller" , Sasha Levin Subject: [PATCH 6.3 050/160] rfs: annotate lockless accesses to RFS sock flow table Date: Mon, 12 Jun 2023 12:26:22 +0200 Message-ID: <20230612101717.321841918@linuxfoundation.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230612101715.129581706@linuxfoundation.org> References: <20230612101715.129581706@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Eric Dumazet [ Upstream commit 5c3b74a92aa285a3df722bf6329ba7ccf70346d6 ] Add READ_ONCE()/WRITE_ONCE() on accesses to the sock flow table. This also prevents a (smart ?) compiler to remove the condition in: if (table->ents[index] != newval) table->ents[index] = newval; We need the condition to avoid dirtying a shared cache line. Fixes: fec5e652e58f ("rfs: Receive Flow Steering") Signed-off-by: Eric Dumazet Reviewed-by: Simon Horman Reviewed-by: Kuniyuki Iwashima Signed-off-by: David S. Miller Signed-off-by: Sasha Levin --- include/linux/netdevice.h | 7 +++++-- net/core/dev.c | 6 ++++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 7db9f960221d3..2771aa046ab2a 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -760,8 +760,11 @@ static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, /* We only give a hint, preemption can change CPU under us */ val |= raw_smp_processor_id(); - if (table->ents[index] != val) - table->ents[index] = val; + /* The following WRITE_ONCE() is paired with the READ_ONCE() + * here, and another one in get_rps_cpu(). + */ + if (READ_ONCE(table->ents[index]) != val) + WRITE_ONCE(table->ents[index], val); } } diff --git a/net/core/dev.c b/net/core/dev.c index b3d8e74fcaf06..6d46eb0402ccd 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4471,8 +4471,10 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, u32 next_cpu; u32 ident; - /* First check into global flow table if there is a match */ - ident = sock_flow_table->ents[hash & sock_flow_table->mask]; + /* First check into global flow table if there is a match. + * This READ_ONCE() pairs with WRITE_ONCE() from rps_record_sock_flow(). + */ + ident = READ_ONCE(sock_flow_table->ents[hash & sock_flow_table->mask]); if ((ident ^ hash) & ~rps_cpu_mask) goto try_rps; -- 2.39.2