From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 880FBFA372C for ; Fri, 8 Nov 2019 19:11:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5E15D20650 for ; Fri, 8 Nov 2019 19:11:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1573240317; bh=9+JX3QyceqFBH2ZujSCe4zG1aCkIeICt4OQm4sNIQHw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=MxqX017wR3sLrrydqEfuK3B8z2ofoDR/Ltszsa3551cTGogFcQ6jqCCNQBZ2oI5ei UH1UHOCUC+J+JaUcrdmsjwcouZlurkS/3VOVVW5x08igHgKIkw7v90lR5HytI/T6xB Kv2Wi4TkDtJ7nGolqsLhYbV0efQM7LN4lfr0NU6Q= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390919AbfKHTL4 (ORCPT ); Fri, 8 Nov 2019 14:11:56 -0500 Received: from mail.kernel.org ([198.145.29.99]:42296 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391078AbfKHTKK (ORCPT ); Fri, 8 Nov 2019 14:10:10 -0500 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0F0A420673; Fri, 8 Nov 2019 19:10:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1573240209; bh=9+JX3QyceqFBH2ZujSCe4zG1aCkIeICt4OQm4sNIQHw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tBrquKBTD3smgRtOBjRXcMmfyG4F1RP9MQp1t1I1fzkuGdJfxhLFtpg0OIkXp7w0a jI0WmntKmW7hkpGSRSoFt/KwHtdRlXW0L6d7Tc2gMyXFhO76FYaNM7IpFdZZFX4coz GPpkVHztQPQb6XXqmulyQY5zLsSQFLLBRePBVVpM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Eric Dumazet , kernel test robot , "David S. Miller" Subject: [PATCH 5.3 127/140] net: reorder struct net fields to avoid false sharing Date: Fri, 8 Nov 2019 19:50:55 +0100 Message-Id: <20191108174912.683710821@linuxfoundation.org> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191108174900.189064908@linuxfoundation.org> References: <20191108174900.189064908@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Eric Dumazet [ Upstream commit 2a06b8982f8f2f40d03a3daf634676386bd84dbc ] Intel test robot reported a ~7% regression on TCP_CRR tests that they bisected to the cited commit. Indeed, every time a new TCP socket is created or deleted, the atomic counter net->count is touched (via get_net(net) and put_net(net) calls) So cpus might have to reload a contended cache line in net_hash_mix(net) calls. We need to reorder 'struct net' fields to move @hash_mix in a read mostly cache line. We move in the first cache line fields that can be dirtied often. We probably will have to address in a followup patch the __randomize_layout that was added in linux-4.13, since this might break our placement choices. Fixes: 355b98553789 ("netns: provide pure entropy for net_hash_mix()") Signed-off-by: Eric Dumazet Reported-by: kernel test robot Signed-off-by: David S. Miller Signed-off-by: Greg Kroah-Hartman --- include/net/net_namespace.h | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) --- a/include/net/net_namespace.h +++ b/include/net/net_namespace.h @@ -52,6 +52,9 @@ struct bpf_prog; #define NETDEV_HASHENTRIES (1 << NETDEV_HASHBITS) struct net { + /* First cache line can be often dirtied. + * Do not place here read-mostly fields. + */ refcount_t passive; /* To decide when the network * namespace should be freed. */ @@ -60,7 +63,13 @@ struct net { */ spinlock_t rules_mod_lock; - u32 hash_mix; + unsigned int dev_unreg_count; + + unsigned int dev_base_seq; /* protected by rtnl_mutex */ + int ifindex; + + spinlock_t nsid_lock; + atomic_t fnhe_genid; struct list_head list; /* list of network namespaces */ struct list_head exit_list; /* To linked to call pernet exit @@ -76,11 +85,11 @@ struct net { #endif struct user_namespace *user_ns; /* Owning user namespace */ struct ucounts *ucounts; - spinlock_t nsid_lock; struct idr netns_ids; struct ns_common ns; + struct list_head dev_base_head; struct proc_dir_entry *proc_net; struct proc_dir_entry *proc_net_stat; @@ -93,12 +102,14 @@ struct net { struct uevent_sock *uevent_sock; /* uevent socket */ - struct list_head dev_base_head; struct hlist_head *dev_name_head; struct hlist_head *dev_index_head; - unsigned int dev_base_seq; /* protected by rtnl_mutex */ - int ifindex; - unsigned int dev_unreg_count; + /* Note that @hash_mix can be read millions times per second, + * it is critical that it is on a read_mostly cache line. + */ + u32 hash_mix; + + struct net_device *loopback_dev; /* The loopback */ /* core fib_rules */ struct list_head rules_ops; @@ -106,7 +117,6 @@ struct net { struct list_head fib_notifier_ops; /* Populated by * register_pernet_subsys() */ - struct net_device *loopback_dev; /* The loopback */ struct netns_core core; struct netns_mib mib; struct netns_packet packet; @@ -171,7 +181,6 @@ struct net { struct netns_xdp xdp; #endif struct sock *diag_nlsk; - atomic_t fnhe_genid; } __randomize_layout; #include