From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5771AC43334 for ; Fri, 17 Jun 2022 17:17:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HmYnzUDA2I5DjHtNfalykJpes5ON/V4Ae7/C4IcDsts=; b=0Ouu7CDBF8MLua tc91BmKxpeHw44wy5JTlOCiYtpAbGuOigmXkYsmVllhuwhdS0XNRBaXYt7kEUsTv6I6IWbYsWUOre N6laFToTo/ZNjScFQ+60XzpqZxBnRkn83g8qJok0KP86J/6ijduWkjiwIZZu85PDIPl1QD1U8DI4+ vXMSmsv+kJgqKCQNXz7kSK9sIoL82bUNACcL1UvINYtyy/YEBB6rsBZRt/ukMjLUr3TNmLmrjLFzY LX27phPFicsOC/6VXvvS3NEth0a9AvrdsRWu0+wLcF6ijm09CybDcTRcqd2ZSyfDtNOIf6lJA84ml QNwLDNa7EkoMTg5ecq6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o2FZm-008P8e-V9; Fri, 17 Jun 2022 17:16:07 +0000 Received: from mga06b.intel.com ([134.134.136.31] helo=mga06.intel.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o2FZj-008P6i-2b for linux-arm-kernel@lists.infradead.org; Fri, 17 Jun 2022 17:16:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655486163; x=1687022163; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=ojZXvlomwwAk5LBD57zzpFvYqmoQfWAuuImQZsSw1Hk=; b=WzhQHROhjA37X+lAFONXJ+l1uT3hU6RfVOjN/EDsPWNw83ttD+pPjQoF QOHSXTrb+c/gRYKg5Eqd5jZqgbBFH9Mz4XnUlJRzGBfOb9TMqttnp9qlt fLzVpPmXdJ/pqccFJK5aHW0CWb0Q4Ts5FNJd+QJ3L6dM1D4TwPiN9Lnow JrQ/OHXpHYMx5rT6+ldMqiwHCyuU5gBrhBj3x0LUep7jjLkIWNR0jZVH+ fRactfYSF47dAWOZHmNtpPsyNbz7O5w5qMZRyieYptHBbbHu3PLPsfHMn XdpHRkDQTFmY/DU4gn642KmQIK/xtXIAwhkKVa8HfBr/5MEXWJYRKAyyN w==; X-IronPort-AV: E=McAfee;i="6400,9594,10380"; a="341213674" X-IronPort-AV: E=Sophos;i="5.92,306,1650956400"; d="scan'208";a="341213674" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jun 2022 09:50:25 -0700 X-IronPort-AV: E=Sophos;i="5.92,306,1650956400"; d="scan'208";a="613597090" Received: from schen9-mobl.amr.corp.intel.com ([10.212.204.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Jun 2022 09:50:24 -0700 Message-ID: Subject: Re: [PATCH v4 1/2] sched: Add per_cpu cluster domain info and cpus_share_resources API From: Tim Chen To: K Prateek Nayak , Yicong Yang , Yicong Yang , peterz@infradead.org, mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, gautham.shenoy@amd.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, prime.zeng@huawei.com, jonathan.cameron@huawei.com, ego@linux.vnet.ibm.com, srikar@linux.vnet.ibm.com, linuxarm@huawei.com, 21cnbao@gmail.com, guodong.xu@linaro.org, hesham.almatary@huawei.com, john.garry@huawei.com, shenyang39@huawei.com, feng.tang@intel.com Date: Fri, 17 Jun 2022 09:50:24 -0700 In-Reply-To: <6bf4f032-7d07-d4a4-4f5a-28f3871131c0@amd.com> References: <20220609120622.47724-1-yangyicong@hisilicon.com> <20220609120622.47724-2-yangyicong@hisilicon.com> <81fbcadb-a58d-2cef-9c05-154555ec1d68@huawei.com> <6bf4f032-7d07-d4a4-4f5a-28f3871131c0@amd.com> User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220617_101603_304040_63739B3C X-CRM114-Status: GOOD ( 22.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 2022-06-17 at 17:50 +0530, K Prateek Nayak wrote: > > > -- > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index e9f3dc6dcbf4..97a3895416ab 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -1750,12 +1750,12 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag) > return sd; > } > > +DECLARE_PER_CPU(struct sched_domain __rcu *, sd_cluster); > +DECLARE_PER_CPU(int, sd_share_id); > DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc); > DECLARE_PER_CPU(int, sd_llc_size); > DECLARE_PER_CPU(int, sd_llc_id); > -DECLARE_PER_CPU(int, sd_share_id); > DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); > -DECLARE_PER_CPU(struct sched_domain __rcu *, sd_cluster); > DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa); > DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); > DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); > -- > > The System-map of each kernel is as follows: > > - On "tip" > > 0000000000020518 D sd_asym_cpucapacity > 0000000000020520 D sd_asym_packing > 0000000000020528 D sd_numa > 0000000000020530 D sd_llc_shared > 0000000000020538 D sd_llc_id > 000000000002053c D sd_llc_size > -------------------------------------------- 64B Cacheline Boundary > 0000000000020540 D sd_llc > > - On "tip + Patch 1 only" and "tip + both patches" > > 0000000000020518 D sd_asym_cpucapacity > 0000000000020520 D sd_asym_packing > 0000000000020528 D sd_numa > 0000000000020530 D sd_cluster <----- > 0000000000020538 D sd_llc_shared > -------------------------------------------- 64B Cacheline Boundary > 0000000000020540 D sd_share_id <----- > 0000000000020544 D sd_llc_id > 0000000000020548 D sd_llc_size > 0000000000020550 D sd_llc > > > - On "tip + both patches (Move declaration to top)" > > 0000000000020518 D sd_asym_cpucapacity > 0000000000020520 D sd_asym_packing > 0000000000020528 D sd_numa > 0000000000020530 D sd_llc_shared > 0000000000020538 D sd_llc_id > 000000000002053c D sd_llc_size > -------------------------------------------- 64B Cacheline Boundary > 0000000000020540 D sd_llc Wonder if it will help to try keep sd_llc and sd_llc_size into the same cache line. They are both used in the wake up path. > 0000000000020548 D sd_share_id <----- > 0000000000020550 D sd_cluster <----- > > > Or change the layout a bit to see if there's any difference, > > like: > > > > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); > > DEFINE_PER_CPU(int, sd_llc_size); > > DEFINE_PER_CPU(int, sd_llc_id); > > DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); > > +DEFINE_PER_CPU(int, sd_share_id); > > +DEFINE_PER_CPU(struct sched_domain __rcu *, sd_cluster); > > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); > > DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); > > > > I need to further look into it and have some tests on a SMT machine. Would you mind to share > > the kernel config as well? I'd like to compare the config as well. > > I've attached the kernel config used to build the test kernel > to this mail. > > > Thanks, > > Yicong > > We are trying to debug the issue using perf and find an optimal > arrangement of the per cpu declarations to get the relevant data > used in the wakeup path on the same 64B cache line. A check of perf c2c profile difference between tip and the move new declarations to the top case could be useful. It may give some additional clues of possibel false sharing issues. Tim > > We'll keep you posted of out finding. Let me know if you need > anything else in the meantime. > -- > Thanks and Regards, > Prateek _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel