From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6088DC433EF for ; Fri, 4 Feb 2022 17:43:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ckp89ZTKTW0F5QBX/Rgyilbh6WTJTu4j3yNQcVs6Y9M=; b=MdiSrMx4C0FN7K cVMchWTRaSQdX4FlHDqyTNBsQtFTIimJmlPAN84ueMdhFE1BsuU0wUWldkQvSeO8y1E3vdRdyODFf UITE/qS3GC4u5dvyKO4kyhX8VB3+JThpKQ37JCSjqtyBupHXnnwE5K1dKHO2xrC+a3ZloGD0LKLLt vdCYqmEqjYV++TZdnqTkMl4ALCdOQVPn4Sa5cG2lKZR3XFhkfUErkd5Gl6zSwKI9090TXKy2F7vRO kz005i97wdt9t7MLuEJPqR3qfftyb4YwOanqC2sLlwl0tPUAsaXyi62DrZsWLBnCebDgu5wAgLpsM klbk74dmohSE97et0iFQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nG2ah-0052gr-Tp; Fri, 04 Feb 2022 17:41:48 +0000 Received: from mga05.intel.com ([192.55.52.43]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nG2ad-0052gH-PT for linux-arm-kernel@lists.infradead.org; Fri, 04 Feb 2022 17:41:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643996503; x=1675532503; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=HNAWq9SbvrJ7bhHuSuHGWfwTAaW9LozmAE/xxrXCg0c=; b=iK5/r1L9yL+Rl6xPuq9iRqJTtV529c0e0uwCLgTXLBUIvbmSWhhCV64J qsTWB+UE7nhMHmCAu64LLLkie7mMtQNrh48zpuG8D7BLqQ8MIFfjV1PDm cvtRh4SOvFeMblkTZUd1IRigkX+Z8OyU+Pu6L90vNI+E+kNYoTJQZVLc8 n9P3lhr7pdgy7n8buzKYRWC2qz6nG+ChZHe11lvJHLwpN6ANhImBVmPL5 LHlDLcdLiyfbfmOPlsKe6u20g/MbFTjr4aovsQdbo4ke8DZA5iefqwpZP Xl/9S71V6L6JpJtU+WUO9NhjLoyzr8hd1dx6ZY9epd9lZdHkk/eex9xZq w==; X-IronPort-AV: E=McAfee;i="6200,9189,10248"; a="334805089" X-IronPort-AV: E=Sophos;i="5.88,343,1635231600"; d="scan'208";a="334805089" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2022 09:41:21 -0800 X-IronPort-AV: E=Sophos;i="5.88,343,1635231600"; d="scan'208";a="480903004" Received: from schen9-mobl.amr.corp.intel.com ([10.251.2.218]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Feb 2022 09:41:21 -0800 Message-ID: <88e9287ef7a86f24999af00f90d6f122de024979.camel@linux.intel.com> Subject: Re: [PATCH v2 2/2] sched/fair: Scan cluster before scanning LLC in wake-up path From: Tim Chen To: Barry Song <21cnbao@gmail.com>, Srikar Dronamraju Cc: "Gautham R. Shenoy" , Yicong Yang , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , LKML , LAK , Dietmar Eggemann , Steven Rostedt , Ben Segall , Daniel Bristot de Oliveira , prime.zeng@huawei.com, Jonathan Cameron , ego@linux.vnet.ibm.com, Linuxarm , Barry Song , Guodong Xu , yu.c.chen@intel.com Date: Fri, 04 Feb 2022 09:41:21 -0800 In-Reply-To: References: <20220126080947.4529-1-yangyicong@hisilicon.com> <20220126080947.4529-3-yangyicong@hisilicon.com> <20220128071337.GC618915@linux.vnet.ibm.com> <20220201093859.GE618915@linux.vnet.ibm.com> <20220204073317.GG618915@linux.vnet.ibm.com> User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220204_094143_871456_079E18D8 X-CRM114-Status: GOOD ( 36.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 2022-02-04 at 23:49 +1300, Barry Song wrote: > On Fri, Feb 4, 2022 at 11:28 PM Barry Song <21cnbao@gmail.com> wrote: > > On Fri, Feb 4, 2022 at 8:33 PM Srikar Dronamraju > > wrote: > > > * Barry Song <21cnbao@gmail.com> [2022-02-02 09:20:32]: > > > > > > > On Tue, Feb 1, 2022 at 10:39 PM Srikar Dronamraju > > > > wrote: > > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 07:40:15]: > > > > > > > > > > > On Fri, Jan 28, 2022 at 8:13 PM Srikar Dronamraju > > > > > > wrote: > > > > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 09:21:08]: > > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 4:41 AM Gautham R. Shenoy > > > > > > > > wrote: > > > > > > > > > On Wed, Jan 26, 2022 at 04:09:47PM +0800, Yicong Yang > > > > > > > > > wrote: > > > > > > > > > > From: Barry Song > > > > > > > > > > > > > > > > I am sorry I didn't get your question. Currently the code > > > > > > works as below: > > > > > > if task A wakes up task B, and task A is in LLC0 and task B > > > > > > is in LLC1. > > > > > > we will scan the cluster of A before scanning the whole > > > > > > LLC0, in this case, > > > > > > cluster of A is the closest sibling, so it is the better > > > > > > choice than other CPUs > > > > > > which are in LLC0 but not in the cluster of A. > > > > > > > > > > Yes, this is right. > > > > > > > > > > > But we do scan all cpus of LLC0 > > > > > > afterwards if we fail to find an idle CPU in the cluster. > > > > > > > > > > However my reading of the patch, before we can scan other > > > > > clusters within > > > > > the LLC (aka LLC0), we have a check in scan cluster which > > > > > says > > > > > > > > > > /* Don't ping-pong tasks in and out cluster > > > > > frequently */ > > > > > if (cpus_share_resources(target, prev_cpu)) > > > > > return target; > > > > > > > > > > My reading of this is, ignore other clusters (at this point, > > > > > we know there > > > > > are no idle CPUs in this cluster. We don't know if there are > > > > > idle cpus in > > > > > them or not) if the previous CPU and target CPU happen to be > > > > > from the same > > > > > cluster. This effectively means we are given preference to > > > > > cache over idle > > > > > CPU. > > > > > > > > Note we only ignore other cluster while prev_cpu and target are > > > > in same > > > > cluster. if the condition is false, we are not ignoring other > > > > cpus. typically, > > > > if waker is the target, and wakee is the prev_cpu, that means > > > > if they are > > > > already in one cluster, we don't stupidly spread them in > > > > select_idle_cpu() path > > > > as benchmark shows we are losing. so, yes, we are giving > > > > preference to > > > > cache over CPU. > > > > > > We already figured out that there are no idle CPUs in this > > > cluster. So dont > > > we gain performance by picking a idle CPU/core in the > > > neighbouring cluster. > > > If there are no idle CPU/core in the neighbouring cluster, then > > > it does make > > > sense to fallback on the current cluster. > > > > We may need to take into consideration the utilization and load average for the source and target cluster to make better decision of whether it is worth placing the task in the next cluster. If the load of the target cluster is too high, it is not worth pushing the task there. Those stats can be gathered during load balancing without adding overhead in the hot task wakeup path. Chen Yu played around with cutting off the idle CPU search in a LLC based on such stats and he saw some good improvements over the default. Tim _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel