From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8279C433F5 for ; Sat, 5 Feb 2022 17:18:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vBGmXLgZIxwY05Qo0K8pq2+xMlpx5i/1ywct+3X6Ogc=; b=jOkWVb7Uku9vnM Fl2Ynno4QdlY08HYXqSjpQaEIN5h99n317mKYtXbKTWyD0GnXVTvLN8BOKIBDoEnO4sKq/hgdod+7 aXUEEGqaPNhooeLFQGuJXl3hOks3llLDF+tuXZUOizDkdLgURBAaa/ik6psGjRXFDDtS8vmkgJ+ZJ LYAWZl7ANSrQegdkssbq1a50x+XoXKNLpG4UxF6iuyvUCfWa9UAZflbXyJyId5mUttvxgwSJS+xWF BhyXQZ/UuQMsW20Gwq74eweVQ0sA/8q8JcDCzFa3xK+CrXhUsklTym561eku2FjHuuml1pdeQ9gNb zI5q2d2SHkS5umXJ4y8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nGOfx-006rxf-Su; Sat, 05 Feb 2022 17:16:42 +0000 Received: from mga05.intel.com ([192.55.52.43]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nGOft-006rwk-P3 for linux-arm-kernel@lists.infradead.org; Sat, 05 Feb 2022 17:16:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1644081397; x=1675617397; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=3bioHvbnd8d7LeCkvwsSoHgNIMmlKm0ZRAuo5zZHHsI=; b=EqM0IzFYbz2lvDu83v7z3wFsSn5r6+2HLfAoUzLutRLG0Q2xKxMKMwPE bsyvT59nFc5OQMrIXf75SM9MM/Rbzoo9XubygDSUO2vrOe9VP/AItb1Fg 70N6ve41TPYJ1e/ZfCf1wX/T0G+2N6rjPbiC6rCQraHU5IJTtv/CKylNY DbFB/4P2EeqAE7eOyxD6yI0DQuElbvXGMpjoQw752ZuGTVJ79v4cDHCga 3O5xpRmI1VwepMh7ELnCw5mORLXkfxoLsqEdCxZyqfgFwn/ezE4YwVmNm kwBg5s/ckM6Z48wgycQ8TRgymTnEGPUhEVFAHIG+ZKkmzb7LjkazTIeq3 Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10248"; a="334923434" X-IronPort-AV: E=Sophos;i="5.88,345,1635231600"; d="scan'208";a="334923434" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2022 09:16:36 -0800 X-IronPort-AV: E=Sophos;i="5.88,345,1635231600"; d="scan'208";a="700001051" Received: from chenyu-desktop.sh.intel.com (HELO chenyu-desktop) ([10.239.158.186]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2022 09:16:31 -0800 Date: Sun, 6 Feb 2022 01:16:47 +0800 From: Chen Yu To: Tim Chen Cc: Barry Song <21cnbao@gmail.com>, Srikar Dronamraju , "Gautham R. Shenoy" , Yicong Yang , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , LKML , LAK , Dietmar Eggemann , Steven Rostedt , Ben Segall , Daniel Bristot de Oliveira , prime.zeng@huawei.com, Jonathan Cameron , ego@linux.vnet.ibm.com, Linuxarm , Barry Song , Guodong Xu Subject: Re: [PATCH v2 2/2] sched/fair: Scan cluster before scanning LLC in wake-up path Message-ID: <20220205171647.GA567616@chenyu-desktop> References: <20220128071337.GC618915@linux.vnet.ibm.com> <20220201093859.GE618915@linux.vnet.ibm.com> <20220204073317.GG618915@linux.vnet.ibm.com> <88e9287ef7a86f24999af00f90d6f122de024979.camel@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <88e9287ef7a86f24999af00f90d6f122de024979.camel@linux.intel.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220205_091637_867263_AB0BFCC0 X-CRM114-Status: GOOD ( 40.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Feb 04, 2022 at 09:41:21AM -0800, Tim Chen wrote: > On Fri, 2022-02-04 at 23:49 +1300, Barry Song wrote: > > On Fri, Feb 4, 2022 at 11:28 PM Barry Song <21cnbao@gmail.com> wrote: > > > On Fri, Feb 4, 2022 at 8:33 PM Srikar Dronamraju > > > wrote: > > > > * Barry Song <21cnbao@gmail.com> [2022-02-02 09:20:32]: > > > > > > > > > On Tue, Feb 1, 2022 at 10:39 PM Srikar Dronamraju > > > > > wrote: > > > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 07:40:15]: > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 8:13 PM Srikar Dronamraju > > > > > > > wrote: > > > > > > > > * Barry Song <21cnbao@gmail.com> [2022-01-28 09:21:08]: > > > > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 4:41 AM Gautham R. Shenoy > > > > > > > > > wrote: > > > > > > > > > > On Wed, Jan 26, 2022 at 04:09:47PM +0800, Yicong Yang > > > > > > > > > > wrote: > > > > > > > > > > > From: Barry Song > > > > > > > > > > > > > > > > > > I am sorry I didn't get your question. Currently the code > > > > > > > works as below: > > > > > > > if task A wakes up task B, and task A is in LLC0 and task B > > > > > > > is in LLC1. > > > > > > > we will scan the cluster of A before scanning the whole > > > > > > > LLC0, in this case, > > > > > > > cluster of A is the closest sibling, so it is the better > > > > > > > choice than other CPUs > > > > > > > which are in LLC0 but not in the cluster of A. > > > > > > > > > > > > Yes, this is right. > > > > > > > > > > > > > But we do scan all cpus of LLC0 > > > > > > > afterwards if we fail to find an idle CPU in the cluster. > > > > > > > > > > > > However my reading of the patch, before we can scan other > > > > > > clusters within > > > > > > the LLC (aka LLC0), we have a check in scan cluster which > > > > > > says > > > > > > > > > > > > /* Don't ping-pong tasks in and out cluster > > > > > > frequently */ > > > > > > if (cpus_share_resources(target, prev_cpu)) > > > > > > return target; > > > > > > > > > > > > My reading of this is, ignore other clusters (at this point, > > > > > > we know there > > > > > > are no idle CPUs in this cluster. We don't know if there are > > > > > > idle cpus in > > > > > > them or not) if the previous CPU and target CPU happen to be > > > > > > from the same > > > > > > cluster. This effectively means we are given preference to > > > > > > cache over idle > > > > > > CPU. > > > > > > > > > > Note we only ignore other cluster while prev_cpu and target are > > > > > in same > > > > > cluster. if the condition is false, we are not ignoring other > > > > > cpus. typically, > > > > > if waker is the target, and wakee is the prev_cpu, that means > > > > > if they are > > > > > already in one cluster, we don't stupidly spread them in > > > > > select_idle_cpu() path > > > > > as benchmark shows we are losing. so, yes, we are giving > > > > > preference to > > > > > cache over CPU. > > > > > > > > We already figured out that there are no idle CPUs in this > > > > cluster. So dont > > > > we gain performance by picking a idle CPU/core in the > > > > neighbouring cluster. > > > > If there are no idle CPU/core in the neighbouring cluster, then > > > > it does make > > > > sense to fallback on the current cluster. > > > > > > > > We may need to take into consideration the utilization and > load average for the source and target cluster to make > better decision of whether it is worth placing the > task in the next cluster. If the load of the target > cluster is too high, it is not worth pushing the task there. > > Those stats can be gathered during load balancing without adding > overhead in the hot task wakeup path. > > Chen Yu played around with cutting off the idle CPU search > in a LLC based on such stats and he saw some good > improvements over the default. > Yes, we used the sum of percpu util_avg to estimate if the LLC domain is overloaded. If it is too busy, skip searching for an idle cpu/core in that LLC domain. The util_avg is a metric of accumulated historic activity, which might be more accurate than instantaneous metrics(such as rq->nr_running) on calculating the probability of find an idle cpu. So far this change has shown some benefits in several microbenchmarks and OLTP benchmark when the system is quite busy. That change has introduced a per-LLC-domain flag to indicate whether the LLC domain is oveloaded, it seems that this flag could also be extended for cluster domain. Maybe I could post the draft patch to see if it would be helpful for this cluster patch serie. thanks, Chenyu > Tim > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel