From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53E45C433FE for ; Tue, 8 Dec 2020 16:16:32 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1618C23B01 for ; Tue, 8 Dec 2020 16:16:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1618C23B01 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UVr6MGsMVEiUI0M5JJ58U1IfWdvNtAg0QrpFupcipL4=; b=2KRfZGJUv/N+sgqGfS/146hEX B5vlU72cyiUmI6/wbHP7BXfMzoNCbzqY5Tbxvd8Yg12NtsSOZsaoQJ9+AfkTF49cMPaTUi07nZwgS qTRYJt6BuH6Jv6z9SxsYqrxUL96eYTm3juKcn5juyawbBii3xTk2oN1wOdkBk7kvsfuQ1yZ0tFv92 5kXdUzws1FMCchbajKkc2C9su8AogiDQW8IKXZNrAQvVSnrbArt+CdX5Blf1afELODCX9OpLHuuz8 3HPzIV4EZhFdlkfRkUgB0/lwIAQ2/wMYH8xkvEZtCMm7esCC9KvrZczPglQ6iR5aJwRn/GjoKo74k NZmoJXdog==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kmfdi-0006gV-QD; Tue, 08 Dec 2020 16:14:58 +0000 Received: from mail-lj1-x243.google.com ([2a00:1450:4864:20::243]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kmfda-0006do-A3 for linux-arm-kernel@lists.infradead.org; Tue, 08 Dec 2020 16:14:53 +0000 Received: by mail-lj1-x243.google.com with SMTP id y22so5515521ljn.9 for ; Tue, 08 Dec 2020 08:14:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jvK/rcBZ+ncoq0z4m0U1M9RN+o8FAfMKBetL37kSxrw=; b=kyRy8HCAMatgEuN8e9NTUBjdlAHxTLViLrlRLxAKVmGBr6I/Zo46rojBGVq6WliZ9D I2/Vqfh+vQczS0j7bPP4nibN18cOx4iDO3r+6dnDbGfp1wxGKXWZX8rjLtPX6mL7woJE Cn3TS4FWsE24BabWAvSKfki4qVOL15vsHDIc7IWzhimo+EiiD+Sw4o1XxlTd4l23rFFm UJi85xP7eHWTU7v12jct1B0wVSenqTaoIfhr86+GqRwa3uG1x6rTH/+3zqdS0rSFL7JY 4u4QK8H/lFs+78UaxsEDSP5YmUQ1doO/I0VX0dsp7sOHD9cwu8x0gQ0zi/UWFpU7Bgqq rpOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jvK/rcBZ+ncoq0z4m0U1M9RN+o8FAfMKBetL37kSxrw=; b=XbGOOb1geJV2nVBS2wfJ9qOQQHYWJkgcvEmUbk/rS0RKlCUww0GPTqs0I7pr5eHKzI gj/T0hvS61DO7HOzwhl7SQVhxxIy5bC72c0U0sPDcBTxiWiz4MKa/V0FM8kV9Xsc2R0L Y0VJomECwZQ7wRFfAH0I/c+J+lWnOFFJChfCQ7pNCxixhmzk414xzUIziHmqdqo63hte MJJWzPCLOcVsulyaHPtY37/Rd4V6TK4OSjK6N78KZaAC8Vkpmome544S9xGjRrk6nTjA deFYHBOZn7Ho2ypAWH5Iha7t753F2Go9yb+sA/Bw8E6brzBcSJelScdkkzJIS6t/Iv7J px8g== X-Gm-Message-State: AOAM530vetxyTzMNKSvr9kDs5IWgzbhIbKHpklmFZqF3bYuBYC4GYUpn HU02LqQWwKMYDX9L/rU7/JXaXddQohvCvsuKKTbzUg== X-Google-Smtp-Source: ABdhPJw4vO+93cI9tySLlFVtUK2VrvMJE/CCZYBgBNIm4Lee9p2hOliys1WxnhUlsOY/sI7R0ixrQj6dQ8FO5C8t1/g= X-Received: by 2002:a2e:7d0f:: with SMTP id y15mr10991904ljc.401.1607444088783; Tue, 08 Dec 2020 08:14:48 -0800 (PST) MIME-Version: 1.0 References: <20201208153501.1467-1-mgorman@techsingularity.net> <20201208153501.1467-4-mgorman@techsingularity.net> In-Reply-To: <20201208153501.1467-4-mgorman@techsingularity.net> From: Vincent Guittot Date: Tue, 8 Dec 2020 17:14:37 +0100 Message-ID: Subject: Re: [PATCH 3/4] sched/fair: Do not replace recent_used_cpu with the new target To: Mel Gorman X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201208_111450_529266_037D0795 X-CRM114-Status: GOOD ( 29.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Barry Song , Peter Ziljstra , Aubrey Li , LKML , Ingo Molnar , Valentin Schneider , Linux-ARM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 8 Dec 2020 at 16:35, Mel Gorman wrote: > > After select_idle_sibling, p->recent_used_cpu is set to the > new target. However on the next wakeup, prev will be the same as > recent_used_cpu unless the load balancer has moved the task since the last > wakeup. It still works, but is less efficient than it can be after all > the changes that went in since that reduce unnecessary migrations, load > balancer changes etc. This patch preserves recent_used_cpu for longer. > > With tbench on a 2-socket CascadeLake machine, 80 logical CPUs, HT enabled > > 5.10.0-rc6 5.10.0-rc6 > baseline-v2 altrecent-v2 > Hmean 1 508.39 ( 0.00%) 502.05 * -1.25%* > Hmean 2 986.70 ( 0.00%) 983.65 * -0.31%* > Hmean 4 1914.55 ( 0.00%) 1920.24 * 0.30%* > Hmean 8 3702.37 ( 0.00%) 3663.96 * -1.04%* > Hmean 16 6573.11 ( 0.00%) 6545.58 * -0.42%* > Hmean 32 10142.57 ( 0.00%) 10253.73 * 1.10%* > Hmean 64 14348.40 ( 0.00%) 12506.31 * -12.84%* > Hmean 128 21842.59 ( 0.00%) 21967.13 * 0.57%* > Hmean 256 20813.75 ( 0.00%) 21534.52 * 3.46%* > Hmean 320 20684.33 ( 0.00%) 21070.14 * 1.87%* > > The different was marginal except for 64 threads which showed in the > baseline that the result was very unstable where as the patch was much > more stable. This is somewhat machine specific as on a separate 80-cpu > Broadwell machine the same test reported. > > 5.10.0-rc6 5.10.0-rc6 > baseline-v2 altrecent-v2 > Hmean 1 310.36 ( 0.00%) 291.81 * -5.98%* > Hmean 2 340.86 ( 0.00%) 547.22 * 60.54%* > Hmean 4 912.29 ( 0.00%) 1063.21 * 16.54%* > Hmean 8 2116.40 ( 0.00%) 2103.60 * -0.60%* > Hmean 16 4232.90 ( 0.00%) 4362.92 * 3.07%* > Hmean 32 8442.03 ( 0.00%) 8642.10 * 2.37%* > Hmean 64 11733.91 ( 0.00%) 11473.66 * -2.22%* > Hmean 128 17727.24 ( 0.00%) 16784.23 * -5.32%* > Hmean 256 16089.23 ( 0.00%) 16110.79 * 0.13%* > Hmean 320 15992.60 ( 0.00%) 16071.64 * 0.49%* > > schedstats were not used in this series but from an earlier debugging > effort, the schedstats after the test run were as follows; > > Ops SIS Search 5653107942.00 5726545742.00 > Ops SIS Domain Search 3365067916.00 3319768543.00 > Ops SIS Scanned 112173512543.00 99194352541.00 > Ops SIS Domain Scanned 109885472517.00 96787575342.00 > Ops SIS Failures 2923185114.00 2950166441.00 > Ops SIS Recent Used Hit 56547.00 118064916.00 > Ops SIS Recent Used Miss 1590899250.00 354942791.00 > Ops SIS Recent Attempts 1590955797.00 473007707.00 > Ops SIS Search Efficiency 5.04 5.77 > Ops SIS Domain Search Eff 3.06 3.43 > Ops SIS Fast Success Rate 40.47 42.03 > Ops SIS Success Rate 48.29 48.48 > Ops SIS Recent Success Rate 0.00 24.96 > > First interesting point is the ridiculous number of times runqueues are > enabled -- almost 97 billion times over the course of 40 minutes > > With the patch, "Recent Used Hit" is over 2000 times more likely to > succeed. The failure rate also increases by quite a lot but the cost is > marginal even if the "Fast Success Rate" only increases by 2% overall. What > cannot be observed from these stats is where the biggest impact as these > stats cover low utilisation to over saturation. > > If graphed over time, the graphs show that the sched domain is only > scanned at negligible rates until the machine is fully busy. With > low utilisation, the "Fast Success Rate" is almost 100% until the > machine is fully busy. For 320 clients, the success rate is close to > 0% which is unsurprising. > > Signed-off-by: Mel Gorman Reviewed-by: Vincent Guittot > --- > kernel/sched/fair.c | 9 +-------- > 1 file changed, 1 insertion(+), 8 deletions(-) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index 5c41875aec23..413d895bbbf8 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -6277,17 +6277,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) > > /* Check a recently used CPU as a potential idle candidate: */ > recent_used_cpu = p->recent_used_cpu; > + p->recent_used_cpu = prev; > if (recent_used_cpu != prev && > recent_used_cpu != target && > cpus_share_cache(recent_used_cpu, target) && > (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && > cpumask_test_cpu(p->recent_used_cpu, p->cpus_ptr) && > asym_fits_capacity(task_util, recent_used_cpu)) { > - /* > - * Replace recent_used_cpu with prev as it is a potential > - * candidate for the next wake: > - */ > - p->recent_used_cpu = prev; > return recent_used_cpu; > } > > @@ -6768,9 +6764,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) > } else if (wake_flags & WF_TTWU) { /* XXX always ? */ > /* Fast path */ > new_cpu = select_idle_sibling(p, prev_cpu, new_cpu); > - > - if (want_affine) > - current->recent_used_cpu = cpu; > } > rcu_read_unlock(); > > -- > 2.26.2 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel