From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 652A2C433FE for ; Mon, 7 Dec 2020 10:00:42 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 048412310B for ; Mon, 7 Dec 2020 10:00:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 048412310B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nACVr47eLCwCubSB8BimGj9c00HSrmyvdJy+KF4YuKk=; b=sqXUUMjYFI1epQG+JyHYnTcK6 CWHBflVZEN8S4tSwZdkopEW8Ie5f//Hwc4nwpmgymw62Z31VkMyO9LozSrhBJI0uaVTKKcsmtZ3+J xekL22uhx87lTvToucJIHn2yahXNQGA5nmi/fvKCjOndUwBa+thIgn0QbsAO3FzmwRIrl9xRRy3wx UEB11TZWnNBKi+ouVVJm53HoxWynDG3t+eeiMmwoP05GclzQ6m8yqKFtgXGb604jBQ3WgWplZhCYM nt+HNLhxbpW2rRsc/EMW7Heb027LTIosHiQ3fMsOci2Q2jzj7kYEcxug/CDmCFU6kuahaXPFC9/l3 mBbnKPi3A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kmDIp-0002cJ-Gp; Mon, 07 Dec 2020 09:59:31 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kmDIm-0002b0-QM for linux-arm-kernel@lists.infradead.org; Mon, 07 Dec 2020 09:59:30 +0000 Received: from fraeml739-chm.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4CqJch6Qfnz67MN5; Mon, 7 Dec 2020 17:56:52 +0800 (CST) Received: from lhreml716-chm.china.huawei.com (10.201.108.67) by fraeml739-chm.china.huawei.com (10.206.15.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Mon, 7 Dec 2020 10:59:24 +0100 Received: from dggemi761-chm.china.huawei.com (10.1.198.147) by lhreml716-chm.china.huawei.com (10.201.108.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.2106.2; Mon, 7 Dec 2020 09:59:23 +0000 Received: from dggemi761-chm.china.huawei.com ([10.9.49.202]) by dggemi761-chm.china.huawei.com ([10.9.49.202]) with mapi id 15.01.1913.007; Mon, 7 Dec 2020 17:59:22 +0800 From: "Song Bao Hua (Barry Song)" To: Vincent Guittot Subject: RE: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters Thread-Topic: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters Thread-Index: AQHWx46tyJL5OgLpakCat9sEfhhBKKni9KEAgACKRLD//5RWAIAAjDiggAAA9pCAAKpoUIAARlmAgACFXCD//4SLAIAG0N6Q Date: Mon, 7 Dec 2020 09:59:21 +0000 Message-ID: References: <20201201025944.18260-1-song.bao.hua@hisilicon.com> <20201201025944.18260-3-song.bao.hua@hisilicon.com> <414fbd167b214452b925ac674575f0d6@hisilicon.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.126.201.26] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201207_045929_090008_687EBCC2 X-CRM114-Status: GOOD ( 29.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Mark Rutland , "Zengtao \(B\)" , Peter Zijlstra , Catalin Marinas , Jonathan Cameron , "Rafael J. Wysocki" , linux-kernel , Steven Rostedt , Dietmar Eggemann , Ben Segall , ACPI Devel Maling List , Ingo Molnar , Linuxarm , Mel Gorman , "xuwei \(O\)" , "gregkh@linuxfoundation.org" , Will Deacon , Valentin Schneider , LAK , "Cc: Len Brown" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > -----Original Message----- > From: Vincent Guittot [mailto:vincent.guittot@linaro.org] > Sent: Thursday, December 3, 2020 10:39 PM > To: Song Bao Hua (Barry Song) > Cc: Valentin Schneider ; Catalin Marinas > ; Will Deacon ; Rafael J. Wysocki > ; Cc: Len Brown ; > gregkh@linuxfoundation.org; Jonathan Cameron ; > Ingo Molnar ; Peter Zijlstra ; Juri > Lelli ; Dietmar Eggemann ; > Steven Rostedt ; Ben Segall ; Mel > Gorman ; Mark Rutland ; LAK > ; linux-kernel > ; ACPI Devel Maling List > ; Linuxarm ; xuwei (O) > ; Zengtao (B) > Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters > > On Thu, 3 Dec 2020 at 10:11, Song Bao Hua (Barry Song) > wrote: > > > > > > > > > -----Original Message----- > > > From: Vincent Guittot [mailto:vincent.guittot@linaro.org] > > > Sent: Thursday, December 3, 2020 10:04 PM > > > To: Song Bao Hua (Barry Song) > > > Cc: Valentin Schneider ; Catalin Marinas > > > ; Will Deacon ; Rafael J. Wysocki > > > ; Cc: Len Brown ; > > > gregkh@linuxfoundation.org; Jonathan Cameron > ; > > > Ingo Molnar ; Peter Zijlstra ; Juri > > > Lelli ; Dietmar Eggemann > ; > > > Steven Rostedt ; Ben Segall ; Mel > > > Gorman ; Mark Rutland ; LAK > > > ; linux-kernel > > > ; ACPI Devel Maling List > > > ; Linuxarm ; xuwei (O) > > > ; Zengtao (B) > > > Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters > > > > > > On Wed, 2 Dec 2020 at 21:58, Song Bao Hua (Barry Song) > > > wrote: > > > > > > > > > > > > > > Sorry. Please ignore this. I added some printk here while testing > > > > > one numa. Will update you the data in another email. > > > > > > > > Re-tested in one NUMA node(cpu0-cpu23): > > > > > > > > g=1 > > > > Running in threaded mode with 1 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 7.689 7.485 7.485 7.458 7.524 7.539 7.738 7.693 7.568 7.674=7.5853 > > > > w/ : 7.516 7.941 7.374 7.963 7.881 7.910 7.420 7.556 7.695 7.441=7.6697 > > > > w/ but dropped select_idle_cluster: > > > > 7.752 7.739 7.739 7.571 7.545 7.685 7.407 7.580 7.605 7.487=7.611 > > > > > > > > g=2 > > > > Running in threaded mode with 2 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 10.127 10.119 10.070 10.196 10.057 10.111 10.045 10.164 10.162 > > > > 9.955=10.1006 > > > > w/ : 9.694 9.654 9.612 9.649 9.686 9.734 9.607 9.842 9.690 9.710=9.6878 > > > > w/ but dropped select_idle_cluster: > > > > 9.877 10.069 9.951 9.918 9.947 9.790 9.906 9.820 9.863 9.906=9.9047 > > > > > > > > g=3 > > > > Running in threaded mode with 3 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 15.885 15.254 15.932 15.647 16.120 15.878 15.857 15.759 15.674 > > > > 15.721=15.7727 > > > > w/ : 14.974 14.657 13.969 14.985 14.728 15.665 15.191 14.995 14.946 > > > > 14.895=14.9005 > > > > w/ but dropped select_idle_cluster: > > > > 15.405 15.177 15.373 15.187 15.450 15.540 15.278 15.628 15.228 > > > 15.325=15.3591 > > > > > > > > g=4 > > > > Running in threaded mode with 4 groups using 40 file descriptors > > > > Each sender will pass 100000 messages of 100 bytes > > > > w/o: 20.014 21.025 21.119 21.235 19.767 20.971 20.962 20.914 21.090 > > > 21.090=20.8187 > > > > w/ : 20.331 20.608 20.338 20.445 20.456 20.146 20.693 20.797 21.381 > > > 20.452=20.5647 > > > > w/ but dropped select_idle_cluster: > > > > 19.814 20.126 20.229 20.350 20.750 20.404 19.957 19.888 20.226 > > > 20.562=20.2306 > > > > > > > > > > I assume that you have run this on v5.9 as previous tests. > > > > Yep > > > > > The results don't show any real benefit of select_idle_cluster() > > > inside a node whereas this is where we could expect most of the > > > benefit. We have to understand why we have such an impact on numa > > > tests only. > > > > There is a 4-5.5% increase while g=2 and g=3. > > my point was with vs without select_idle_cluster() but still having a > cluster domain level > In this case, the diff is -0.8% for g=1 +2.2% for g=2, +3% for g=3 and > -1.7% for g=4 > > > > > Regarding the huge increase in NUMA case, at the first beginning, I suspect > > we have wrong llc domain. For example, if cpu0's llc domain span > > cpu0-cpu47, then select_idle_cpu() is running in wrong range while > > it should run in cpu0-cpu23. > > > > But after printing the llc domain's span, I find it is completely right. > > Cpu0's llc span: cpu0-cpu23 > > Cpu24's llc span: cpu24-cpu47 > > Have you checked that the cluster mask was also correct ? > > > > > Maybe I need more trace data to figure out if select_idle_cpu() is running > > correctly. For example, maybe I can figure out if it is always returning -1, > > or it returns -1 very often? > > yes, could be interesting to check how often select_idle_cpu return -1 > > > > > Or do you have any idea? > > tracking migration across nod could help to understand too I set a bootargs mem=4G to do swapping test before working on cluster scheduler issue. but I forgot to remove the parameter. The huge increase on across-numa case can only be reproduced while i use this mem=4G cmdline which means numa1 has no memory. After removing the limitation, I can't reproduce the huge increase for two NUMAs any more. Guess select_idle_cluster() somehow workaround an scheduler issue for numa without memory. > > Vincent > > > > Thanks Barry _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel