From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E68A6C63777 for ; Thu, 3 Dec 2020 09:12:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 613FB20709 for ; Thu, 3 Dec 2020 09:12:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 613FB20709 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=T6XWx6y/PC6ZuYBxHis22f+MdU9xq+atfoh8XsN9sfw=; b=mRFQwN7Ntlh+hWeBxCOpezUTh qjfcuBkin4CDTlP4nMbTiQo6NPYngouL2KtCzjFH6nmcr5nNnUPJAEuEH+XwYmExpKxQYg8Au98uB h2Lb+jlz//ljFSJUBQ9eHTy6MaKQayLeiOvOcJR2xBApoq5RvPnfI5At7T5Z2d17PxAzCz9b5Ynrz uPI7X5Q87pTjf3xn2hEUWc9n6JR/77r8TOCbQ0PTu5wP5grpXkFrcihsf3HqD+LFUa4q2xd0gN7m6 iMj36VP7I3LlaaxWoetxWxn6pEyP8RbPw4Ava5iOoeKubqvqFTzNY5uGRPUS5PUsPeGh8V/49Gn9z rkGPNESSw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkke4-0002T2-GT; Thu, 03 Dec 2020 09:11:24 +0000 Received: from frasgout.his.huawei.com ([185.176.79.56]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kkke1-0002Ri-F5 for linux-arm-kernel@lists.infradead.org; Thu, 03 Dec 2020 09:11:22 +0000 Received: from fraeml714-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Cmqlj6mWGz67LZp; Thu, 3 Dec 2020 17:09:21 +0800 (CST) Received: from lhreml716-chm.china.huawei.com (10.201.108.67) by fraeml714-chm.china.huawei.com (10.206.15.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Thu, 3 Dec 2020 10:11:18 +0100 Received: from dggemi761-chm.china.huawei.com (10.1.198.147) by lhreml716-chm.china.huawei.com (10.201.108.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1913.5; Thu, 3 Dec 2020 09:11:17 +0000 Received: from dggemi761-chm.china.huawei.com ([10.9.49.202]) by dggemi761-chm.china.huawei.com ([10.9.49.202]) with mapi id 15.01.1913.007; Thu, 3 Dec 2020 17:11:15 +0800 From: "Song Bao Hua (Barry Song)" To: Vincent Guittot Subject: RE: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters Thread-Topic: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters Thread-Index: AQHWx46tyJL5OgLpakCat9sEfhhBKKni9KEAgACKRLD//5RWAIAAjDiggAAA9pCAAKpoUIAARlmAgACFXCA= Date: Thu, 3 Dec 2020 09:11:15 +0000 Message-ID: References: <20201201025944.18260-1-song.bao.hua@hisilicon.com> <20201201025944.18260-3-song.bao.hua@hisilicon.com> <414fbd167b214452b925ac674575f0d6@hisilicon.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.126.200.109] MIME-Version: 1.0 X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201203_041121_744003_BCD79E98 X-CRM114-Status: GOOD ( 18.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Mark Rutland , "Zengtao \(B\)" , Peter Zijlstra , Catalin Marinas , Jonathan Cameron , "Rafael J. Wysocki" , linux-kernel , Steven Rostedt , Dietmar Eggemann , Ben Segall , ACPI Devel Maling List , Ingo Molnar , Linuxarm , Mel Gorman , "xuwei \(O\)" , "gregkh@linuxfoundation.org" , Will Deacon , Valentin Schneider , LAK , "Cc: Len Brown" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > -----Original Message----- > From: Vincent Guittot [mailto:vincent.guittot@linaro.org] > Sent: Thursday, December 3, 2020 10:04 PM > To: Song Bao Hua (Barry Song) > Cc: Valentin Schneider ; Catalin Marinas > ; Will Deacon ; Rafael J. Wysocki > ; Cc: Len Brown ; > gregkh@linuxfoundation.org; Jonathan Cameron ; > Ingo Molnar ; Peter Zijlstra ; Juri > Lelli ; Dietmar Eggemann ; > Steven Rostedt ; Ben Segall ; Mel > Gorman ; Mark Rutland ; LAK > ; linux-kernel > ; ACPI Devel Maling List > ; Linuxarm ; xuwei (O) > ; Zengtao (B) > Subject: Re: [RFC PATCH v2 2/2] scheduler: add scheduler level for clusters > > On Wed, 2 Dec 2020 at 21:58, Song Bao Hua (Barry Song) > wrote: > > > > > > > > Sorry. Please ignore this. I added some printk here while testing > > > one numa. Will update you the data in another email. > > > > Re-tested in one NUMA node(cpu0-cpu23): > > > > g=1 > > Running in threaded mode with 1 groups using 40 file descriptors > > Each sender will pass 100000 messages of 100 bytes > > w/o: 7.689 7.485 7.485 7.458 7.524 7.539 7.738 7.693 7.568 7.674=7.5853 > > w/ : 7.516 7.941 7.374 7.963 7.881 7.910 7.420 7.556 7.695 7.441=7.6697 > > w/ but dropped select_idle_cluster: > > 7.752 7.739 7.739 7.571 7.545 7.685 7.407 7.580 7.605 7.487=7.611 > > > > g=2 > > Running in threaded mode with 2 groups using 40 file descriptors > > Each sender will pass 100000 messages of 100 bytes > > w/o: 10.127 10.119 10.070 10.196 10.057 10.111 10.045 10.164 10.162 > > 9.955=10.1006 > > w/ : 9.694 9.654 9.612 9.649 9.686 9.734 9.607 9.842 9.690 9.710=9.6878 > > w/ but dropped select_idle_cluster: > > 9.877 10.069 9.951 9.918 9.947 9.790 9.906 9.820 9.863 9.906=9.9047 > > > > g=3 > > Running in threaded mode with 3 groups using 40 file descriptors > > Each sender will pass 100000 messages of 100 bytes > > w/o: 15.885 15.254 15.932 15.647 16.120 15.878 15.857 15.759 15.674 > > 15.721=15.7727 > > w/ : 14.974 14.657 13.969 14.985 14.728 15.665 15.191 14.995 14.946 > > 14.895=14.9005 > > w/ but dropped select_idle_cluster: > > 15.405 15.177 15.373 15.187 15.450 15.540 15.278 15.628 15.228 > 15.325=15.3591 > > > > g=4 > > Running in threaded mode with 4 groups using 40 file descriptors > > Each sender will pass 100000 messages of 100 bytes > > w/o: 20.014 21.025 21.119 21.235 19.767 20.971 20.962 20.914 21.090 > 21.090=20.8187 > > w/ : 20.331 20.608 20.338 20.445 20.456 20.146 20.693 20.797 21.381 > 20.452=20.5647 > > w/ but dropped select_idle_cluster: > > 19.814 20.126 20.229 20.350 20.750 20.404 19.957 19.888 20.226 > 20.562=20.2306 > > > > I assume that you have run this on v5.9 as previous tests. Yep > The results don't show any real benefit of select_idle_cluster() > inside a node whereas this is where we could expect most of the > benefit. We have to understand why we have such an impact on numa > tests only. There is a 4-5.5% increase while g=2 and g=3. Regarding the huge increase in NUMA case, at the first beginning, I suspect we have wrong llc domain. For example, if cpu0's llc domain span cpu0-cpu47, then select_idle_cpu() is running in wrong range while it should run in cpu0-cpu23. But after printing the llc domain's span, I find it is completely right. Cpu0's llc span: cpu0-cpu23 Cpu24's llc span: cpu24-cpu47 Maybe I need more trace data to figure out if select_idle_cpu() is running correctly. For example, maybe I can figure out if it is always returning -1, or it returns -1 very often? Or do you have any idea? > > > Thanks > > Barry Thanks Barry _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel