From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751136AbcGLSiI (ORCPT ); Tue, 12 Jul 2016 14:38:08 -0400 Received: from mail-sn1nam01on0103.outbound.protection.outlook.com ([104.47.32.103]:26544 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750826AbcGLSiG (ORCPT ); Tue, 12 Jul 2016 14:38:06 -0400 X-Greylist: delayed 97980 seconds by postgrey-1.27 at vger.kernel.org; Tue, 12 Jul 2016 14:38:06 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=waiman.long@hpe.com; Message-ID: <578538FF.7040306@hpe.com> Date: Tue, 12 Jul 2016 14:37:51 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Boqun Feng CC: Alexander Viro , Jan Kara , Jeff Layton , "J. Bruce Fields" , Tejun Heo , Christoph Lameter , , , Ingo Molnar , Peter Zijlstra , Andi Kleen , Dave Chinner , Scott J Norton , Douglas Hatch Subject: Re: [RFC PATCH v2 6/7] lib/persubnode: Introducing a simple per-subnode APIs References: <1468258332-61537-1-git-send-email-Waiman.Long@hpe.com> <1468258332-61537-7-git-send-email-Waiman.Long@hpe.com> <20160712031456.GA5885@insomnia> In-Reply-To: <20160712031456.GA5885@insomnia> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [72.71.243.165] X-ClientProxiedBy: SN2PR17CA0012.namprd17.prod.outlook.com (10.169.188.150) To CS1PR84MB0311.NAMPRD84.PROD.OUTLOOK.COM (10.162.190.29) X-MS-Office365-Filtering-Correlation-Id: 96c6aca4-a9ff-4f9a-1761-08d3aa83a8da X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0311;2:xdNMzaCo/WUKFujP1SFsuyjgKgPgWcyLMKrbLMx5grXM0tInk4IdEG3yhLJ09oj7Omr20cyA0X8KpH9VKxFVxEKKFDqPjS7xoc1Xr3NXVVLJnTw7PgWWntl38939fGj0ycJeAccwEIbJURR8xg5COXZ340SKdCrErs393KqQYhzH88UJvrFlCcU0brv40P+8;3:KohOhgHO686RYb4yD322yQxZzvqJNg2XH00IXimA5DBjAxuxzr36nB7IKRHgirbUJVllKKNq0XN6tZ3ceSym0Sl2ioNeoITks2G9GlGtkcV7Oa6SKXsTVGa/2oH1+6YR X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0311; X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0311;25:M931ICzqB9O0IUSZwXemhJCjEeM03xVZQnjsRZidHM64+k4rycqoH6TOsY4Hx53Lniiw0VJTm2sbHibMTEFyQtziISxy+SQQ33PrsvP54c5s14GZDd5XFGDniVfRx24w6NxqnfZixFlSTVaLrO1ZIPbL7pUjtlkJyBqna/euNwLgPYI8kmHQIgcoaljiybC8NXVUjDjQ1g0z51RX9HliY7gfGkQUe8AcAV/TFuGsqgoary5XRFJTrxIvmmstt4OMnwMVPy1696o2e/IYnaz4pHyi7u0u/qG8gEcygYCHyhWWoUFz5d9S8VqO+Lb+Q4PITCHsJZZ11gg7oRxx+TKqFfWQUl0z7HecenrSsy30r9dI3FoPzllHB1qTaPr8bcsHxmGRmSHLxBUb+U+CpH5xq/GdQnQlB9Rdt1IOyfJ3JMZUDN3NrdMivPOODKVAzKR7uVgbZzyyF91YjmcnDLb9LCeUGph/3lgzR7G+nEP4OsrE1DMM0sDjnKJnb1NSnQbTRVTNnPqzHJ3bH/qbHDCpPK+XkNHFsGehosGzMmtfh9jmyw5ZaXUYQu52iCuH4Dh8oaVbS3fRHAwPSh0KMcmQ889uwW5iMKTHsyb43CZdITKGJS2ck+z6Zqp+N0bc3RFWp1wMlPwUYf0rQJhl428cqFuYuPxXn4m74GXQNv0DikdLM9nz4hBIfFEuZjihnqN0dRb2+/q/3gNHOlG9OSM/bw==;31:D8cEXAGn20rQHGHH+ZpicVm8Au9h3uB9iFWXwKgP7kxYlUMpmBa8EmSHJ+6KFGki16EanuZKkmJOqK1XfuTHrQPtxlxsG2DxalctPqeLJovYtAFse7JEErjrDBVl7Yxkye/frOURhcc/BoE/Kqp7D98MKcYft5A0QXmTKH1UGAKbK3QlLb6RcCHpHGi6PF4fMWBt504XVgI/mcu/3Otm+A== X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0311;20:bSWTMdCK0JzWrXNbAR/AFk9jIRoe9TknQR4LNntxJ0bZMZ5+2rMcgPWY2xcMDGMUuCqJDe80neYGlra3PSVrEr8fkaj+xeTQ13su5TkaFN/NqnA5qc2MccKCx6qlq6oxsX0HsuXarKeaJSByZIJhLNqO6Mr70Nudc1M7C8wAc1EMSf/Gx2elb7mxSWdhrjVZfAHGb2Qb3qq5b1EWln/gtwwv0NwkdHORJCr3B/PtqquzCibYIC8G6VPAmmQTpECrbh6zf0MslBYvhqe58Aft7NTtUYDmU+5ctIWdd0UJm59EGrR313aoochpYTTFFy90HpHXYlI691GyosSZNOghDhofPpGZ58JCIC/sV+373i6bQcpBSSMx/o9C9oVQNsmMEWMZS1NXRMEMJLaYfbnWGhvZFjhFt0zBUSyRZCaIOyZU3pKi4MIFNDt/Z0gUagNywBlcGesiaWS8giqBanfKwZ+8i41Up52b6001azaQnQN+FGLClpO3mbVHbL/10wdp;4:K13ewsoZfHDIFZtYl5Y4xyA46ztVmbFFyouIyWuTPVv7aqFMgm/P1n6g737M+A+G2isuR/CK75O1ayo882zLD60iGJrhdRWSnF9GAJwzCp1O/mXfR6zPvVbbiyN5XLeLL4xQJVGjald0RtuxE+/wvq6u/koUkyiO5wQB24A1+8wcX/tZ7HFM1rKhUnGvmu2XtusOawiS++ajKOYYCs/lsUbDXVFwh7A4nQR9MtJH9SBLoH+XQ1U71qolzkqaJ75E4ON5mARaOKCOSEHIjt44bASTO1KqpXCknwGkf4KMNAOEbKGk/YgPmYjUMF/F/GIqks7ULW9fJrgbScPC7XDmpiLPIPUNeHZBvbyjoTa7Apd5n1r92FzXmpgydiwXIh2s X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:CS1PR84MB0311;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0311; X-Forefront-PRVS: 0001227049 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(7916002)(24454002)(189002)(377454003)(199003)(6116002)(8676002)(4001350100001)(68736007)(230700001)(189998001)(50466002)(99136001)(92566002)(105586002)(3846002)(83506001)(33656002)(80316001)(42186005)(586003)(65956001)(65806001)(66066001)(117156001)(2906002)(59896002)(76176999)(50986999)(65816999)(87266999)(54356999)(97736004)(36756003)(86362001)(101416001)(47776003)(305945005)(77096005)(64126003)(110136002)(23756003)(2950100001)(4326007)(7736002)(7846002)(106356001)(81156014)(81166006)(217873001);DIR:OUT;SFP:1102;SCL:1;SRVR:CS1PR84MB0311;H:[192.168.142.166];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;CS1PR84MB0311;23:UBWH1StZdaW1icUiEPfnol5pRUQVoi6ZBn+5xrw?= =?iso-8859-1?Q?pPk4Xbcuw5XRfmXMUQmOour1zuYwYHc8XXkwlxuykHkRH96fk64vYjk86L?= =?iso-8859-1?Q?6xabpjAim6gwBJ4zB+p3UCRBJwQKfzaRpCbWgpsWTLXsnftNtGx60sgvPV?= =?iso-8859-1?Q?UfasaBQ3wdG+apQUcm+py5B2gpiHTQX5SG4g1g25xAQqIvE4OIb2zKCVQh?= =?iso-8859-1?Q?ZFTmb7C20/LvRYiV66+dY0YvNm9VlnQVksrK23PnE+1aq8v48T2ZNBAedP?= =?iso-8859-1?Q?iitApt3TEw1Epqhjnx3TZxBdjSPEJzE2D5pkgcMhuwF0xYu0ZXpqhozXm9?= =?iso-8859-1?Q?Df4HXRrPqL3oa3pWgf0l7hPnlHUhXFkPxVhlXjfqhsHmrG4MndtUP72SHw?= =?iso-8859-1?Q?Y+6/GliIRudJsBVpwxxjj2g8aNqna7CJlkBtCwTwqJEPyfJJ2VKT3ohxGL?= =?iso-8859-1?Q?mccovR9IbC2SOdVUyX2xJTiwx8jUU2OsHY1IHftKl1ILLp3xLT/cjkNThk?= =?iso-8859-1?Q?GZNqHiR4x7/ANCPDVYqFUvHaNhy80LbETvGQ+JgV/28uLq3OR7dKFpqy/e?= =?iso-8859-1?Q?tzAM2JC9f5OtzCQSoif7uyAsGRLWbuhrfkqjV2tcFMu2Gett3oXyRu6Plo?= =?iso-8859-1?Q?/22eyFmj1vOF4YWYkCfJflVY7ESe+BhWIO0XxjL/yRvOjsNYwgtSemZEub?= =?iso-8859-1?Q?JnHp8WLtQJ/XK2tToNyCoroQYehSb0vGzf9m3OGVMXnWnnZ9cEj9I/DoSx?= =?iso-8859-1?Q?8Q3Qjq4cycBLaqFOl+rbjur5UbdPIhT5ssEQLum9zXE8ZMuDqnr37k3M4m?= =?iso-8859-1?Q?5LuwXc3bMviZNeyOa5GKtE3EyKUo8VluWyUQNMDzoe4v9Hj4EpQiwVCJ1u?= =?iso-8859-1?Q?abFcqEG2WAr1Mv9oetP35HPEmVT0sfFMrez9fyyWEZ1QYWXjAStw4NHJm/?= =?iso-8859-1?Q?QaDFXjcJcM/fxQm1RkRI3MBIFxxgoDUGF4zu5b87q5CyztaH3lMFFO6qCD?= =?iso-8859-1?Q?ebPy7h2BPFArgbvdVQtBGa9OWdxbSJeStM4RQtI8GiZY/onrwRr3iC5Jkg?= =?iso-8859-1?Q?SMiVV34aTYpyizksen4kuoSq5kXGrEV38pICyZKlQtWmljNGurdy7Klw2B?= =?iso-8859-1?Q?ify5/IOwDSU01XyOItrE37yvOYg/fdsMoz4JIZlhfmxh5+MQpYUffrg7zS?= =?iso-8859-1?Q?ep9FfjwDXtNUIwrq40sNs1vkBosYubf/4VnRAOv99VmnrVgo6ZQ90VLh/B?= =?iso-8859-1?Q?EatmfsFDTehQJMmPVdLyMYZJ+XfKU6OjtJLX7RW1SpGA3ymMlK69Ka7XO4?= =?iso-8859-1?Q?r/zWzIJ0ty/F+Q+U+vO90lsLDq5gDGJrpj2Up3p8KlPRQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0311;6:XYinbqCDfI//k7CSt5t3aUkhNYsFXWQxwaZafDp2mwa+D7xPTolcqw525Zq1+XYQs0sxhoCYsctrwp10nO5Pna8KPscWhZPHwCuAtDPLOvg9uD3GPoCwMcTJPd03qMEFeh1OhyOAOWsrW6vWSxLp2HE/ydRpSC+kgvhJz2VVp/v8P4z6rQ6seYEK4Vn3jiSztFY5+zlx8rRQwgipFlZGlrT+LZS7i9NSXu2UVyaxLJ3h/EhydM+767aEIb5k1i8pQh0sJyaSiVXza6Nc9e9+X5ulzNfgk2Y+KFHmrDypCeI=;5:9pHGj0vmbP2JOQh1uYV/NVhbP8aF60Puu4KvUk5NzVSlDXg14LHUXEqJiGtZLjbCIk4qGdbi05Q4+kQhbJQvhFYknmOXhLR3fMIWc287CryKh7gc5iLDdvBDJxF3a2KPTBI9K8hEaeZA593R/6KroQ==;24:wo53StiI4xfVPLLKialccr6qrzm3uS86i3VHeTIVjDyUK8XG2MX5iaRcCUceU3f7pORnt/kMIe/wm9KyYd9jdQyaLmOgV+l07bP0bdbK2d0=;7:QW6pD6BDKh3BjvwpC/8d8y3P+C63aEsautTVq9Dra2FoVM8LgwnY1cZkmeqUyOUV/Wa0qboWEA7DHfBQqxCuNrn4pgXvFIxhM9Aao6XSQU0EfawPdZcm+GtTw+LgBqZFpfJ4GNuy5oNAV+yRdGsHxOp9RXslxnKeVLvzHpUAHMCYfmN75p4CKM1KehaXXOrpq27pAX3CzuMHYO2h4vgEP7C6CX9U4LNOyUFd8sLlwcxai2nyqukEE5w45jTHRiNh SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2016 18:38:00.1933 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CS1PR84MB0311 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/11/2016 11:14 PM, Boqun Feng wrote: > On Mon, Jul 11, 2016 at 01:32:11PM -0400, Waiman Long wrote: >> +/* >> + * Initialize the subnodes >> + * >> + * All the sibling CPUs will be in the same subnode. On top of that, we will >> + * put at most 2 sibling groups into the same subnode. The percpu >> + * topology_sibling_cpumask() and topology_core_cpumask() are used for >> + * grouping CPUs into subnodes. The subnode ID is the CPU number of the >> + * first CPU in the subnode. >> + */ >> +static int __init subnode_init(void) >> +{ >> + int cpu; >> + int nr_subnodes = 0; >> + const int subnode_nr_cpus = 2; >> + >> + /* >> + * Some of the bits in the subnode_mask will be cleared as we proceed. >> + */ >> + for_each_cpu(cpu, subnode_mask) { >> + int ccpu, scpu; >> + int cpucnt = 0; >> + >> + cpumask_var_t core_mask = topology_core_cpumask(cpu); >> + cpumask_var_t sibling_mask; >> + >> + /* >> + * Put subnode_nr_cpus of CPUs and their siblings into each >> + * subnode. >> + */ >> + for_each_cpu_from(cpu, ccpu, core_mask) { >> + sibling_mask = topology_sibling_cpumask(ccpu); >> + for_each_cpu_from(ccpu, scpu, sibling_mask) { >> + /* >> + * Clear the bits of the higher CPUs. >> + */ >> + if (scpu> cpu) >> + cpumask_clear_cpu(scpu, subnode_mask); > Do we also need to clear the 'core_mask' here? Consider a core consist > of two sibling groups and each sibling group consist of two cpus. At the > beginning of the outer loop(for_each_cpu_from(cpu, ccpu, core_mask)): > > 'core_mask' is 0b1111 > > so at the beginning of the inner loop first time: > > 'ccpu' is 0, therefore 'sibling_mask' is 0b1100, in this loop we set the > 'cpu_subnode_id' of cpu 0 and 1 to 0. > > at the beginning of the inner loop second time: > > 'ccpu' is 1 because we don't clear cpu 1 from 'core_mask'. Therefore > sibling_mask is still 0b1100, so in this loop we still do the setting on > 'cpu_subnode_id' of cpu 0 and 1. > > Am I missing something here? > You are right. The current code work in my test as the 2 sibling CPUs occupy the a lower and higher numbers like (0, 72) for a 72-core system. It may not work for other sibling CPU assignment. The core_mask, however, is a global data variable and we cannot modify it. I will make the following change instead: diff --git a/lib/persubnode.c b/lib/persubnode.c index 9febe7c..d1c8c29 100644 --- a/lib/persubnode.c +++ b/lib/persubnode.c @@ -94,6 +94,8 @@ static int __init subnode_init(void) * subnode. */ for_each_cpu_from(cpu, ccpu, core_mask) { + if (!cpumask_test_cpu(ccpu, subnode_mask)) + continue; /* Skip allocated CPU */ sibling_mask = topology_sibling_cpumask(ccpu); for_each_cpu_from(ccpu, scpu, sibling_mask) { /* Thanks for catching this bug. Cheers, Longman