From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755164AbcBZTvm (ORCPT ); Fri, 26 Feb 2016 14:51:42 -0500 Received: from mail-by2on0098.outbound.protection.outlook.com ([207.46.100.98]:26955 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754856AbcBZTvj (ORCPT ); Fri, 26 Feb 2016 14:51:39 -0500 X-Greylist: delayed 66295 seconds by postgrey-1.27 at vger.kernel.org; Fri, 26 Feb 2016 14:51:39 EST Authentication-Results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=caviumnetworks.com; Message-ID: <56D0ACC4.1060605@caviumnetworks.com> Date: Fri, 26 Feb 2016 11:51:32 -0800 From: David Daney User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: Will Deacon CC: David Daney , , Rob Herring , Pawel Moll , Mark Rutland , Ian Campbell , Kumar Gala , , Ard Biesheuvel , Frank Rowand , Grant Likely , Catalin Marinas , Matt Fleming , , Ganapatrao Kulkarni , Robert Richter , , David Daney Subject: Re: [PATCH v12 4/5] arm64, numa: Add NUMA support for arm64 platforms. References: <1456192703-2274-1-git-send-email-ddaney.cavm@gmail.com> <1456192703-2274-5-git-send-email-ddaney.cavm@gmail.com> <20160226185341.GN29125@arm.com> In-Reply-To: <20160226185341.GN29125@arm.com> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [64.2.3.194] X-ClientProxiedBy: SN2PR07CA015.namprd07.prod.outlook.com (10.255.174.32) To CY1PR07MB2133.namprd07.prod.outlook.com (25.164.112.11) X-Microsoft-Exchange-Diagnostics: 1;CY1PR07MB2133;2:0rgXt20zRXvnxcXF9uo9uR6jC7WJqTHDLfD/UanR/l8hu7NHyHqbOBTFjCyb69auHbTZqAXXTJedbBJRC6JUKL+/6OFZ3Pis42Bhgk1JM1MSefxlhuEM8DjAajGzwUJs+wNkWNyBq0ZSERREYLRezw==;3:KL8Xv7PpImZR3salWxhx+wV64ZD3DPwvaQPi7Ae+sxmQ9FWFtJ1ka5aCsMunoBB300HYpROhOo4rV9i/Arw52kKTFaKIWg5+uTZ8SPgkhzSGIkAxfxua11fR+gpw2eLE;25:1PsLs44XYtHF/gSyU1Y5Q/7ckFNUI/m3fQjtnrHLTKDOWL4gRfae5a34APfTp07wwJhKHCbsnRVO9cc8ShnIOM/ei44Q2sVvlMAU/4xk59cFTRgRS/f5dptQa6F5WnBMBx9VTccjgIu/Fph98gVcinGhPrjKtRXvLVSGP15cJdJZg5ugQBFd03ambPqTOYZsSDhunvxD1hHTrLJIw2EQwAifvAwfOlDiaz56YejRyf4APH58u9xgmIUf1WNGrDbS3e1TFje+Z3C8vvz/AqhbUknW9c+K7MScPGJ/O0mE627s2SGJ03JsaMuu3A+usj0EMHJAzBuuvPWnA3XNJ9bohA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR07MB2133; X-MS-Office365-Filtering-Correlation-Id: eaa96d5d-fda6-4416-1888-08d33ee63c9f X-Microsoft-Exchange-Diagnostics: 1;CY1PR07MB2133;20:x9YNLBfx9XaKBO94FPPUUc0zKTOHXkbhiRjA/e9LyzUfssNwS6c+fxBA6vTBQeWwmJSmQIsoYzj+OgXApa6PXM1rkhknotSxzK//0S3KrAJRkkSLA1S6kamsGCTn66aynxzkikA/QX0OAD6axlLmkCpuKndJPPq1QvHsD7/kOAe38GoWv2cDZfnpWHAiECiSeX5CRaZ1xFQRWTCxnfdbJ7pEOQQR8DaYaiGkZ9TRNcOfsbo9SsFy/fZS8EbGG3Nv3espLQXUzDZSyFPL2sKN2/anBXWiMWWrcL3oRXinKGjDsEcpkubh2DzPgtgP8kTlk1FLccpSVy6AKCwNq2x1GZ0SfqRAM6ebcZgWjq78MQL/wRq6fmYuxHL3k04fxK1GZweVTpTNLWACjE7cInh7bO3gRywEMattulMq/tgTuOZabBduhoY2C+LgGL0bVG4ju0Sj1H+c9105okaAvbeMc0zai4zSs0fSSUlrvJ9U5Pjozt7kgNg3hlSctflhnPwgCWLpRZnNyF6FsuRuW8Ip4bASg16LaACWkez8rAW88+UzX1AhsTnZcSsD0QII3vRc+pNGlPZ8kcCXVfJUtvDGj4f5NtVDTycaSFpHyupH7y8= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:CY1PR07MB2133;BCL:0;PCL:0;RULEID:;SRVR:CY1PR07MB2133; X-Microsoft-Exchange-Diagnostics: 1;CY1PR07MB2133;4:PSzFtKGXDPO754lpQn+VPrNh2Ube4Bp4YSJa8y2Xk4FsZwij5mOcTHz4MEqOgrVoFqrRyE1kXEKqrj+EmJjn4+YUgjqW/WSszg0xKy1/BB+74MfT694SuMRn1CsAjEkVgo0nxPuzMl+doSXmbO1kgj6kzdukvSbIaCa8Xh1rjS7XlCKT8JX1X0tWN866A6/RmJgpc0IS44Nus55O5Izax0E7erNpd/u6EMHM52e9eUR3YE+K5TzaFrhLo+BsMHo5taQThkhCC3CCX7mx8qCN5GzQGYHwrEDS3C1v9TV+s7yrIWCZeAdfWz0JUW8KSwTHr9HOaiVc3ZA+J6U2mgsEI8kuizhKyUCa8WmQ22OmFUme1knq7GAkcqpz7DzvoOPt X-Forefront-PRVS: 0864A36BBF X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(479174004)(24454002)(377454003)(42186005)(53416004)(50986999)(3846002)(54356999)(87266999)(189998001)(76176999)(65816999)(110136002)(5001960100002)(65806001)(87976001)(1096002)(36756003)(66066001)(65956001)(47776003)(64126003)(5004730100002)(59896002)(586003)(50466002)(6116002)(83506001)(92566002)(40100003)(23756003)(33656002)(5008740100001)(230700001)(4326007)(2906002)(4001350100001)(77096005)(122386002)(2950100001);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR07MB2133;H:dl.caveonetworks.com;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;CY1PR07MB2133;23:8k8B3iFU3/jIDFumVXeFmc8tqBwX+gsQDm7rPzf?= =?iso-8859-1?Q?MLqrMZZfztAXiCP9xLhLHfwJuC//5xxfxV5TPHGS2pdKE74FyuOjU5FqLy?= =?iso-8859-1?Q?NW2C0gtv73M6/4y/ivTkglRpijyjfI7/hyH3NDzn4hVQktR7dRYv2dZdj9?= =?iso-8859-1?Q?lRLzPorvHFrUrBeXd2Txkn9uQq0PAyWr2zqr+0Z44zau9xlMHL71deR+sG?= =?iso-8859-1?Q?Pgj3u4oMYMPVnu5inHgTVbbipnOHnzIvydRMfxPV4ErQAUIAIv7vnoRQ+3?= =?iso-8859-1?Q?5sQIh4ck+uB0PTxX9JCn0O5Y/6ddPXnvhNCliFaFFKMxRfewtdSVpFHsjo?= =?iso-8859-1?Q?iVjM8a2zsWtdLbObqXEHBkQnS7FVS5Fddv9eqno5MEhUPwF0lClw2TnnfZ?= =?iso-8859-1?Q?uC85WJfU3kNPWfjtxXWwOMVjuGqc9NK5PumaTj5EZc94lmodfFYTByBxpw?= =?iso-8859-1?Q?2g6HKooe/D5sapeOxn8EdL81aAkmiawGoVmyc0GkOjT4HKZAjw58ujx/KA?= =?iso-8859-1?Q?syxCAPwZ5mxnaSPL7tKVWF3RyWEllD26kGEnQQPi4AMS89b2VJLupFkwz2?= =?iso-8859-1?Q?KBpwaOXHsI4QPiAmF9S9Cx4KAbHqCaN7F6WRJMoZOGspaihhfV32EN72ua?= =?iso-8859-1?Q?wgyYBoytHBSt3n97LKNC7CardrMbSh9GpAQsqSEPIuTR3/6nIqD5MuXRjx?= =?iso-8859-1?Q?DbEM0a+mdx5iwaX1uVsQoisM4UFMLfKTBBl+xihwkutdG1Z+5vVn72jkyv?= =?iso-8859-1?Q?jcoANDFIE1mg/AMDT6lN7E+A9bsfOMSbakD2QlQlmRcwpumfDWExQDjoqA?= =?iso-8859-1?Q?GW624pRfr9v2N/DQD2pZGJVGLXOiLfzQoS5tkTyZud2flB5FBQXYbwYUoo?= =?iso-8859-1?Q?jCQY9uiL4AoLWdWzvMvVNKGRU8HSBb5sFKg9P+qlfhgm+aqRtQCqJ2cxun?= =?iso-8859-1?Q?ELY0A02LTu/B3otrAnRoHavGTrzttjkOk8BYEtfRawfRbCL6i6nohAbYNG?= =?iso-8859-1?Q?kYCqKHSTeXaOBvNntJAkDj/hhxxvelFmWVaUIb07/vNC/JqwXJBRUdJtQP?= =?iso-8859-1?Q?PqQR+QebhQCM95gTAC31Y5as6FRcOIVwYGRMiJc6H1XEp0QuimF9CG/FPH?= =?iso-8859-1?Q?0lxBv+DSb4TZZksykJIgIAcwKbRLr0onjv4Hj5y+S9g9JVxc=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR07MB2133;5:VaOYTs5Xli1g+VesFKCzTsW1fhrbt5j42HKYS9EcPIdNaXXDxq/T68Cbn7ykWhUOe5mMLYMj0cDoj41k4xHJxwoJdvRREfNzkvVUXQPwi60vw6yaM8z5vPPkSDI7Kj5VCs/6NKunvbKjcF3hnQjeTw==;24:6YnCp2sUV9VSLmpkTdJNevWRqEd3nKlwSGr/9gEWPA+TUw+Dy0EJlxHpl86CjICnDraw3ka5KIcGoZ2kkP+wyWDStNIQ59ym1AIIUEf5kFs= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Feb 2016 19:51:35.4206 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR07MB2133 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/26/2016 10:53 AM, Will Deacon wrote: [...] >> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c >> new file mode 100644 >> index 0000000..604e886 >> --- /dev/null >> +++ b/arch/arm64/mm/numa.c >> @@ -0,0 +1,403 @@ [...] >> + >> +static int numa_off; >> +static int numa_distance_cnt; >> +static u8 *numa_distance; >> + >> +static __init int numa_parse_early_param(char *opt) >> +{ >> + if (!opt) >> + return -EINVAL; >> + if (!strncmp(opt, "off", 3)) { >> + pr_info("%s\n", "NUMA turned off"); >> + numa_off = 1; >> + } >> + return 0; >> +} >> +early_param("numa", numa_parse_early_param); > > Curious, but when is this option actually useful? > Good point. I will remove that bit, it was used as an aid in debugging while bringing up the patch set. >> + >> +cpumask_var_t node_to_cpumask_map[MAX_NUMNODES]; >> +EXPORT_SYMBOL(node_to_cpumask_map); >> + >> +#ifdef CONFIG_DEBUG_PER_CPU_MAPS >> + >> +/* >> + * Returns a pointer to the bitmask of CPUs on Node 'node'. >> + */ >> +const struct cpumask *cpumask_of_node(int node) >> +{ >> + if (WARN_ON(node >= nr_node_ids)) >> + return cpu_none_mask; >> + >> + if (WARN_ON(node_to_cpumask_map[node] == NULL)) >> + return cpu_online_mask; >> + >> + return node_to_cpumask_map[node]; >> +} >> +EXPORT_SYMBOL(cpumask_of_node); >> + >> +#endif >> + >> +static void map_cpu_to_node(unsigned int cpu, int nid) >> +{ >> + set_cpu_numa_node(cpu, nid); >> + if (nid >= 0) >> + cpumask_set_cpu(cpu, node_to_cpumask_map[nid]); >> +} >> + >> +static void unmap_cpu_to_node(unsigned int cpu) >> +{ >> + int nid = cpu_to_node(cpu); >> + >> + if (nid >= 0) >> + cpumask_clear_cpu(cpu, node_to_cpumask_map[nid]); >> + set_cpu_numa_node(cpu, NUMA_NO_NODE); >> +} > > How do you end up with negative nids this late in the game? > It might be possible with some of the hot plugging code. It is a little paranoia programming. If you really don't like it, we can remove it. >> + >> +void numa_clear_node(unsigned int cpu) >> +{ >> + unmap_cpu_to_node(cpu); > > Why don't you just inline this function? Good point, I will do that. [...] >> +int __init numa_add_memblk(int nid, u64 start, u64 size) >> +{ >> + int ret; >> + >> + ret = memblock_set_node(start, size, &memblock.memory, nid); >> + if (ret < 0) { >> + pr_err("NUMA: memblock [0x%llx - 0x%llx] failed to add on node %d\n", >> + start, (start + size - 1), nid); >> + return ret; >> + } >> + >> + node_set(nid, numa_nodes_parsed); >> + pr_info("NUMA: Adding memblock [0x%llx - 0x%llx] on node %d\n", >> + start, (start + size - 1), nid); >> + return ret; >> +} >> +EXPORT_SYMBOL(numa_add_memblk); > > But this is marked __init... (and you've done this elsewhere in the patch > too). I will fix these. > > Will >