From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932977AbbIYSED (ORCPT ); Fri, 25 Sep 2015 14:04:03 -0400 Received: from mail-by2on0073.outbound.protection.outlook.com ([207.46.100.73]:58816 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932812AbbIYSD6 (ORCPT ); Fri, 25 Sep 2015 14:03:58 -0400 X-Greylist: delayed 3588 seconds by postgrey-1.27 at vger.kernel.org; Fri, 25 Sep 2015 14:03:57 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Tirumalesh.Chalamarla@caviumnetworks.com; Subject: Re: [PATCH] arm64: Increase the max granular size To: Robert Richter , Will Deacon References: <1442944788-17254-1-git-send-email-rric@kernel.org> <20150922182902.GO7356@arm.com> <20150925144516.GM1820@rric.localdomain> CC: Robert Richter , Catalin Marinas , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Tirumalesh Chalamarla From: Tirumalesh Chalamarla Message-ID: <560576EB.2010208@caviumnetworks.com> Date: Fri, 25 Sep 2015 09:31:39 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150925144516.GM1820@rric.localdomain> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [64.2.3.194] X-ClientProxiedBy: BN3PR0601CA0004.namprd06.prod.outlook.com (25.162.30.14) To BY1PR0701MB1781.namprd07.prod.outlook.com (25.162.141.19) X-Microsoft-Exchange-Diagnostics: 1;BY1PR0701MB1781;2:/x49vxyn/sQ9QtBNJIWJb0orWX/+qR9rM6vLSmxUX+fCBZ6nIJolfU2pZwULI7jwi9XeKGGJPdYBDAFtN8qk51bHdYK0hRpUgydL5Ek9fc65qG9UkXYCLGZsbuSUb/B/XFcUfZ2uyI/R11rTbndpdSg7fTOpN/OlTLhmw4pQtKA=;3:Kw4Yj19ahUx+b8iDMymRZpU31ndu/KjftyybKG1O2TBu9yHX6xcSKeV7swXUbTcoFYxbxJuF7ILxJxH2m5kx75jgj2+1x0gWkxIq08Qm/1oe632DNU/Zdt7HJLLGjDv5ez0flxxXnTWG/AICWsG9Dg==;25:H6UplJ+zL28HGNr2YF64Xp5MYi+58TyRuQOiYO6Xj4/FO8DHCoDjs0mFnV74Vti/updTZdRnP7qek/BzT5Zn7orX0/Ct4A4002JfETe6WheR5beQZGIbTweDTEjTOXdraz5zTSbTIg4cObwV+/EYk8QwdVkYy+Eocq8kKplcY+ThN8rhrHf/+MQTgktXAPtRtPjSan09Xd4+eoXEMd3+wgzEVYqJu8kCZvsUam9hpXLzqU98Mw3RUT90C1xo3JTT X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY1PR0701MB1781; X-Microsoft-Exchange-Diagnostics: 1;BY1PR0701MB1781;20:ptDrRMXaQnyDuzROJQD1r36QTR4bbYB0ITYYwHr80FMbgzbLcEHEwykdjI/JnzUtSoVbRUF4hjnJi/zPsUmv03MKZfgHoHuUjM6F3X+pmxJ0CVJ07r16KxKjYwXdr1neuliKNgRd7PaAnMZkB0qPsvBPPygMZn9WM2OFRliS/IIl/YW4OzJwToFs0s5XJ4MMtEF+S3yCqZIqkxMboUobsRvQnDPlRCw+lUT6B36fwhWyRtHMuT1gGYh7hi61QLByNbU/sSua4dHWd3/jvsPOx2fMpWTRZCHNTqQNGFidutNIX3GlfnwIJN3DGAWL1qOD2Y1ngkFcynui2JV6kShncdB55w2b56Q1pW+8xh4MDZxCDBZnyueSpB+YHUFv9N8JLzj3vEwlkNEis708IGDisRUO+zHic/NLLWjd7iuHGUToC4wWu/6fwM7tEgQg4hlogTMKdkWPY+R2mR5T8OIf4B55WQGm8x2Zp8C8OZIVkll1L/x6wy7TEIncvilaJKrBu2jk5CIDUqUl1CemNdXIoLR+ZNaUA2Tr6stEIXNC8FR1hBAniJsNsh/Lgi6/mcY2vKm7CMVbkJiHU+Tl1uk2/wstWE3Y5OM5VkJk8U6+KwM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001);SRVR:BY1PR0701MB1781;BCL:0;PCL:0;RULEID:;SRVR:BY1PR0701MB1781; X-Microsoft-Exchange-Diagnostics: 1;BY1PR0701MB1781;4:D3UT01SFUrsfmiMqRrqVMMf+giPJnFuyZq1uLGJCsD25rOxyHBtWgysTb5OwR2kF3MRdpb9KYPKOPUrrOoqJv309n3G9xXQHlQOO5PhPksJZGMHiPxnPY55tCZEliGgYLP9rYJk9NVNgrmT3L1uMgulxRRar33bHCoGFQbG7NuzaxIRY7mueA/yajRd+BBHhF5BAQrHtWqTdX2olbWz1/eVIX/Nuvs+vuXCevkyXZJcjytyv0zNRI0qn/Kn6QtEFUZRNp5GW4gfe3Suw332chTUiHpcK+Pu40vhJddbCcK2YuG/icNn54/WxlQlZMWBQEmF8oLZyyOKXAw9kiBxUHcdYLnqHU7nj+t1boTAdjTs= X-Forefront-PRVS: 07106EF9B9 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6049001)(6009001)(164054003)(377454003)(199003)(189002)(479174004)(24454002)(50466002)(5001960100002)(122386002)(189998001)(77096005)(40100003)(64126003)(68736005)(2950100001)(47776003)(81156007)(97736004)(4001540100001)(65806001)(5001830100001)(36756003)(5001770100001)(64706001)(4001350100001)(5001860100001)(5007970100001)(65956001)(66066001)(101416001)(99136001)(59896002)(23746002)(5004730100002)(87976001)(62966003)(106356001)(87266999)(42186005)(19580395003)(105586002)(54356999)(83506001)(76176999)(46102003)(92566002)(65816999)(77156002)(19580405001)(80316001)(33656002)(50986999);DIR:OUT;SFP:1101;SCL:1;SRVR:BY1PR0701MB1781;H:[10.18.104.236];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BY1PR0701MB1781;23:3FUk/ujX8F7WQSXI9iSIego4Y0wHh6qaacv?= =?Windows-1252?Q?vT2yYdy2MF3i8ccNX1AS+hLxqt8Ws/TZFUxOEuHg52KBWNBKTNHcNt4N?= =?Windows-1252?Q?2cYh74OzAgp31qP7+5ANKV0CZX6/Mr4ZWO4Fwm3j6rZb+Hldw6SWKWgh?= =?Windows-1252?Q?TMwWSWkRtcjxbQshqScUpOONz97mjD6cZcfEhyq7WFyYyyQzlCQbFN7X?= =?Windows-1252?Q?SnYTCPdFZQJGvIPP4hN0euApXOqJPZ9oJM1xxgrI7nIM+rQuKvM5ierx?= =?Windows-1252?Q?MHGTkii+phRYDhIVsnalnsgVceZxCKZr80R59mK5REsB6LQBz7rgIyaG?= =?Windows-1252?Q?PLcdSLtGHFRcp9+bdbSYgnqhW/CGzZiqIuogSvLN6fmhTbUcbPuJQC+Z?= =?Windows-1252?Q?3V0Uc0Ssmml2qU7gTsyGmdkoX347IgElQU7CPgVMuDvRsnvx/0jdS8iE?= =?Windows-1252?Q?mCE38XR4J12SoQkAFEx0fWBT3qY/uV+/U2walXhXh9YrHR1FHybQiwyX?= =?Windows-1252?Q?vhvLOhP1IDVHy632xSKy3CuaJpc/yd3i/UEyZAFJYcJVIBxpkpTv7Rvm?= =?Windows-1252?Q?utTDt+F53Cv60f2D3pEGdoQr+NCqZjQrCNNCQgkg3mU/SS4jGWqeoh7J?= =?Windows-1252?Q?sfuf73NeZ82p1JO/ZYvNLe6ENoCwKvhpEb+lik+NJMGlzu2FeRyZR4sX?= =?Windows-1252?Q?7O6dIWS34AE1QOyFqsXmZfgXVGfma7XL2moOutXC+hK85I113uGBII5I?= =?Windows-1252?Q?iQBMPxMjxh2PW41SixLHJF18XcLyC5bhj1Bf/HJb83AsJXhuMUvrW/o3?= =?Windows-1252?Q?XR5rM5XE7FthX1VuLfOExe0N/ywLIvpIpBaoccQj761UBrW/797OOnmE?= =?Windows-1252?Q?Fe+qBBbrsu8TFdTMdx4/oLF5BNbbfCNfYxvjXXoil7FY4l2IG8aPwtP0?= =?Windows-1252?Q?+3Fg65C6uUDffaiaCtGj/K192D/1zWHP0SN6TXCoifzC8KXV5UyNdS+D?= =?Windows-1252?Q?alPdkBHkh/MvBS3cYWMuB/Zo8Pc6tC0bsEaeapRxNIwhqH16Plgg1UDJ?= =?Windows-1252?Q?lYz/pYCDBbO2MdW0bEc+g2zJtc5SrhdMet0dyCnD1cLINJVf3c6qaesd?= =?Windows-1252?Q?Wmva1ncNpmX0GPgXSu/qKqfXEpUhQQnkT9y5PRwU8PhVbjFO2B4HFCMp?= =?Windows-1252?Q?1p7l8hcZ+OpxAHwbi7cfHcI6mmz69ZFKdq0cUZHHEON9Bs3FNR39bnYe?= =?Windows-1252?Q?cnW37l+Z5Y7DBAjAY19IlnqQlMQiK+LANdSzOXfPdmduJUxTUJxMKta+?= =?Windows-1252?Q?/gw12avBQu+xnBzXv2qG0eBMyLQ+xLtOBA/uN+FNqjlmE8qAZ9w5ndVt?= =?Windows-1252?Q?sB+djyQSka5Y8CndcaIFNYCvb+F6qH3vu1JnVuhoNRbij9KYSc8igtt1?= =?Windows-1252?Q?VVGIkekU5wsMPkLWvQJoOBnEnoHfISthRI8nZkWsrnO92bCps2C8i3IS?= =?Windows-1252?Q?Y6t7f4k0aLT32iX0kH2n90Zg2B0QmEyg3seFkG7y9WKbkyie38Q=3D?= =?Windows-1252?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;BY1PR0701MB1781;5:Qo0DXew1QxR63qH1VAymL5Lu5tB2yce6NtFcyog4sn3J0U2W1ubpKQIcW7zutC5iZ0g8nZiUEhi6APlvyaCvfAP5HDQ3vd1ODJx1NBwRkD8vvgwqG0HAPCcBp0IAGMbzhagkRVbJj4JAwSwfFMFn4A==;24:et8E8BU9sEGjhxububD/emCSMkD0nlLC7I5jMY1nWMzlm1Rfye8QM8VvKuNXCq5oh9Ko1mzJQSSwfRm9pzv0AOffoE/7hPMaYiwglGQ4Hqs=;20:sst+agtBXAy0xDAYQzBtrd9bdznTiAgu6z1hQ+aHqCE+9ob7X1+Gkw4qeGnr5klzSOXJYU17NrHWoc9QlG7RpA== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2015 16:31:45.6411 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1781 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/25/2015 07:45 AM, Robert Richter wrote: > Will, > > On 22.09.15 19:29:02, Will Deacon wrote: >> On Tue, Sep 22, 2015 at 06:59:48PM +0100, Robert Richter wrote: >>> From: Tirumalesh Chalamarla >>> >>> Increase the standard cacheline size to avoid having locks in the same >>> cacheline. >>> >>> Cavium's ThunderX core implements cache lines of 128 byte size. With >>> current granulare size of 64 bytes (L1_CACHE_SHIFT=6) two locks could >>> share the same cache line leading a performance degradation. >>> Increasing the size fixes that. >> Do you have an example of that happening? > I did some 'poor man's kernel build all modules benchmarking' and > could not find significant performance improvements so far (second > part with the patch reverted): > > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m10.490s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.747s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.264s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.435s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.569s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.274s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m0.507s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.551s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 6m59.073s > build-allmodules-4.2.0-01404-g5818d6e89783.log:real 7m1.738s > > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m10.644s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.814s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.315s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.610s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.885s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 6m59.281s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.869s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.953s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.787s > build-allmodules-4.2.0-01406-g638c69fddc40.log:real 7m0.656s > > I will check what kind of workloads this patch was written for. > Tirumalesh, any idea? mainly for workloads where compiler optimizes based on cache line size, let me write a small bench mark > Thanks, > > -Robert > >>> Increasing the size has no negative impact to cache invalidation on >>> systems with a smaller cache line. There is an impact on memory usage, >>> but that's not too important for arm64 use cases. >> Do you have any before/after numbers to show the impact of this change >> on other supported SoCs?