From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752551AbcCWAJA (ORCPT ); Tue, 22 Mar 2016 20:09:00 -0400 Received: from mail-bl2on0076.outbound.protection.outlook.com ([65.55.169.76]:42528 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750988AbcCWAI4 (ORCPT ); Tue, 22 Mar 2016 20:08:56 -0400 Authentication-Results: spf=fail (sender IP is 63.163.107.225) smtp.mailfrom=sandisk.com; linux.vnet.ibm.com; dkim=none (message not signed) header.d=none;linux.vnet.ibm.com; dmarc=none action=none header.from=sandisk.com; X-AuditID: ac160c71-ef3ff70000003396-21-56f1daf4f39d Subject: Re: RCU stall To: "paulmck@linux.vnet.ibm.com" References: <56F1A8F2.9000905@sandisk.com> <20160322204510.GS4287@linux.vnet.ibm.com> CC: "linux-kernel@vger.kernel.org" From: Bart Van Assche Message-ID: <56F1DAF6.3030804@sandisk.com> Date: Tue, 22 Mar 2016 16:53:26 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160322204510.GS4287@linux.vnet.ibm.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrJLMWRmVeSWpSXmKPExsWyRoxnke6XWx/DDBYek7O4vGsOm8Xbzd9Z HZg8HhzazOLxeZNcAFMUl01Kak5mWWqRvl0CV8a2D59ZC/5zVJzd8JqlgXEyexcjJ4eEgInE kVO/wWwhgQOMEu198V2MXED2DkaJH+/2M8MUfWy8ywSRWMoosXLaREaQhLCAmMTOV41sILaI gK3E54PzmSEmhUtM39IGVsMs4Chxe+9bJhCbTcBI4tv7mSwgNq+AlsTrqweANnNwsAioSmx/ lwQSFhWIkGh98JQdokRQ4uTMJ2DlnALmEp/X3oEaaSExc/55KFteYvvbOcwgt0kItLFK3P2z mQniBnWJk0vmM01gFJ6FZNYsJP2zkPQvYGRexSiWm5lTnJueWmBoolecmJeSWZytl5yfu4kR EuCFOxhf3/Y+xCjAwajEw1tw7mOYEGtiWXFl7iFGCQ5mJRHeQ9eAQrwpiZVVqUX58UWlOanF hxilOViUxHkvRgOlBNITS1KzU1MLUotgskwcnFINjFYhV9QTjvn3iXTMPaJ591oxX7+46OIl R6Vfdl49ZaFttW9v9XPj69duK1624Wue++bzZmk+Bt9zKyQsazbcKAv/Ne9LQcBBs+VJvBWf TkyY7fSrIe/Y0sp/ld72WqyvT4ddyjv87qLFtPKb54PTjyW4Nmfvjg2JUE7/eO/wtMPLUw1L 1jf6/lNiKc5INNRiLipOBAD04O8GbAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHJMWRmVeSWpSXmKPExsXCtZEjRffbrY9hBsfnsltc3jWHzeLt5u+s DkweDw5tZvH4vEkugCmKyyYlNSezLLVI3y6BK2Pbh8+sBf85Ks5ueM3SwDiZvYuRk0NCwETi Y+NdJhBbSGAxo8SZLcUgtrCAmMTOV41sILaIgK3E54PzmSFqwiWmb2ljBLGZBRwlHl2+AxZn EzCS+PZ+JguIzSugJfH66gGg+RwcLAKqEtvfJYGERQUiJFofPGWHKBGUODnzCVg5p4C5xOe1 d6BGmknM2/yQGcKWl9j+dg7zBEa+WUhaZiEpm4WkbAEj8ypGsdzMnOLc9MwCQyO94sS8lMzi bL3k/NxNjOAQ44zawXh9ovkhRiYOTqkGRinNgmuxLNn3vuZa7ef5Hl68QNrZPIer5YzFf53+ J0tmaHomSmbvuui0afafxz/8MtvX8kdFJDu/M3kq+ONw8tkH99tTOe6YqnRm6ka9rNw1db/i geM3vOZVfqh79uNiL1dItf3tN/an1X615NnzXA7LWLdrRqqQj8vCebzTRX6sm/Pw75N/jkos xRmJhlrMRcWJABw8NHzhAQAA X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:63.163.107.225;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(1109001)(339900001)(377454003)(24454002)(199003)(164054003)(189002)(47776003)(85426001)(5003600100002)(53416004)(92566002)(230700001)(83506001)(77096005)(86362001)(23676002)(110136002)(54356999)(76176999)(87266999)(81166005)(50986999)(65816999)(189998001)(4001350100001)(87936001)(2270400002)(65956001)(65806001)(586003)(2906002)(2501003)(5008740100001)(105606002)(36756003)(11100500001)(221733001)(2351001)(19580395003)(64126003)(80316001)(2950100001)(33656002)(50466002)(1220700001)(4326007)(1096002)(106466001)(3480700003);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR0201MB1513;H:milsmgep14.sandisk.com;FPR:;SPF:Fail;MLV:sfv;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BN1AFFO11FD040;1:rb4FdSEiV+i/nJmpys8J+MYOJP/NgxHXsVPIdxj49VACZfUXg8nJ2JCkwi7BKZ4j9pWK9aMSXzDfR8C0iuYoKYUZ/oc/GTb/etZWNnuhvl+TtMMVD72GL8XLJAVgdTAxBeL0TwGOv0aQiH5geWokm7gn9DqCEt7HaoBEKK2gp3nbrBL3Fgt0NtBkBJGH8AgxIoT2+ZZdqI+S+kJJCyg3HGryKLQj4oMjXmbG6ZGQwp3mmUsGSIbI4BNjavryPDjbOLR6D8kYd9yjeIOCttVQ4oPXLEezngbA8jSyDW4NBOnzYvi3jdP4qTwOB4ONQd0ulg8cHr8j9AaFbgVoSE50BI12zdmo7vH9PEW9O1VlZWWldpG5tLSSDUuY8z3UKUWZ8/CJR27L+zZqREv9ZpO3hbbdZ0AJKH+7FrYVKD+Vogp+BU55iCPMCChNawzfv3Ue X-MS-Office365-Filtering-Correlation-Id: 3e8c1a8a-661a-40ba-bd82-08d352ad29ba X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1513;2:eyqK5rzNa3qbav/b5b19c2cqn5MxUtqU/2R5okmVpm5WC4qbwo/YRo+3YXINnUYwSZCZ4FsByhszCoUtelijToosFhL8ElYIGv3O2SJZVVI0E25vaULJaRXRT/fheXz8KeTymALLY/5RPwSs7CtUfTQPMeNnFHif2XrDDXwb36kJqhc8uMuFeZp5QxP8bk8C;3:176KZtyOSV002qtFtNAkhI8rg+pHG4Ic1suruV5inFFNYnL9SGomximxAHVWDvChyRpRMMePbt+OOJMV/bvCdlqy48+Of2Mqo9OoKynO86JVrYMXL8fz5w7PjaMbqpOmFDMM3Ey0aTLrfVtVSLthl2jLdWaFyxatc8uefe60NoIC6Xui/zitbucqfuKuORiOsQwKQMj/IpXLR3KDB4EEN0f8VffoJENO9PNZmFBAeAA=;25:/YcLKCiQSK+Rs6J/qpOMRkpTLdVcmQm2KXFDbNskGQ3EZkFUdHQi17wXK2lOQea0TYtPeGwMNNqKbxbWKr9KOxbOlc8YtoBAmCKcgufAOr2VzhpAydXLt2NVMWCLxZ/maVgdntXMmg8h45cidn/zLBjQPpgLFgQPoG6cmI34rw/XIVCSKzzksFY+WrNU6LKOQYU2ivIDf7TkVxz8C78cfqZVBi2MtMQqBKjB5RfmtCVD16dc4bFY5YXMjFmHRgfyj+PmzK48Afp1mLx7cyHga9t4ej3FKHYIrVsWUvzm9V9Pd9Dc98bvYlK7afv+aO79I2G5eN3YVgdg1RllRoWtXHh/dY7D8xWK4MV0XlfYz9RvgsmPXXSSFqvdrItB7k8+MJVGbTBomt+R3C9wJWQ3zNHjUT9CCs5N2AW0Q09bjZ6vjJF0Z7kmHM5o7rtI0B7Z63ilDOnqVJndzzhMowTMmB8dZErfu2Gl/5gEu7vvotyFTX2Xd66oXw8BhSs7lo9n4AZwO40pUAZI3EGOOvFe+w== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0201MB1513; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1513;20:DZGflSyy1poGO6AULb32b+20euyzMlnHNPmlD87Cs+ZFADeuziu9xZyVWvdt13tqqk1gvDc+jW6GNoknjYdFoFT9cjJl0pPQeoxxusMODa5nVZZp2lBfW27DuisKbYb+F5RMIOYl6ScBjnveHyqcCqDzHXhUTI7QMq6Sg04/py+OOmYRwANOR3vZwn7CGJ7rCrhR3EMhJcdBgeV0Ew51uSY/8n7D+oT0j+VeGePooqUgfFAKEIjCNTegS8cwFrhsi1J2Yxwnzk99NEbdXyld5AEmtBgw0k8L6f7M0dtpOcerQzXkXo/3fWw9U0J6IoBWb0oeRaNKUlYgItJeX6SLQ4nVoVpBV4XK1N8Q6OnVRL5qOh9hWPujLrmJxaZR7QfvlGMcC2HmGJ1SBv6L4x/7ZV46GLy8ms/+l8332XWVBTx198L1Fd1mLhtoL34aL2Xh3YHNIiFHtGvgrmyDNZKExvGqgQgKFxYr/v7161i9sMgGfS6l7Lsvl0nMGbkVnORf X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(13024025)(13023025)(8121501046)(13015025)(5005006)(13017025)(13018025)(3002001)(10201501046);SRVR:CY1PR0201MB1513;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0201MB1513; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1513;4:EDgth0Mi9X87L9JSGDybppQBgJp79VVmW5ApGYVzEMV0k1koT4JDnN8Hryie4szkiMahdnF8PPTyBhopR0NBBfO9YL1csiXMmU4jCM0EiVML5ZJ1XR7JRjTIgeKzxgD20x02PuhcHHTF33fc/OY2yFglbUa4+CybMyR68PhjMk9Z3nLiAGziltNUFRDYo9YxYKjrN+l9yccY91cHMKO0YgmK6QeK4oDyNIlq26NJgBfyoomwSWIZlmnYTiLRovIpXUUSvWDbhHrJ7uDWaTDslS+iaHbyQwBMBWfrnJCoxRty/mveTGfZoxJfT+zS8yQQKHj6Ytw+jMZNagxcImaOxYLRQzo9YWPeA/HmlEfv5k7/60Nuz90dMbtpwTVf/dQJ6XoFQKoG5nwPvLStvPlwulFZQWlHwakLn3W8WVVsLfK6X4oBG0F34nBjJOy1BWL6bHvwfDTmLiymRw8syhwiqQ== X-Forefront-PRVS: 08897B549D X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTFQUjAyMDFNQjE1MTM7MjM6Tms2TnYxTGNoRHM5ZnlGS0ZweTdQQ0FL?= =?utf-8?B?VTArU1BUUmdMSG1UM1UzMlBhcTF1QjVLVHVqY0w0emZtNCtNZ3lzcVNaaDk0?= =?utf-8?B?ek11Z25IZXNnaFVlVzdQSENCY1lrZEk4UGlpdE4xck5kazdjWllsUzQybUxs?= =?utf-8?B?T1F6dlVLL3EwUXpiRHMvSEtPazh2eXF5QWNUMFFpdHhOU1hOUzBhR294bmdH?= =?utf-8?B?MHpqWTZWMUNaWGgxS3J5S200RXd3OXUrZGNLOWUyNkZCQ0lRNmlRQyt1RnlV?= =?utf-8?B?VTFPcHdvSkloVTF5ZDNoWGw3Yjd5Mk9udUNEN1dGRm5FeWg3OGsrdElqZXQz?= =?utf-8?B?R1ZWeG1UNWhVdzFXM3FUMEdhV1BpWmdSSDJTcGRpQTJUejBwQllrcGxwZ2sy?= =?utf-8?B?Y00vZk1oSEVHNnhPS3hROXJxeUt4WllhY2RHVWxXemZOenlwd1FEdE1kdExV?= =?utf-8?B?aGpEZXFURDE3Rml4ZmJiVEU4bmhtbWo3R3lwcEFMZCswZm1YOEVVQXlCejNV?= =?utf-8?B?SWpUdW9MSXpzNG1GQmhnVVA0aHFGTjUvZDc0b2pVTDZ3K0MvM0s5WWN3WWo1?= =?utf-8?B?WC9CbXFEVWhESkdKRktqUmFZV3dkeWJEZFVvTzVrR0NTcjYzVG1yd1ltcFdJ?= =?utf-8?B?aGx6UlBOMTdrRXlDRExISjRXRUIzYnBrRVFvUEZDRWQ0bmdXZm5lYlYzc2t2?= =?utf-8?B?OUVJSEYzM0U2NEt0Yk5EVXh0RUZHTTFHbjRIQnQvdEdzSDZ1WEtoOHcyTFRj?= =?utf-8?B?MnRSZUNyamxNKzNWUEtDakwxOU5ZbGxTSG1ZcW8zREZBTFlQc24rMjZiNmZ4?= =?utf-8?B?cDNRdmNqemEyNFVSMDZ2ZUVGUU12NFdjbG1kcXhyWEVmTWNybTZPRmd4L0J4?= =?utf-8?B?UktIRmpZbmdrVVlmYjhYZjQ3L1MydFZLWEtPNC9SSk5vSHB2bHgydlRldmFK?= =?utf-8?B?NmF4aGgyUVluQXBkdDRmaTFoTEZMT1ZzUW1BL3l4N1oxMG92b1ZxR3RQS0xx?= =?utf-8?B?ZmVYR2FCcG5IejNEcVAvYk1RN3V4L0E0R1NqRHQvY1VNVDFBTUJzMi91UmVz?= =?utf-8?B?aERBSEtHRlptSmlGUWVFc2M2WUd6Q0ZrRVhHOXdsbDJwdnBnQkxqUU1jMEhq?= =?utf-8?B?V3owendybCtmMXFjTHJkMUMwTVI0ZHIzc3d0VVJkdGpQYW85QTRLSGwxRVNH?= =?utf-8?B?TE5Ca1MyTTdwWTlvQ2liWEk2RWNmTURwdTg5OG8wV1BIMkVPK0JXY3d3WU5o?= =?utf-8?B?VFZqaXpnM0Ryc2NySW1YVVk2UCtqM1RLMFhaU2FmNGlwVVZPVWtJL3c2eDdR?= =?utf-8?B?djlhN1BERE9VZWVDcmdyczR2ZkRYQ0dUY1NHR2xQVkplZzlaU0NXMGJ4YVNR?= =?utf-8?B?N2tHRDV5QmIxZ202TVhvRVdoVHFZWVowVithbTNpaG15TW0zUGswMGNnY2xx?= =?utf-8?B?eDUvZTFuanh5MExCalhiM0tmYm1XRTlTdTR4S1NDVGVQZXlEZmZONWxSOE56?= =?utf-8?B?dVJYRWhwV0U1K1JTMGFxWnRZTTVMazN3ZDZPbFh1UG1zelErendnV0VkWG5q?= =?utf-8?B?am4relg4VkhFYkJGRDN1Um9COGp3Z1grRXhTdWpqNXcyWWwzVnIwME1Wbkg5?= =?utf-8?B?VnpNOFY4UEkxN05abG5VYTJ0NUlEVUdLVWtHZmtPMXUrZURST1loZEZsVlhO?= =?utf-8?B?eFg4cHR4bGZDOHJHTFYrTVJKekk2SHZZRzVXekhzVUU1WmVlMDdNRXdNZ0cz?= =?utf-8?Q?mwqDDTLsNtFPyGh+q/bu6rNIfD1xFaCoaYcY4bM=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1513;5:PnY8H1FwvyIecF+mEsWBiAazQ9CuFk0/CW146kwSQBv61hjsv8hdYeDwRjm4LkEd32faO/HTc5jiV/kfx/zDOpIN5snxE1PzWgs17BFQ0JOgjofjfdFYFMKfFPF/ftxThLH7DEORYTeCL1rFJsTGmQ==;24:gLEPGoX2Z4fP/zIlV14kPo8B0QfBXJ4QynGFe48utTo/Sto3PAEnxnIGdXtuvwMrNnHkEA6uYLVkxTHoY5ypV67X+Nhg0RGU8cP7V6fsafE=;20:+dKkwxKtftgsIN0G2CNw3IaUyKzotfcuvXSyJaBSRTBBWRLXQKlUuGNZzh4H4q/Dx8bsplCUhQUp+VhMaLc22hOUKOdLsZVQqDMZgy50Ny13WpqGl4RmJYeK7+q8hqCL8RAHzDfCiGRn5xOt7CzLu8MbwCr6x28Ky1u0MOyJXmRP0pWcVKb5EswQH/wPTgweQgtgrTvBV7HPNfblGYGhxyMNTBF2uxZ4IV1feVyeqYp8Q8kFJo5ZcdKBtMFIh2bW SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2016 23:53:26.9175 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.225];Helo=[milsmgep14.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0201MB1513 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/22/2016 01:45 PM, Paul E. McKenney wrote: > You are getting a soft lockup as well as an RCU CPU stall warning, so > it looks like something is taking a very long time in blk_done_softirq(). > > You have multiple occurrences at different times, so it looks to be > a long time as opposed to an infinite time. Are you perhaps doing > something that would make a huge amount of work for blk_done_softirq()? > > See Documentation/RCU/stallwarn.txt in the kernel source tree for more > info on how to debug this sort of thing. Hello Paul, None of the drivers involved in the test I ran contain RCU code that has been changed recently. The block and SCSI subsystems processes I/O completions in softirq context but until last week I hadn't seen any RCU lockup complaints when I ran an SRP test against a kernel with lockdep and several other kernel debugging options enabled. This is why I sent an e-mail to you. I have read Documentation/RCU/stallwarn.txt after I received your reply but this didn't provide me any clue about where to look for the root cause. Any further help would be appreciated. Thanks, Bart.