From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932078AbbIXSJo (ORCPT ); Thu, 24 Sep 2015 14:09:44 -0400 Received: from mail-bn1bon0064.outbound.protection.outlook.com ([157.56.111.64]:35126 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755436AbbIXSJl (ORCPT ); Thu, 24 Sep 2015 14:09:41 -0400 X-Greylist: delayed 70879 seconds by postgrey-1.27 at vger.kernel.org; Thu, 24 Sep 2015 14:09:40 EDT Authentication-Results: spf=pass (sender IP is 63.163.107.173) smtp.mailfrom=sandisk.com; gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac160a69-f79f76d000007db2-60-56043c5da05d Subject: Re: [PATCH 3/3] blk-mq: Fix the queue freezing mechanism To: Tejun Heo References: <560323AB.80900@sandisk.com> <56032432.6080006@sandisk.com> <20150924112251.2ec061fd@tom-T450> <56042844.60603@sandisk.com> <20150924165354.GB25415@mtj.duckdns.org> <5604346D.3080000@sandisk.com> <20150924174938.GC25415@mtj.duckdns.org> CC: Ming Lei , Jens Axboe , "Christoph Hellwig" , "linux-kernel@vger.kernel.org" , Akinobu Mita From: Bart Van Assche Message-ID: <56043C5D.3090307@sandisk.com> Date: Thu, 24 Sep 2015 11:09:33 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20150924174938.GC25415@mtj.duckdns.org> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupikeLIzCtJLcpLzFFi42JZI8azSDfWhiXM4GaTlMWrAx2MFv/3HGOz WLn6KJPF5V1z2Cx+LT/KaPH+x3V2BzaPic3v2D12zrrL7rFpVSebx+6bDWwenzfJBbBGcdmk pOZklqUW6dslcGVMaWpjLpgeVrHrYRNrA+Na3y5GTg4JAROJjzcuMkHYYhIX7q1n62Lk4hAS OMEo8evfDihnB6NE36xdbDAdHTMfsUIkNjFKvDj6hREkISzgKHFq5n4WEFtEQFbiyrSHjBBF /xgl+htfgY1iFjjJKNEz6wlYFZuAkcS39zPBbF4BLYkrsyFsFgFVicczb4DZogIREqfOvmWD qBGUODkTopdTwFRifXcf0BkcQEM1Jdbv0gcJMwvIS2x/O4cZZJeEwElWiXfNB8HqhQTUJU4u mc80gVFkFpJRsxDaZyFpX8DIvIpRLDczpzg3PbXA0EivODEvJbM4Wy85P3cTIziGuDJ3MK6Y ZH6IUYCDUYmHd4YOc5gQa2JZcWXuIUYJDmYlEd4DIixhQrwpiZVVqUX58UWlOanFhxilOViU xHl7c3VChQTSE0tSs1NTC1KLYLJMHJxSDYy6/HzGn5QWvxFusd92xXWL6dM2q9r1frlnZjQc ytilUJIvFpWg8Pv4zGal1yGLzpjXrGXrmBSz0fdk4PPC9L2z979ieLH1yp+QY88VLbewPVko JXBL/M4BhRpxd8fTr9JY32xyLBPSlWI3eHrh0bGEUrudjEkxdy3/GhfyBF/nYOSScj39LEiJ pTgj0VCLuag4EQCzwKCmnQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkluLIzCtJLcpLzFFi42Lh2siRohtrwxJmsOgNv8WrAx2MFv/3HGOz WLn6KJPF5V1z2Cx+LT/KaPH+x3V2BzaPic3v2D12zrrL7rFpVSebx+6bDWwenzfJBbBGcdmk pOZklqUW6dslcGVMaWpjLpgeVrHrYRNrA+Na3y5GTg4JAROJjpmPWCFsMYkL99azdTFycQgJ bGCUeNO+gwkkISzgKHFq5n4WEFtEQFbiyrSHjBBF/xgl+htfgXUwC5xmlJjYuA+sg03ASOLb +5lgHbwCWhJXZkPYLAKqEo9n3gCzRQUiJE6dfcsGUSMocXLmE7A4p4CpxPruPrCTmAXUJf7M u8QMYctLbH87h3kCI/8sJC2zkJTNQlK2gJF5FaNYbmZOcW56ZoGhoV5xYl5KZnG2XnJ+7iZG cCBzRu5gfDrR/BAjEwenVANjlYFPjN7PBJmqP7EPj/seLBar2Cjukb5gFrvWtH/hvWrVF8sY dLt8m9f2rrWTaFr173jDl1pJZjvvqU8lykqv3WMTfajzu7TL/MPHK1LPLZ65mX022thwULvn /tE1vHffHPmyxpBx4re8Q+d1Gz1XCOc4hx9QzDot7j2j4PDDGxLJe5fX9pxVYinOSDTUYi4q TgQAUQsI4BQCAAA= X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;BN1AFFO11FD041;1:tXm6RplvolzC6YSirY+ESgFa3yS7M+cxnTW8vULhR6WXbf0VOGxh5J1J6Pf79OX0KnzONxN4P8oD/HIjIGtR5qVjTDHxi5TRa4K48mYypRQFoAApDZwZigwqmHHaJf1uDqIUoSIgrTNEexcABtVCcPRQGsnpcmfXo99UuwjmwUVgis4SsafWvSEwXHFaCg971A4jVgxRbJeptq4+cqxuiN3+eIvc4DeIoSMBuVlMJvjQXCZvJqTVULj/u6flUpV1G36pCJHYZ39JXrv06twknEJE4i1Hx8R31inb8t35+Ee+qtEs7TTPiAsBVkITCgKOKe1dRIYNSoKYtd7Oq3RP0+QzPOkQ8rn0ahbiYYa/SGwaFl/v4Sih/2D33JeJEdSi X-Forefront-Antispam-Report: CIP:63.163.107.173;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(438002)(189002)(377454003)(51234002)(479174004)(199003)(24454002)(2950100001)(64706001)(47776003)(11100500001)(5001960100002)(110136002)(99136001)(23676002)(50466002)(77096005)(65956001)(80316001)(68736005)(92566002)(54356999)(76176999)(87266999)(50986999)(69596002)(62966003)(86362001)(77156002)(65816999)(65806001)(5007970100001)(189998001)(87936001)(4001540100001)(64126003)(81156007)(97736004)(59896002)(36756003)(93886004)(5001830100001)(83506001)(46102003)(106466001)(5001860100001)(4001350100001)(33656002);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR02MB1364;H:milsmgep12.sandisk.com;FPR:;SPF:Pass;PTR:ErrorRetry;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BY2PR02MB1364;2:nHiBeLt2pVsqRR+Mqe3w7/ZayBqcd9hFd5l+9pkSqU8bbaZ25d9HljNRrkzYzHpf3fCbujAIrJSgF5/FXLWbQFWzghS2o5wj5C2yInQtgQdkbo4OZhuM3J58pcT5wBI9q2ggk4CHaMa8QMHM4EhLtujL7LTOQL043ADtk/2Jsh0=;3:hCnu8kNp3hRI+ElcN79Xp7gHXICMdIgS3kAmMrlmvcNDMMURbTfzwyZU3jvuNIuDGtzg1rh3bYJ5zRc+FBiceuregz60bda2QYvFKoA0FUMYy/B9t0KacuLpIXE6M2VvG3M6UgfJ5fLIxEYanmun3CH49fdKZMRwRLFD2jrVijai3wQr7rkK2LA0F6VhmW5bQGO6xqE0KEdtCBm9DbS++xlAUF1d9Rraxt8NsfFxmDKsKzJqgMFGRcP7QLFrEDBSBtgaacherUvx7fWuq5u99Q==;25:hTJ8vXiUTdz/Fe7HiJg5zHwQgPYniC5Y9hyyjgcxUeqBJntYC1q4NgfCJTIvlW/hQsNoH2ZhClUyoDHOJ8SWNlNDUX5ngecIqbEiTfn9nG1k3EtBtT0zybODyEvxBrdwTxo6IIPe2H5ScH7l3D1LKIcbJc6nwDdX7O3r3RPPz5715iVix7DkzxgEah5a1MUD1TTDChVga+bRKXgJ6sWpgLyNWHzIsez9vfFgVPsom1qJxvu8NFQcQ39VKOtQ0uXUSME9ddHgyFYVzL9EAYbArA== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:BY2PR02MB1364; X-Microsoft-Exchange-Diagnostics: 1;BY2PR02MB1364;20:gJ4hKlwaZ/ci9FRl0ye2N4NPzqWLVtVcdnw21HIkVrRjkc0L5008TynkmkJlq2To28l0BPRvyzPFskrgV9PZcPmrXDlATAqDDvNXqttU2NxFMWZAzQKfpZvYGck2HlTFAK+VBlvR4A+pD+ZKndr4rQhHb51RhfAhVCV6Dh4bX2kdEzhAOJWr0VB244Jtnh3qzK7QKXMlivdhFQGiiBTbW+YQJN+sMmlGhonfkQDaD7s7QdSi5ZBOlpnAqK7C+QH5HSsd8fwy2avBA+U/TyY+6BgzCTnUjuKnlvBZp0o7TrtJZucS9uPpzntutbi+YmpZLUCih1iOh5DFYO3wiThuw4UWVQwmxedKrcj75d/TO4hxppqmEG2Hf0MuRwqTDwKJkmlFaBXojeaTYLJ55MmxfGT1gNIJ/ew0G6M4+yAoFGFh0GvSq16MzudlG4AcOsFAX2Xxg1WDFQJOgNSxaRtZMkUXoy64iggxH0R79KqUdk2bJT2Pin3596TeUAo5qdHN;4:vjieRhjWbCQFh2mqFb5OpnntfRI5Cb8t0BGOeVlAWHDa1OTixMPHZgtmPl6s3jP/PI9xMAPt4fG/JCpFEvwyfyS68rqD3W29XgVpaJ/n+mqBb+3y6huH4ew73HKyZ4j0LWNPgLVjfmFZCc/0Nn6cBIqAIcRYyuG9F5kQ0oE4ovhxjNSrWz1ynGtx2zUmyg7AFUAzu+ssUbkkMkasSFC6ozBikQnTbbL6bcvGPNmvPYJvG7OgyTOzTFD+qAi8seV7e9aCNE6ul2j/hjrtkcTGOPPqIqgwtXIK3iD73iohz2HAaLv+LAsKPgmCFnPSie/aJQFU+oQv23ztpQzmZrAL20otmzsU+O0+dyPcKkBrJ4A= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001);SRVR:BY2PR02MB1364;BCL:0;PCL:0;RULEID:;SRVR:BY2PR02MB1364; X-Forefront-PRVS: 070912876F X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCWTJQUjAyTUIxMzY0OzIzOkluV0IxdFo4bXRPc2V4ejFqeVN3ZDYwbUdy?= =?utf-8?B?dk1PblBSNzkwNDY3amhNNWg0SlN6NTJRNHZuWUswd0JFcWdCQjY1aHRPVklv?= =?utf-8?B?STRuc25MS0RzY3VaaGl6ZGIxbmxxd1pzMEwrakc5eXI1ZThwSVo0K1BjWGJ0?= =?utf-8?B?NDhzY1RMSGgzdVMwTWRqSGU2QnIvcnBzY09nOEVIcTNuVlhlZFhLUXRvaFRk?= =?utf-8?B?T21JZjNGSitWYVd5Z0dONHNZWVBZWDc5d1pWRHhlMVAzM0p0dWg3UVJtTVJU?= =?utf-8?B?RDlqSnh3YlBMTnlXYzNrbDNsRXV4ME5TZkw5WTJGcUoxUEhTUUF1dzhKUzBx?= =?utf-8?B?S2V5aEdwcXEyeWVQZ2Q2eVVkNVZSaDQ5VEx3WGRwYlozOGlnQ3R1bVYxREJP?= =?utf-8?B?M0xtQUV4dkpybFIvNFE0V1gzb1NyeXVValZ3TzU2b1pNTUQrb041V3ZIdTR5?= =?utf-8?B?VC82aGlVeUJSNmJWbTQ2ZzRmdTZ6Zm5uODIzaHh6bHJKVUNiNE8zSzdqMDhq?= =?utf-8?B?aWRTcGVQWk42dmZqUVE0dGg5TnVFWVRUblhLdWgvZjBzbDJDa1FuK09SN1VL?= =?utf-8?B?dkRzYmZUZkVySEorcjZGOXIrNkVOaW92c0pPa0Vvb1RRTTFWMkZocUR6V05o?= =?utf-8?B?ckl1elVSdDRTQ2dYbXNjYkRlaDZlQmVLc1B0R3Q4QVdsQkQvVVdrZW1zU01x?= =?utf-8?B?MlFDb25xWS9LdXlyR2JQTE1yK1BjamNhWitoWnd1azFGeU81T1hGUmJVZlZu?= =?utf-8?B?cndodGpOVGxaYmdnZ2tKb3h4UkdtRExYQXM1SExCMVFDbnprSE5Gd2dJK3dn?= =?utf-8?B?SU0xNEIwcmc2R3pGUGhlT2h3a2ZjbW44YS9LN28vOWh5bHlkMnh4TVhsR3hO?= =?utf-8?B?QzhHQnRyN2piNFk4NDUvVUg3VGxNRUFOYjZKNmpwcEdlVlptVWx0ZnNVekFD?= =?utf-8?B?cFlNOHhXREpRc0s0QjRmaEVqbkhOdjFQZkJHRHJzYk9wTXJGRXNXNWR1YnND?= =?utf-8?B?WE5FOEFCWmZXVEliQUk4ODQ5dlVGUlNvcENhMU82SHJMU0RQc0NPQW1zSDQ3?= =?utf-8?B?aVhrVW1nT2JLNzVVblV0TGRCNUs0V0xXRHNWazVEVzdZQVgwdUtaTE5LWmlx?= =?utf-8?B?ZGZndzRHeWllejNIRjZmQnM4RnBMQU4xL0g1RTFrWUt5bWFLbjdBN1RmWENw?= =?utf-8?B?ZC92eGdQbVpYWW9LcmVmYlZ6enJwRzJwTmFlL3NQSmY2bkFkZ3huM2dPOTdm?= =?utf-8?B?aE51aGMzbkthM21hWGR2VzFiNkpGRDhMWVV4RlJCRWJ5SWFqQThkdkNGb1k1?= =?utf-8?B?YnFEVm0wWVBaT2lXT0tZVEVVTTRkSVVMcDE2WjNGTWN4Y1dzRVQ0b2p5YkNK?= =?utf-8?B?OTg4NHZtYzZaSFNuK3Q5UlFmZDNvOTNCLzlDdXEyTi8zSjFRWDFxdXJHbVhZ?= =?utf-8?B?a1ROTTdTd2pnSGN1Z2pNQXEzc3ZxNXN5Z2ZjQXIrNFg3Q0RnY2R0NnE3cjA1?= =?utf-8?B?MUxkRTBoL0pOUHlPdTlyV3RRdzR4bkJLTUZveVBGUXJ1MURYWnY0dmtjU2lv?= =?utf-8?B?S205U0NGcWE4dzg2RklmY0VwQVZlaFcvaEIwYXg2bFM1Z3kvSk9qUDU5VUc4?= =?utf-8?B?SVJUT3o4YTFweGtUQ0phcTJwQjlXRmUvbUFPZURhQUt6TlFMU2JjSkVnNWp4?= =?utf-8?B?QWZDUUc4SG1YRDBZWmFuSEYxcUxnaUZ4RWVvZ1Z4QnVaQ0dwMlR6TzZncGJo?= =?utf-8?Q?kRA5kqdbpd9LWzBWLQV+KI6vl+BmmVt2H95DA=3D?= X-Microsoft-Exchange-Diagnostics: 1;BY2PR02MB1364;5:moo3Ic7JVPbldJsDsSHy08CyJSVRQY2lANcQ0we/Hc8i6K+NnkJiSi1qpUaLHrmyHQTht1anyBo1bmgxNlzVLv19XP+sTpbvfxZIKtQYMC2O/tSVMqdLZW6fJhyBnGaTSglEw/fRnZ7Wlu1Sy6DXnA==;24:rSantZ0pI2Izoi1R8gZbH/e+jHy4xII+91LvCjzvYpfXnML3cczt1lAdhMd5shBYgN6rAFFRq7RweGDZ26QkGeY0ttt90f7aI/9gs43APyA=;20:c1sv+lYJ+nbLxiAFgvQldBYggp0C93CZV6274Hi8JjWnKFa5yW5eevsH80lO371tRc3eQ6jCn9p7YoA4NnuydGFpz+q+Jix7a8DnUDQ/rBuqKC/aUL1kKwXK7GFizFLN3BBOyPVRzLGhDLb+3kNBpymbCKrSOj7uaqZa66UQC6XnAM3r2drNfS9xXFlejUWIX4KYV76YDa7TglQ4fAYu8B4DrLOwMx7RIBi+bbf4yp/ksFEdfFXMkg3UO3t46uHR SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Sep 2015 18:09:33.9430 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.173];Helo=[milsmgep12.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR02MB1364 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/24/2015 10:49 AM, Tejun Heo wrote: > Again, that doesn't happen. Hello Tejun, In case anyone would be interested, the backtraces for the lockup I had observed are as follows: sysrq: SysRq : Show Blocked State task PC stack pid father kworker/4:0 D ffff88045c5d5a00 0 29 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff88045c767c08 0000000000000086 ffffffff810ba11d ffff88047fd15ad8 ffff88045c5d5a00 ffff88045c768000 ffff88045c768000 ffff88041c737ab8 ffff880036f7bcf8 ffff8804158b55e8 0000000000000100 ffff88045c767c20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/1:2 D ffff88045c648000 0 264 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff880444d47c08 0000000000000086 ffffffff810ba11d ffff88047fc55ad8 ffff88045c648000 ffff880450a9da00 ffff880444d48000 ffff88042826c8d8 ffff880428969790 ffff880413b27b50 0000000000000040 ffff880444d47c20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/7:0 D ffff88045c675a00 0 27179 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff8803f7333c08 0000000000000086 ffffffff810ba11d ffff88047fdd5ad8 ffff88045c675a00 ffff8803f7139680 ffff8803f7334000 ffff880403d76e40 ffff88040c412408 ffff8803fe493cf8 00000000000001c0 ffff8803f7333c20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/3:0 D ffff88045c649680 0 21529 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff880392f2fc08 0000000000000086 ffffffff810ba11d ffff88047fcd5ad8 ffff88045c649680 ffff88039a72c380 ffff880392f30000 ffff880420f2fab8 ffff880425896ed8 ffff880416043cf8 00000000000000c0 ffff880392f2fc20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/5:0 D ffff88045c64ad00 0 10950 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff88037111bc08 0000000000000086 ffffffff810ba11d ffff88047fd55ad8 ffff88045c64ad00 ffff880055991680 ffff88037111c000 ffff88041ff348d8 ffff880424463080 ffff8804145155e8 0000000000000140 ffff88037111bc20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/6:1 D ffff88045c670000 0 17446 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff880052effc08 0000000000000086 ffffffff810ba11d ffff88047fd95ad8 ffff88045c670000 ffff88035e401680 ffff880052f00000 ffff8803ecf8ee40 ffff8803eed58b18 ffff880067a88b18 0000000000000180 ffff880052effc20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 kworker/0:0 D ffff88035d205a00 0 23356 2 0x00000000 Workqueue: srp_remove srp_remove_work [ib_srp] ffff88035ec83c08 0000000000000086 ffffffff810ba11d ffff88047fc15ad8 ffff88035d205a00 ffff880360a3ad00 ffff88035ec84000 ffff8803e75a61c8 ffff8803eed59790 ffff8803e4c10b18 0000000000000000 ffff88035ec83c20 Call Trace: [] ? trace_hardirqs_on+0xd/0x10 [] schedule+0x3a/0x90 [] blk_mq_freeze_queue_wait+0x56/0xb0 [] ? prepare_to_wait_event+0xf0/0xf0 [] blk_mq_update_tag_set_depth+0x44/0xb0 [] blk_mq_free_queue+0x50/0x110 [] blk_cleanup_queue+0x148/0x240 [] __scsi_remove_device+0x65/0xd0 [scsi_mod] [] scsi_forget_host+0x69/0x70 [scsi_mod] [] scsi_remove_host+0x82/0x130 [scsi_mod] [] srp_remove_work+0x90/0x1f0 [ib_srp] [] process_one_work+0x1d8/0x610 [] ? process_one_work+0x14b/0x610 [] worker_thread+0x114/0x460 [] ? process_one_work+0x610/0x610 [] kthread+0xf8/0x110 [] ? kthread_create_on_node+0x200/0x200 [] ret_from_fork+0x3f/0x70 [] ? kthread_create_on_node+0x200/0x200 modprobe D ffff88045c64ad00 0 24137 17294 0x00000004 ffff8804146e7c48 0000000000000082 ffff880400000001 ffff88047fd55ad8 ffff88045c64ad00 ffff88045c5d0000 ffff8804146e8000 ffff8804146e7df0 ffff8804146e7de8 ffff88045c5d0000 ffff8804146e7dd0 ffff8804146e7c60 Call Trace: [] schedule+0x3a/0x90 [] schedule_timeout+0x1f3/0x290 [] ? mark_held_locks+0x66/0x90 [] ? _raw_spin_unlock_irq+0x2c/0x40 [] ? trace_hardirqs_on_caller+0x102/0x1d0 [] wait_for_completion+0xd6/0x110 [] ? wake_up_q+0x70/0x70 [] flush_workqueue+0x1cf/0x6b0 [] ? flush_workqueue+0x5/0x6b0 [] srp_remove_one+0xc1/0x130 [ib_srp] [] ib_unregister_client+0x11e/0x1a0 [ib_core] [] srp_cleanup_module+0x10/0xb9f [ib_srp] [] SyS_delete_module+0x16f/0x1f0 [] ? trace_hardirqs_on_thunk+0x17/0x19 [] entry_SYSCALL_64_fastpath+0x16/0x7a