From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: [PATCH 08/13] IB/srpt: chain RDMA READ/WRITE requests Date: Tue, 29 Dec 2015 10:58:24 +0100 Message-ID: <56825940.5070404@sandisk.com> References: <1449521512-22921-1-git-send-email-hch@lst.de> <1449521512-22921-9-git-send-email-hch@lst.de> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1449521512-22921-9-git-send-email-hch@lst.de> Sender: linux-scsi-owner@vger.kernel.org To: Christoph Hellwig , linux-rdma@vger.kernel.org Cc: sagig@dev.mellanox.co.il, bart.vanassche@sandisk.com, axboe@fb.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org List-Id: linux-rdma@vger.kernel.org On 12/07/2015 09:51 PM, Christoph Hellwig wrote: > Remove struct rdma_iu and instead allocate the struct ib_rdma_wr array > early and fill out directly. This allows us to chain the WRs, and thus > archive both less lock contention on the HCA workqueue as well as much > simpler error handling. Please consider folding the patch below into this patch. Thanks, Bart. [PATCH] IB/srpt: Fix a recently introduced kernel crash BUG: unable to handle kernel paging request at 0000000100000198 IP: [] __lock_acquire+0xa2/0x560 Call Trace: [] lock_acquire+0x62/0x80 [] _raw_spin_lock_irqsave+0x43/0x60 [] srpt_rdma_read_done+0x57/0x120 [ib_srpt] [] __ib_process_cq+0x43/0xc0 [ib_core] [] ib_cq_poll_work+0x25/0x70 [ib_core] [] process_one_work+0x1bd/0x460 [] worker_thread+0x118/0x420 [] kthread+0xe4/0x100 [] ret_from_fork+0x3f/0x70 --- drivers/infiniband/ulp/srpt/ib_srpt.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index 8068aff..3daab39 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -1395,7 +1395,7 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) { struct srpt_rdma_ch *ch = cq->cq_context; struct srpt_send_ioctx *ioctx = - container_of(wc->wr_cqe, struct srpt_send_ioctx, ioctx.cqe); + container_of(wc->wr_cqe, struct srpt_send_ioctx, rdma_cqe); WARN_ON(ioctx->n_rdma <= 0); atomic_add(ioctx->n_rdma, &ch->sq_wr_avail); @@ -1418,7 +1418,7 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) static void srpt_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc) { struct srpt_send_ioctx *ioctx = - container_of(wc->wr_cqe, struct srpt_send_ioctx, ioctx.cqe); + container_of(wc->wr_cqe, struct srpt_send_ioctx, rdma_cqe); if (unlikely(wc->status != IB_WC_SUCCESS)) { pr_info("RDMA_WRITE for ioctx 0x%p failed with status %d\n", -- 2.1.4 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752884AbbL2J6h (ORCPT ); Tue, 29 Dec 2015 04:58:37 -0500 Received: from mail-bn1bon0089.outbound.protection.outlook.com ([157.56.111.89]:54547 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751649AbbL2J6c (ORCPT ); Tue, 29 Dec 2015 04:58:32 -0500 Authentication-Results: spf=pass (sender IP is 63.163.107.173) smtp.mailfrom=sandisk.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac160a69-f79f76d000007db2-40-56825942d390 Subject: Re: [PATCH 08/13] IB/srpt: chain RDMA READ/WRITE requests To: Christoph Hellwig , References: <1449521512-22921-1-git-send-email-hch@lst.de> <1449521512-22921-9-git-send-email-hch@lst.de> CC: , , , , From: Bart Van Assche Message-ID: <56825940.5070404@sandisk.com> Date: Tue, 29 Dec 2015 10:58:24 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <1449521512-22921-9-git-send-email-hch@lst.de> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpkkeLIzCtJLcpLzFFi42JZI8azSNcpsinMYPZ1Q4v/e46xWaxcfZTJ 4vKuOWwWzw71slh0X9/BZvH6+FMmBzaP6TM2sXpMbH7H7rH7ZgObx+dNcgEsUVw2Kak5mWWp Rfp2CVwZD5evZSxYLlhxqfcHewPjc74uRg4OCQETiWf/pbsYOYFMMYkL99azgdhCAicYJa7P YO5i5AKydzBKvHl4kRmiyETiwvI5rBBFmxklrm5zApkjLOAk0bLYDiQsIuAgMePTTHaIkmyJ p8saWEDmMAv0MEocvbefESTBJmAk8e39TBYQm1dAS2Lio/VgcRYBVYn2R6eZQGaKCkRILNqR CVEiKHFy5hOwck4Ba4mpl9qZQGxmAQOJI4sgzmEWkJfY/nYO2M0SAjdZJb7MncgIcYS6xMkl 85kmMIrMQjJrFpL+WUj6FzAyr2IUy83MKc5NTy0wNNIrTsxLySzO1kvOz93ECI4YrswdjCsm mR9iFOBgVOLhzZjUGCbEmlhWXJl7iFGCg1lJhNd1C1CINyWxsiq1KD++qDQntfgQozQHi5I4 r3WLWpiQQHpiSWp2ampBahFMlomDU6qB0d99kTRX2Kwoq4rk6r7ihtS85N+GwUrtonv3sTIb ei6RW3r7sWvCizO6GVZ/P7zgnxUXIDvn7sJ7AWyCb79MqKzUe2Hj3tNwYRaHss+xRYulcoPu KTgmGtiULpRe2hWnvSHVbN/JWrejCjZlC9r0n2xjnPx8i75417Y10qXKUzUS90/5q7RHiaU4 I9FQi7moOBEAavtgHpQCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpmluLIzCtJLcpLzFFi42Lh2siRousU2RRm0LdB1eL/nmNsFgd/tjFa rFx9lMni8q45bBbPDvWyWHRf38Fm8fr4UyYHdo/pMzaxekxsfsfusftmA5vHtDXnmTw+b5IL YI3isklJzcksSy3St0vgyni4fC1jwXLBiku9P9gbGJ/zdTFyckgImEhcWD6HFcIWk7hwbz1b FyMXh5DARkaJfVNfMnYxcnAICzhJtCy2A6kREbCTWP+6CaxeSCBb4umyBhaQemaBJkaJe193 soEk2ASMJL69n8kCYvMKaElMfLSeEcRmEVCVaH90mglkpqhAhMSiHZkQJYISJ2c+ASvnFLCW mHqpnQnEZhbQk9hx/RcrhC0vsf3tHOYJjPyzkLTMQlI2C0nZAkbmVYxiuZk5xbnpmQWGhnrF iXkpmcXZesn5uZsYwcHMGbmD8elE80OMTBycUg2MQXe+vPOKO7XqlcCCT4z6vSEBFy8rLhX+ ne2sGTPpp5Li95rq4JdztC+IhyXN0fLXMDY87BfweqXE7onS6vbfz+42VkyJiRG1ZDL9KrGQ rUEy2+fVmpXKW+78vDOr7s5eK52DhQeWPxW2FfQ/sqVZ7/EswTr2+SU1DmVJ+TUR1kzSa41S 6pyUWIozEg21mIuKEwExhOJtFgIAAA== X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1;BN1BFFO11FD032;1:yQA6cuOceAS6YmrKN+IgeMTIVTYofeWIszb8giH8r/eN4xsFcb73mFM6TEjlolfL6JfYTN1HWeeitByo7NwoDyD3/1LPDIY2/kWPs2Q/qEF+1LA+YF6Za9yHaYpnXyp5NhlVr6PtoVPtTGabEZh8uEpe1dLsGGLbL3pFAfGwLm0OBRFiEg20QoEaSMSxsDBaxQITA5GK6Do5xOCGjpuZ2Z2i11NGTc6tmQsnv/Kh3TBIgt5dweth67cyBwa+zoSaYdbEc4Fiug5kUPPdYHL1XqowBBxnxj7xiHPlXT26u7/025UD1VeSfU9obaNcoFA+2dY82k8iRpbspdCqd0sPbW/i2MhVR8CzUwCXdP9YnhukdAetbaBK32qzJr58lNtB6q7ZEk1mjKQlvOtG35gxkw== X-Forefront-Antispam-Report: CIP:63.163.107.173;CTRY:US;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(2980300002)(438002)(479174004)(189002)(24454002)(199003)(377454003)(164054003)(87936001)(64126003)(65816999)(97736004)(5008740100001)(189998001)(4001350100001)(11100500001)(230700001)(81156007)(77096005)(59896002)(1220700001)(2950100001)(586003)(92566002)(5001770100001)(47776003)(65956001)(65806001)(5001960100002)(50466002)(106466001)(36756003)(50986999)(83506001)(54356999)(86362001)(1096002)(99136001)(69596002)(87266999)(23746002)(117636001)(76176999)(19580395003)(62816006);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR0201MB0826;H:milsmgep12.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB0826;2:HEbsVUpf9yaLko1etuM21msMwhcIAm9QoJl3FFk7GDUOUWUSRJ42JZDlApDSpEK5BrGRurBDZ7q/YRTLzAbkIgYUhhc/qDfcQ/7N/se3laJU0pI0SAIjj7U31Ar5pO/tWq0cNbJUW5xUUQfHr+vjmA==;3:HAsOjSeGjGwe4cLzj5qeZK4IrjoWl3EqXweZzVDYlQAM8yBoHB/Ig1cA5xaq+2qwSKn5GKTBfAm9biMQqPs/GfVuSZWt03CXAhLlHYRJNjqkW8vO7BRo25eoWqzkLbx5rNwcZSQlcbx0PUkZiBzl3sDheqVt2cgVP6lSfgcNvoSG/n/BNPx9ZCLMLVsjnBNdznrUXyRAy1nvubpRNeefPr72zb7fWk3aAsreskGxcjga2uWhybxGzg/ZBVVQ+nEtrbT9d9w+fl0guguGAmfUzg==;25:FP4MMz8ZYkDx5/jqQX7AEJcdvXxweNoDTYit+k+ohSgoO76QRPLiHy/oBlL4nvVqV6ysIeiD2FxsnD7fxQxGNe860+PrzYe3XC7iD8pBTQ/uXqERmMmZZQckvr6yO5iKcMU6DWljLJ2AL7U/whJgf3W8mvK9HWtqw6/VpnnvKXvLowFPbvnDnE651Q76PDI2Pb7CJkBbX1n4wICdkw5A2vJlvqQt0cYVEg4oikCNtzeLBTwA4xSXqSpHaWJ5O1MD X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501001);SRVR:CY1PR0201MB0826; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB0826;20:MaJhNvNEm1Cgyh8swl6MQHiK/jADwRIMJsSK0Udn0rjVE/TYyrqWh5dj21j5Uj/FhZDhAfHe/LhzpAnkWvFF7n6DNkv+Z+Nbc7Ku+LpkyBA/mNFXzzytZJ5JAdNC9plWhYC10rST6dJydtH6uFahQT1KOWgXR8CDth84YXfc/8im2PcXg9tFIj5giHtfeR6Hdb1jvIqzoTLvE8BuPpEiAxbHfvX32QkXsQx3Mgi3CWpDpejycj6kOL4PF4qMNtdOkWunldfc7BoVMUAWLswZn81OZ9O76T9Rseznj1mLaFAsiiVljtsm6jJbONu/Vw41OGoXCtNbGeSB2hqygXVQq+Q+ak7Qd3W6Qw7oboTY4wb+yU6jPIxcw3uq5TYuf8/mc6Bq7CBV5Ukf4KqI5oXhGzKV+ipixrJV+phyKUDAe8hBrIkLkHKcefERIN3LRj2wwRap8yN/WDClMtvoN+EUD0l2MEfKOYMIBd2sTi0qN/y3s4MrYZeRWbZz9g0cUaa6;4:a3WOtdYY2j9RfqxRqpoXR0vDafEg24Vi7ZLMK4tvDqSCtIsSJQ/yTHpnLt4i7rTChdvO+Xc8aBupTTK2oaDrkrNrKEVuVCn4EoJqg9ca61v3/B1pfkL2lJB9gBd5ozwfsDYQHIRv7XKbGbYeHj5cXPPfroFo4SP7jrTEStqCsx/lYGNb9ENdqwYgDnyrKiOW4EVnOpWpMKb9TuenVt6oi70ixqmt22ntf2xSNuRIWn4V6Z7Bag71jjgpvZj/BpCWRjbUnvGkQzJAgEQQhLl/cIYhbKzk/tY1tbyTT0lbfFgx4AmqBnGvGCvE9mwc+zrY/fspCsUv4lrv69nsU7tyJJK5Q4zr2SZV/BJqFvqHKTTpKKBHGj2OFqo87TpHF90t X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(520078)(5005006)(3002001)(10201501046);SRVR:CY1PR0201MB0826;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0201MB0826; X-Forefront-PRVS: 0805EC9467 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY1PR0201MB0826;23:7SMN1AQhNkupmGgp+ywoKTFbp7+zAk+4loa?= =?Windows-1252?Q?MGYUAUP7CsiXnygwY35yzvKXAfWFpUF0BZvc8/0f1fk39y+n2crRLBKR?= =?Windows-1252?Q?6fTK5C5XIZ4LQSjMK35jYJ23PpE0wSwkZy7JJHYPpJxhfW2QAAZ8TSe2?= =?Windows-1252?Q?vssN3zI7Cdcgq4DBjMyOBK/4pVOnldfvRHVtVg01+pMllv4O4GMD+04u?= =?Windows-1252?Q?g3zv/ITvJ1wdTigkSOI1oCNa8ve7/hPflSg8k24QUAz2eX8AHlcV+EEr?= =?Windows-1252?Q?C4rjchl/yn/7FtT2KzkW2tzbsIA8NyD+PjGU5oaK8jeSGuVCNrFGIJRb?= =?Windows-1252?Q?3LydeiGLbLv8Y/l3R1BgeLhxFW9zEBopLeAZCHQ+HprSeSxS87iaTfp+?= =?Windows-1252?Q?Wj5Y0+3cIc3zif+5XetB/iAjxtgP4HbNKwDGvGPqOPyOju8abRbd8bMu?= =?Windows-1252?Q?s4Lqfr+hmlNaH40x/PH6uGnPn1mktqmJuDEAcGIJ/RQJhTN3hJGVpJ3J?= =?Windows-1252?Q?PF+vY/eKNzxJ9AMGixGN1JY6qxPaNMb6SpXoqZ2k8i6aHXMuLbGRBGAU?= =?Windows-1252?Q?LqYOo66acy6ta/sHun8p0wGDIVz7oXwGcB9Ym17Rad2/shTAZeoHWbeE?= =?Windows-1252?Q?shr0kGE7Yuo/yrnO3ZBMZGLxdRQC0DunCvOUHk2Gm2HEcuMR8M2nIbjX?= =?Windows-1252?Q?LiZSCgJTo9tQU6Nd3gYGHOEGj58ryaBdn/99KGrLnG8A3D0Uogp2mC6A?= =?Windows-1252?Q?umc7OpgC2d3BNGWlcIvkrC5NfbN8hmVj9xXvH7cr4Rxbz3VKK6L5IWEg?= =?Windows-1252?Q?UJKDY0wuj939nF6e9K9+5sYVECa/541GH8EUCgBvPp8tjNyZNWEEiRTw?= =?Windows-1252?Q?VdfKrvsalFPQ6WZrjI3CdznuqCpNz7pn553h/X4O+o3PPSucWnyYlkOt?= =?Windows-1252?Q?OdXFDF0P2VJQXRoBodS/xZRMcXj280rmtXszd35GH+wS1ryu1AK3Rpdf?= =?Windows-1252?Q?AHGEBR8cQyAEPl741gc+EUOk9fjhQBoo/QZvprJj3UBTyvUTXqMT/Vms?= =?Windows-1252?Q?nHs5eyXHHjQ6lUbOQ4ZlEtHr+i2MA3ncnz7idcwKSn5FpdrGhg3zW4Yn?= =?Windows-1252?Q?0tqMg8hk19JtAjxO8ZEmwfSFR/JMn8hQrBjGFTUxkwVCAuzWfWmhDq5+?= =?Windows-1252?Q?BELn+AydIFZkPWApmFbz5XP6kIPRpb7ysix2rMJM9vvGrvcKsOUpwR7R?= =?Windows-1252?Q?DAzjX5C17Ky6AyOHvURjkjepGhxoyRHmZvHGlIRjYOejl9QwiJX0ogMZ?= =?Windows-1252?Q?m/PLn?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB0826;5:k+LaAcw7h0I4zMhL2/PPDCzWvwhEdi97aqKnh2ZeAoVfGbDbnJORTVBAFQeyRdLvfTQclsOKVkhX9TdkdGCKvsgKK1H6LH86Lo1aZgVOTfOUrRO042eCW59XeOOiVfDiYuuTGQOmILLB8cuaa7sJCg==;24:i9QzP7W6aZPL5txO/PT+a1S0ulXVxq/X/fMpjoZ/kNvINJ2LbkctkWF9NtkBHJ7B+EmGhvyEYZO+mxlOmokb7PL6a7qELm1duDKrRQzJ4/4=;20:CUL8Z9+lmxGD0Fr/hMWU4DGqIvLU6YlWUqTr4eaY1R2+B5xu4XsRu9nmv44bckxMGbiyq+a5p7UlyCB9+wach+fPHmKR03+EMayExhCN+h10T9qf8/gCOjayjR2/JnuWPHKW53W8NxG/pZKO93asi2Z/ymgl9zItKV5h0P12pblkjy1uGfnZ7tLWIPeGcUYXSZtXVoOtq0y+hstnSNtw8/ASmvS0SMkKFc00fU/gPFAFLWqQm/Lb/zOlzTi/FoZL SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Dec 2015 09:58:27.1906 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.173];Helo=[milsmgep12.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0201MB0826 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/07/2015 09:51 PM, Christoph Hellwig wrote: > Remove struct rdma_iu and instead allocate the struct ib_rdma_wr array > early and fill out directly. This allows us to chain the WRs, and thus > archive both less lock contention on the HCA workqueue as well as much > simpler error handling. Please consider folding the patch below into this patch. Thanks, Bart. [PATCH] IB/srpt: Fix a recently introduced kernel crash BUG: unable to handle kernel paging request at 0000000100000198 IP: [] __lock_acquire+0xa2/0x560 Call Trace: [] lock_acquire+0x62/0x80 [] _raw_spin_lock_irqsave+0x43/0x60 [] srpt_rdma_read_done+0x57/0x120 [ib_srpt] [] __ib_process_cq+0x43/0xc0 [ib_core] [] ib_cq_poll_work+0x25/0x70 [ib_core] [] process_one_work+0x1bd/0x460 [] worker_thread+0x118/0x420 [] kthread+0xe4/0x100 [] ret_from_fork+0x3f/0x70 --- drivers/infiniband/ulp/srpt/ib_srpt.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index 8068aff..3daab39 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -1395,7 +1395,7 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) { struct srpt_rdma_ch *ch = cq->cq_context; struct srpt_send_ioctx *ioctx = - container_of(wc->wr_cqe, struct srpt_send_ioctx, ioctx.cqe); + container_of(wc->wr_cqe, struct srpt_send_ioctx, rdma_cqe); WARN_ON(ioctx->n_rdma <= 0); atomic_add(ioctx->n_rdma, &ch->sq_wr_avail); @@ -1418,7 +1418,7 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) static void srpt_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc) { struct srpt_send_ioctx *ioctx = - container_of(wc->wr_cqe, struct srpt_send_ioctx, ioctx.cqe); + container_of(wc->wr_cqe, struct srpt_send_ioctx, rdma_cqe); if (unlikely(wc->status != IB_WC_SUCCESS)) { pr_info("RDMA_WRITE for ioctx 0x%p failed with status %d\n", -- 2.1.4