From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FF63C07E97 for ; Wed, 29 Nov 2023 10:14:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231354AbjK2KOd (ORCPT ); Wed, 29 Nov 2023 05:14:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229513AbjK2KOc (ORCPT ); Wed, 29 Nov 2023 05:14:32 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58997E1 for ; Wed, 29 Nov 2023 02:14:38 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 64A58C433C7; Wed, 29 Nov 2023 10:14:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701252878; bh=x3EsyBCyTPvX/m9MXNOcgKDKbuv2rGH9gFcXsxJcg8k=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=j4RfxldFxov7IXn9fDbe0YkCvsaqJnB0QlWgOFoalPjzF4RBpo5NQzn+8RkYId4dp x7GwjoCEoSII52tOLVXpqg+DiryX5LDdjIzBJDnc/NcD5R1BSFZaHNdXnniUWmkla0 yjH+maTWEmKOO2wSNuQdyxhlCNiktVdQ2ngiSgMAPJa47O2T8VXmsl5+2+DIgPWFWb hH4hZJWJYZDUaGer5aN0l0jOS56M6zmnKQAXjcfE3kZoLQGt3zPIGC1U/Yqs77qwb5 NOdt5O2SzblHMI2YE5a8V1ZIKKL6OxlsEfijb1BnVcV00hayJTmVk4EopIOkZ+S5sR c1SRDb29cPA1A== Date: Wed, 29 Nov 2023 12:14:33 +0200 From: Leon Romanovsky To: Shifeng Li Cc: "Ismail, Mustafa" , "Saleem, Shiraz" , "jgg@ziepe.ca" , "linux-rdma@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Ding, Hui" Subject: Re: [PATCH] RDMA/irdma: Avoid free the non-cqp_request scratch Message-ID: <20231129101433.GC6535@unreal> References: <20231120083122.78532-1-lishifeng1992@126.com> <15f80347-e9e9-49f9-bcab-784974856332@126.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15f80347-e9e9-49f9-bcab-784974856332@126.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Nov 24, 2023 at 10:28:58PM +0800, Shifeng Li wrote: > On 2023/11/22 1:25, Ismail, Mustafa wrote: > > > -----Original Message----- > > > From: Shifeng Li > > > Sent: Monday, November 20, 2023 2:31 AM > > > To: Ismail, Mustafa > > > Cc: Saleem, Shiraz ; jgg@ziepe.ca; > > > leon@kernel.org; linux-rdma@vger.kernel.org; linux-kernel@vger.kernel.org; > > > Ding, Hui ; Shifeng Li > > > Subject: [PATCH] RDMA/irdma: Avoid free the non-cqp_request scratch > > > > > > When creating ceq_0 during probing irdma, cqp.sc_cqp will be sent as a > > > cqp_request to cqp->sc_cqp.sq_ring. If the request is pending when removing > > > the irdma driver or unplugging its aux device, cqp.sc_cqp will be dereferenced > > > as wrong struct in irdma_free_pending_cqp_request(). > > > > > > crash> bt 3669 > > > PID: 3669 TASK: ffff88aef892c000 CPU: 28 COMMAND: "kworker/28:0" > > > #0 [fffffe0000549e38] crash_nmi_callback at ffffffff810e3a34 > > > #1 [fffffe0000549e40] nmi_handle at ffffffff810788b2 > > > #2 [fffffe0000549ea0] default_do_nmi at ffffffff8107938f > > > #3 [fffffe0000549eb8] do_nmi at ffffffff81079582 > > > #4 [fffffe0000549ef0] end_repeat_nmi at ffffffff82e016b4 > > > [exception RIP: native_queued_spin_lock_slowpath+1291] > > > RIP: ffffffff8127e72b RSP: ffff88aa841ef778 RFLAGS: 00000046 > > > RAX: 0000000000000000 RBX: ffff88b01f849700 RCX: ffffffff8127e47e > > > RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff83857ec0 > > > RBP: ffff88afe3e4efc8 R8: ffffed15fc7c9dfa R9: ffffed15fc7c9dfa > > > R10: 0000000000000001 R11: ffffed15fc7c9df9 R12: 0000000000740000 > > > R13: ffff88b01f849708 R14: 0000000000000003 R15: ffffed1603f092e1 > > > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 > > > --- --- > > > #5 [ffff88aa841ef778] native_queued_spin_lock_slowpath at ffffffff8127e72b > > > #6 [ffff88aa841ef7b0] _raw_spin_lock_irqsave at ffffffff82c22aa4 > > > #7 [ffff88aa841ef7c8] __wake_up_common_lock at ffffffff81257363 > > > #8 [ffff88aa841ef888] irdma_free_pending_cqp_request at ffffffffa0ba12cc > > > [irdma] > > > #9 [ffff88aa841ef958] irdma_cleanup_pending_cqp_op at ffffffffa0ba1469 > > > [irdma] > > > #10 [ffff88aa841ef9c0] irdma_ctrl_deinit_hw at ffffffffa0b2989f [irdma] > > > #11 [ffff88aa841efa28] irdma_remove at ffffffffa0b252df [irdma] > > > #12 [ffff88aa841efae8] auxiliary_bus_remove at ffffffff8219afdb > > > #13 [ffff88aa841efb00] device_release_driver_internal at ffffffff821882e6 > > > #14 [ffff88aa841efb38] bus_remove_device at ffffffff82184278 > > > #15 [ffff88aa841efb88] device_del at ffffffff82179d23 > > > #16 [ffff88aa841efc48] ice_unplug_aux_dev at ffffffffa0eb1c14 [ice] > > > #17 [ffff88aa841efc68] ice_service_task at ffffffffa0d88201 [ice] > > > #18 [ffff88aa841efde8] process_one_work at ffffffff811c589a > > > #19 [ffff88aa841efe60] worker_thread at ffffffff811c71ff > > > #20 [ffff88aa841eff10] kthread at ffffffff811d87a0 > > > #21 [ffff88aa841eff50] ret_from_fork at ffffffff82e0022f > > > > > > Fixes: 44d9e52977a1 ("RDMA/irdma: Implement device initialization > > > definitions") > > > Signed-off-by: Shifeng Li > > > --- > > > drivers/infiniband/hw/irdma/utils.c | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/drivers/infiniband/hw/irdma/utils.c > > > b/drivers/infiniband/hw/irdma/utils.c > > > index 445e69e86409..222ec1f761a1 100644 > > > --- a/drivers/infiniband/hw/irdma/utils.c > > > +++ b/drivers/infiniband/hw/irdma/utils.c > > > @@ -541,7 +541,7 @@ void irdma_cleanup_pending_cqp_op(struct > > > irdma_pci_f *rf) > > > for (i = 0; i < pending_work; i++) { > > > cqp_request = (struct irdma_cqp_request *)(unsigned long) > > > cqp->scratch_array[wqe_idx]; > > > - if (cqp_request) > > > + if (cqp_request && cqp_request != (struct irdma_cqp_request > > > +*)&cqp->sc_cqp) > > > irdma_free_pending_cqp_request(cqp, cqp_request); > > > wqe_idx = (wqe_idx + 1) % IRDMA_RING_SIZE(cqp- > > > > sc_cqp.sq_ring); > > > } > > > -- > > > 2.25.1 > > > > Hi Li, > > > > Could you describe how you hit this issue? It seems like the probe would not have completed successfully if the create cceq request was still pending. > > > > A better fix might be to set the scratch to 0 in this case: In irdma_create_ceq(), pass 0 for scratch in call to irdma_sc_cceq_create(&iwceq->sc_ceq, 0), since it doesn't appear to be used in this case. Is it possible to test this fix? Thanks! > > 1. The firmware of net card or the hardware may enter into some kind of error state. And I observed > that the sq_ring of sc_cqp is full through crash. This means that all the 2048 cqp_requests were pending. > > crash> struct irdma_cqp.sc_cqp ffff88afe3e4eee8 > sc_cqp = { > ... > sq_ring = { > head = 91, (The queue head caught up with the queue tail.) > tail = 92, > size = 2048 > }, > ... > } > > 2. The issue is not reproducible. But I have tested the solution that pass 0 for scratch in > call to irdma_sc_cceq_create(&iwceq->sc_ceq, 0) in irdma_create_ceq(). It works well. Will you submit v2? Thanks > > Thanks! > >