From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44751C433EF for ; Tue, 12 Apr 2022 13:49:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356260AbiDLNwB (ORCPT ); Tue, 12 Apr 2022 09:52:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242810AbiDLNv7 (ORCPT ); Tue, 12 Apr 2022 09:51:59 -0400 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24CD24E392 for ; Tue, 12 Apr 2022 06:49:39 -0700 (PDT) Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1649771377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JF7pyaW0Ol89JibMy6y1N1DxObi47mpqU5PSCenmn6U=; b=JmvYE1VKn2N8SinVICVaNSzqovcqtga0182voUEl75J/mnctin3FH2N4auKAY9WoMqXWBv Laf8Wwy9gW4tyAb8adZ+2qEQsRT025hYPjwp+hS+lTHcD5if6lo2mvMr0dkUWi8NBa2gPZ Z5uIEokaqcRiTsbtpDFCYdt0lmfBL9o= Date: Tue, 12 Apr 2022 21:49:27 +0800 MIME-Version: 1.0 Subject: Re: [PATCH for-next] RDMA/rxe: Fix "Replace red-black trees by xarrays" To: Jason Gunthorpe Cc: "Pearson, Robert B" , "linux-rdma@vger.kernel.org" References: <20220410223939.3769-1-rpearsonhpe@gmail.com> <6296dc52-1298-6a52-a4fb-2c6fe04ab151@gmail.com> <20220411113836.GD2120790@nvidia.com> <20220411160217.GA4139526@nvidia.com> <4314cea9-b5c4-b0e6-60f3-b3a91abce505@linux.dev> <20220412133946.GH2120790@nvidia.com> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Yanjun Zhu In-Reply-To: <20220412133946.GH2120790@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org 在 2022/4/12 21:39, Jason Gunthorpe 写道: > On Tue, Apr 12, 2022 at 09:32:25PM +0800, Yanjun Zhu wrote: >> 在 2022/4/12 0:02, Jason Gunthorpe 写道: >>> On Mon, Apr 11, 2022 at 03:52:23PM +0000, Pearson, Robert B wrote: >>>>> Yes, you cannot use irq_save variants here. You have to know your calling context is non-atomic already and use the irq wrapper. >>>> Not sure why. If you call irqsave in a non-atomic context doesn't it >>>> behave the same as irq? I.e. flags = 0. xarray provides >>>> xa_lock_xxx() for people to use. Are you saying I have to call >>>> xa_alloc_cyclic_xxx() instead which is the same thing? >>> The xa_lock is a magic thing, when you call a __xa_*(.., GFP_KERNEL) >>> type function then it will unlock and relock the xa_lock internally. >>> >>> Thus you cannot wrapper an irqsave lock across the GFP_KERNEL call >>> sites because XA doesn't know the 'flags' to be able to properly >>> unlock it. >>> >>>> The problem is I've gotten gun shy about people calling into the >>>> verbs APIs in strange contexts. The rules don't seem to be written >>>> down. If I could be certain that no one ever is allowed to call a >>>> verbs API in an interrupt then this is correct but how do I know >>>> that? >>> rxe used GFP_KERNEL so it already has assumed it is a process context. >> Got it. >> >> __xa_alloc_cyclic(..., GFP_NOWAIT) will unlock xa lock internally. But it >> does not handle flags. So the irq is still disabled. Because GFP_NOWAIT is >> used in __xa_alloc_cyclic, it will not sleep. >> >> __xa_alloc_cyclic will lock xa lock and return to xa_unlock_irqrestore. >> >> So the following code should be OK? >> >> xa_lock_irqsave(&pool->xa, flags); >> err = __xa_alloc_cyclic(&pool->xa, &elem->index, elem, pool->limit, >> &pool->next, GFP_NOWAIT); >> xa_unlock_irqrestore(&pool->xa, flags); > No, use GFP_KERNEL an use the proper irq or bh version of > xa_alloc_cyclic. > > save is only required if you don't know the contex you are in, and > here we know we are in a process context. if we use xa_lock_irq/xa_unlock_irq or xa_lock_bh/xa_unlock_bh instead of xa_unlock_irqrestore, the warning as below will appear. This means that __rxe_add_to_pool disables softirq, but fpu_clone enables softirq. " Apr 12 16:24:53 kernel: softirqs last  enabled at (13086): [] fpu_clone+0xf6/0x570 Apr 12 16:24:53 kernel: softirqs last disabled at (13129): [] __rxe_add_to_pool+0x49/0xa0 [rdma_rxe] " As such, it is better to use xa_unlock_irqrestore + __xa_alloc(...,GFP_ATOMIC/GFP_NOWAIT). Zhu Yanjun > > Jason