From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7B4DC4338F for ; Tue, 27 Jul 2021 12:45:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B392E61A4F for ; Tue, 27 Jul 2021 12:45:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231975AbhG0MpC (ORCPT ); Tue, 27 Jul 2021 08:45:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231840AbhG0MpB (ORCPT ); Tue, 27 Jul 2021 08:45:01 -0400 Received: from mail-oi1-x231.google.com (mail-oi1-x231.google.com [IPv6:2607:f8b0:4864:20::231]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C102C061757 for ; Tue, 27 Jul 2021 05:45:02 -0700 (PDT) Received: by mail-oi1-x231.google.com with SMTP id w6so14923763oiv.11 for ; Tue, 27 Jul 2021 05:45:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=i5Nt6NGzbA51toYcT/VTgo7QuL6cA4aE1IVXhXN32jc=; b=XZCIkGC5Sj/YjxEjhp7DbdT4aDGKQ8YqwwzxdrgXM/yvMuPkkg/DNllOpQQMp+6uv5 WyQ7eXvvqPcdR0dlDdR2zngrb4c6kSw55gD13RJUu/I3Of/ZwafM6SIz4WImOUixJnoT PRxKPIhiOmquQb1AnTRQFwhwGWcKq1AQIKRi5mdPnG3XYMSLlyXKCjTtIKtQIxlxkPfw DQAL/+Q3Hlg2/QycPl2+fPgJmmIqVhaPRI2JkHolVrFvkCGeDebzifRTwNnrbEaJZR33 Oe0bFKKrzFxoMG/zYgq/2OE8u6z6GnZU8kr34aWl8NYtH3lLY1/xgxARei7jv9JMiN10 XJxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=i5Nt6NGzbA51toYcT/VTgo7QuL6cA4aE1IVXhXN32jc=; b=ElT5Smr545uHZXSZBgplGa7MdoDnziC72KqY7+uOKMwiixow7rw3imBxUTTAP+GSXk FxYyHhrp0lnLKxKsTAzb70Vlq0FHwZkbWWLuCNuZhLEB18oyWG+kM+XOoxAyv1Irb8S7 6zqnSwO8pno+mkeQ2NffNJEy5C6fLecqmCqcmWJnCEdGlibqmUmfsdOZdmyJFyu/1bzt FGUEs9Jh/67WbVzPy5SURoCbrmRI7kxWo0K2TM97B839hW8onCpbqGLkwD/IlWBM1VP3 Uwvu1D3RFb98ZvcP57dNHpBLAheT4+Eo95TeuxPBD4i8OM7d5WKwo6v5K/ivt18iqG+H HnTA== X-Gm-Message-State: AOAM5320tN0XYDmaA62yU5lkl8gLjIKsyK/Gz/ap6bk69/oYO0PMtEeW FrqEm19MHv98ya9fd5d0ezQok4pUGQ8= X-Google-Smtp-Source: ABdhPJzWrzKdXqWbr3X2FrndVl3e4iVSuyxII3o9i9nPSBbwpGnmvBgbf6ADbur5wmy2pPHB71i2Mg== X-Received: by 2002:aca:d404:: with SMTP id l4mr14424423oig.21.1627389901387; Tue, 27 Jul 2021 05:45:01 -0700 (PDT) Received: from ?IPv6:2603:8081:140c:1a00:79a:6fd8:e907:f2b3? (2603-8081-140c-1a00-079a-6fd8-e907-f2b3.res6.spectrum.com. [2603:8081:140c:1a00:79a:6fd8:e907:f2b3]) by smtp.gmail.com with ESMTPSA id r5sm492786oti.5.2021.07.27.05.45.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 27 Jul 2021 05:45:00 -0700 (PDT) Subject: Re: [PATCH for-next] RDMA/rxe: Let rdma-core manage PDs To: Leon Romanovsky Cc: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org References: <20210726215815.17056-1-rpearsonhpe@gmail.com> From: Bob Pearson Message-ID: Date: Tue, 27 Jul 2021 07:44:52 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 7/27/21 7:02 AM, Leon Romanovsky wrote: > On Tue, Jul 27, 2021 at 06:31:59AM -0500, Bob Pearson wrote: >> On 7/27/21 6:30 AM, Leon Romanovsky wrote: >>> On Mon, Jul 26, 2021 at 04:58:16PM -0500, Bob Pearson wrote: >>>> Currently several rxe objects hold references to PDs which are ref- >>>> counted. This replicates work already done by RDMA core which takes >>>> references to PDs in the ib objects which are contained in the rxe >>>> objects. This patch removes struct rxe_pd from rxe objects and removes >>>> reference counting for PDs except for PD alloc and PD dealloc. It also >>>> adds inline extractor routines which return PDs from the PDs in the >>>> ib objects. The names of these are made consistent including a rxe_ >>>> prefix. >>>> >>>> Signed-off-by: Bob Pearson >>>> --- >>>> drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++-- >>>> drivers/infiniband/sw/rxe/rxe_loc.h | 4 ++-- >>>> drivers/infiniband/sw/rxe/rxe_mr.c | 8 +++---- >>>> drivers/infiniband/sw/rxe/rxe_mw.c | 31 +++++++++++---------------- >>>> drivers/infiniband/sw/rxe/rxe_qp.c | 9 +------- >>>> drivers/infiniband/sw/rxe/rxe_req.c | 2 +- >>>> drivers/infiniband/sw/rxe/rxe_resp.c | 4 ++-- >>>> drivers/infiniband/sw/rxe/rxe_verbs.c | 26 ++++++---------------- >>>> drivers/infiniband/sw/rxe/rxe_verbs.h | 24 +++++++++++++++------ >>>> 9 files changed, 48 insertions(+), 64 deletions(-) >>> >>> Last time when I looked on it, I came to conclusion that all RXE >>> references can be dropped. >>> >>> Thanks >>> >> This is a step in that direction. There are more coming. > > Glad to hear, thank you for your work. > >> Regards, >> >> Bob The other ones I can immediately get rid of are AHs, CQs and SRQs. The ones I think may require ref counting are MRs, MWs, QPs, and XRCSRQs. Each of these get looked up from information in RoCE packets from rkeys, qpns, and srq_nums, For reliable transports on slow networks this can require that these objects hang around for a while and the user has no visibility to this unless there is a completion event waiting but not for e.g. reads, writes or atomics on the target side. There can be races between destroying these objects and messages completing causing kernel oops. Do you know another way to address these cases? Bob