From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7045FC4320E for ; Thu, 19 Aug 2021 23:06:07 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C80A61029 for ; Thu, 19 Aug 2021 23:06:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2C80A61029 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BDC0F6E90C; Thu, 19 Aug 2021 23:06:05 +0000 (UTC) Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by gabe.freedesktop.org (Postfix) with ESMTPS id 767146E90C for ; Thu, 19 Aug 2021 23:06:04 +0000 (UTC) Received: by mail-qk1-x730.google.com with SMTP id az7so9008685qkb.5 for ; Thu, 19 Aug 2021 16:06:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=lUX7epF7foRJPnJW8zAvGrpG9sbsCy+8eUVnLXUioaw=; b=k/GgjG+KGvsRHDTaR/VlVvRNrfg9uPX5JU23M6lnlFazEhBaLpFG2yLgIA5DjuqITc ZtITYZTtiD4QoA9YVBs+h0X1AVz3p2dPiahcKu4x64AcrhVQXiJIob4kXZsFDqfHLPEH MDAy0kMdT7vfNS453Pn3keQ8XOV6pBw+XrZ9JH3zS6LTaGidh2NOFMUyf6POnFxjJjMV dteg7esI3k4ovVAkSAOQhtzZ0/kyjPmh1I1TnfpWjWVKuXx7m8P9gtlyRFtfWMupn+rj +p4DvEIaviEiwP7Tg+vxG/hXWAG9QrvF2o7GcnaIaJ8USAjvXaOGmBneEWy8U7uYSyyp KRiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=lUX7epF7foRJPnJW8zAvGrpG9sbsCy+8eUVnLXUioaw=; b=ENPQ2MO724gu9S0sNxHCCp/l/xq42Jolv8uK0m81TY3bP2tKYM/+L9m3CoXXMoTvfe HHdz7Yc7thfJo2yWFXKvJk35KMbsCAv2i+9f3COsIuNSNqquMuCOL968tif3ChdfwYMO Eld2Dx+Zm+WwWLQTeL+qhzs63wSX71+OHhUG2NYmsble6xJ1Y/UX4+ITZw1+OR70ExRp +YpVTS/YYEgHnkhmDTkayYCQNjUzzDCxG4N5QeeSlCPqtpPpHcixrncxzVsbAZ+7PoP+ kp7AjBFcUzPz3gH+gWHfEjRWix7IWjdYe2F+L0oil2qxO1a85x2eIP0xC5kA/FnwAVrF Q5sw== X-Gm-Message-State: AOAM5311oyx6O0MP7F1Ks01me1O86VSOdMB5M3s4mLSCbkc1RK9Y+6/x +5tW9RSOK8cEzCfdSy0alqntrA== X-Google-Smtp-Source: ABdhPJy/ltQJWz3bEvwivJHVegVN2SJG5oJXKMzn032Cjp7qnG0C8JFFt9RanDCCQRXqWfZ4YPIPmA== X-Received: by 2002:a37:2753:: with SMTP id n80mr5973504qkn.223.1629414363407; Thu, 19 Aug 2021 16:06:03 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id n20sm2401213qkk.135.2021.08.19.16.06.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Aug 2021 16:06:02 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1mGr6o-001dOi-8m; Thu, 19 Aug 2021 20:06:02 -0300 Date: Thu, 19 Aug 2021 20:06:02 -0300 From: Jason Gunthorpe To: Daniel Vetter Cc: Gal Pressman , Sumit Semwal , Christian =?utf-8?B?S8O2bmln?= , Doug Ledford , "open list:DMA BUFFER SHARING FRAMEWORK" , dri-devel , Linux Kernel Mailing List , linux-rdma , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong , John Hubbard Subject: Re: [RFC] Make use of non-dynamic dmabuf in RDMA Message-ID: <20210819230602.GU543798@ziepe.ca> References: <20210818074352.29950-1-galpress@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Aug 18, 2021 at 11:34:51AM +0200, Daniel Vetter wrote: > On Wed, Aug 18, 2021 at 9:45 AM Gal Pressman wrote: > > > > Hey all, > > > > Currently, the RDMA subsystem can only work with dynamic dmabuf > > attachments, which requires the RDMA device to support on-demand-paging > > (ODP) which is not common on most devices (only supported by mlx5). > > > > While the dynamic requirement makes sense for certain GPUs, some devices > > (such as habanalabs) have device memory that is always "pinned" and do > > not need/use the move_notify operation. > > > > The motivation of this RFC is to use habanalabs as the dmabuf exporter, > > and EFA as the importer to allow for peer2peer access through libibverbs. > > > > This draft patch changes the dmabuf driver to differentiate between > > static/dynamic attachments by looking at the move_notify op instead of > > the importer_ops struct, and allowing the peer2peer flag to be enabled > > in case of a static exporter. > > > > Thanks > > > > Signed-off-by: Gal Pressman > > Given that habanalabs dma-buf support is very firmly in limbo (at > least it's not yet in linux-next or anywhere else) I think you want to > solve that problem first before we tackle the additional issue of > making p2p work without dynamic dma-buf. Without that it just doesn't > make a lot of sense really to talk about solutions here. I have been thinking about adding a dmabuf exporter to VFIO, for basically the same reason habana labs wants to do it. In that situation we'd want to see an approach similar to this as well to have a broad usability. The GPU drivers also want this for certain sophisticated scenarios with RDMA, the intree drivers just haven't quite got there yet. So, I think it is worthwhile to start thinking about this regardless of habana labs. Jason