From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90AC7C433F5 for ; Fri, 17 Sep 2021 12:25:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5809861164 for ; Fri, 17 Sep 2021 12:25:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5809861164 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 78B836EC3E; Fri, 17 Sep 2021 12:25:25 +0000 (UTC) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by gabe.freedesktop.org (Postfix) with ESMTPS id 46C006EC3A for ; Fri, 17 Sep 2021 12:25:24 +0000 (UTC) Received: by mail-wr1-x42d.google.com with SMTP id w29so14874717wra.8 for ; Fri, 17 Sep 2021 05:25:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=B4xbw9gEcqky1hkIaKGJsZCnwDYURbkYMy8wOSBcFZM=; b=DBbT+Z73wXFx7Ix5ddCqc9nQajcrsU49nmmjtksdzvj4VIUhu9a12X6mxydVGcyr31 hkbTnOZYoHZBblsPGhC4cqiN5vTy2EZ/ap7N566BgNdEzkAG+xpa6VwhSk6rstGS1sPD 4IZWgIghyDL60XFxmzUrIJNR7pW0cc0ak4VI0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=B4xbw9gEcqky1hkIaKGJsZCnwDYURbkYMy8wOSBcFZM=; b=rAQkLdE76/o3EQd7SJxlQ6GzaK/RM4ycoJwXSKPkpoiU56RZ6nPbUyHSp9Gbt9i+va 5bHwxa4b691wifLenrUxrxL0kvsAOUDAKginL7NtM4n4nT3yiFfP4syo8NaF1Yd2IEGN ALiouVuSLe+ksQU8CXudjVRrqTfj8/Gyir6nWD3kzpxswxWBQw5zoJ5bf2VcR7zONZoN 7mgzDhMK45rj/07w+27H8Rlwi23lbjtfr3weu7F2hEVdv3wXPvql8GNEWRkPhKMFtD86 ldRMGsFPv0pa736K2om3n254vxTuOX0uZBO9QlgiV1Tqbwy27V8X4ysulcAHL+NuyIEJ TBYA== X-Gm-Message-State: AOAM531oeobNmCsep6nvUJp9xYTIJWit1mD7Pjg4r/4+oCSynTrv5QnM aCVsFezkH0Ay1NsWOKX7TT76Pw== X-Google-Smtp-Source: ABdhPJyMTcWyIQ8nzIGUt+EPkzWzvWClhn2Q1W8KmqNiWLuXSRA7doL/03EbIbn3t2AAXB96e7Moyg== X-Received: by 2002:adf:f084:: with SMTP id n4mr12033899wro.362.1631881522631; Fri, 17 Sep 2021 05:25:22 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id m18sm6529557wrn.85.2021.09.17.05.25.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Sep 2021 05:25:21 -0700 (PDT) Date: Fri, 17 Sep 2021 14:25:19 +0200 From: Daniel Vetter To: Oded Gabbay Cc: Jason Gunthorpe , "Linux-Kernel@Vger. Kernel. Org" , Greg Kroah-Hartman , Christian =?iso-8859-1?Q?K=F6nig?= , Gal Pressman , Yossi Leybovich , Maling list - DRI developers , linux-rdma , Linux Media Mailing List , Doug Ledford , Dave Airlie , Alex Deucher , Leon Romanovsky , Christoph Hellwig , amd-gfx list , "moderated list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [PATCH v6 0/2] Add p2p via dmabuf to habanalabs Message-ID: Mail-Followup-To: Oded Gabbay , Jason Gunthorpe , "Linux-Kernel@Vger. Kernel. Org" , Greg Kroah-Hartman , Christian =?iso-8859-1?Q?K=F6nig?= , Gal Pressman , Yossi Leybovich , Maling list - DRI developers , linux-rdma , Linux Media Mailing List , Doug Ledford , Dave Airlie , Alex Deucher , Leon Romanovsky , Christoph Hellwig , amd-gfx list , "moderated list:DMA BUFFER SHARING FRAMEWORK" References: <20210912165309.98695-1-ogabbay@kernel.org> <20210914161218.GF3544071@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: Linux phenom 5.10.0-8-amd64 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Thu, Sep 16, 2021 at 03:44:25PM +0300, Oded Gabbay wrote: > On Thu, Sep 16, 2021 at 3:31 PM Daniel Vetter wrote: > > > > On Wed, Sep 15, 2021 at 10:45:36AM +0300, Oded Gabbay wrote: > > > On Tue, Sep 14, 2021 at 7:12 PM Jason Gunthorpe wrote: > > > > > > > > On Tue, Sep 14, 2021 at 04:18:31PM +0200, Daniel Vetter wrote: > > > > > On Sun, Sep 12, 2021 at 07:53:07PM +0300, Oded Gabbay wrote: > > > > > > Hi, > > > > > > Re-sending this patch-set following the release of our user-space TPC > > > > > > compiler and runtime library. > > > > > > > > > > > > I would appreciate a review on this. > > > > > > > > > > I think the big open we have is the entire revoke discussions. Having the > > > > > option to let dma-buf hang around which map to random local memory ranges, > > > > > without clear ownership link and a way to kill it sounds bad to me. > > > > > > > > > > I think there's a few options: > > > > > - We require revoke support. But I've heard rdma really doesn't like that, > > > > > I guess because taking out an MR while holding the dma_resv_lock would > > > > > be an inversion, so can't be done. Jason, can you recap what exactly the > > > > > hold-up was again that makes this a no-go? > > > > > > > > RDMA HW can't do revoke. > > > > Like why? I'm assuming when the final open handle or whatever for that MR > > is closed, you do clean up everything? Or does that MR still stick around > > forever too? > > > > > > So we have to exclude almost all the HW and several interesting use > > > > cases to enable a revoke operation. > > > > > > > > > - For non-revokable things like these dma-buf we'd keep a drm_master > > > > > reference around. This would prevent the next open to acquire > > > > > ownership rights, which at least prevents all the nasty potential > > > > > problems. > > > > > > > > This is what I generally would expect, the DMABUF FD and its DMA > > > > memory just floats about until the unrevokable user releases it, which > > > > happens when the FD that is driving the import eventually gets closed. > > > This is exactly what we are doing in the driver. We make sure > > > everything is valid until the unrevokable user releases it and that > > > happens only when the dmabuf fd gets closed. > > > And the user can't close it's fd of the device until he performs the > > > above, so there is no leakage between users. > > > > Maybe I got the device security model all wrong, but I thought Guadi is > > single user, and the only thing it protects is the system against the > > Gaudi device trhough iommu/device gart. So roughly the following can > > happen: > > > > 1. User A opens gaudi device, sets up dma-buf export > > > > 2. User A registers that with RDMA, or anything else that doesn't support > > revoke. > > > > 3. User A closes gaudi device > This can not happen without User A closing the FD of the dma-buf it exported. > We prevent User A from closing the device because when it exported the > dma-buf, the driver's code took a refcnt of the user's private > structure. You can see that in export_dmabuf_common() in the 2nd > patch. There is a call there to hl_ctx_get. > So even if User A calls close(device_fd), the driver won't let any > other user open the device until User A closes the fd of the dma-buf > object. > > Moreover, once User A will close the dma-buf fd and the device is > released, the driver will scrub the device memory (this is optional > for systems who care about security). > > And AFAIK, User A can't close the dma-buf fd once it registered it > with RDMA, without doing unregister. > This can be seen in ib_umem_dmabuf_get() which calls dma_buf_get() > which does fget(fd) Yeah that's essentially what I was looking for. This is defacto hand-rolling the drm_master owner tracking stuff. As long as we have something like this in place it should be fine I think. -Daniel > > 4. User B opens gaudi device, assumes that it has full control over the > > device and uploads some secrets, which happen to end up in the dma-buf > > region user A set up > > > > 5. User B extracts secrets. > > > > > > I still don't think any of the complexity is needed, pinnable memory > > > > is a thing in Linux, just account for it in mlocked and that is > > > > enough. > > > > It's not mlocked memory, it's mlocked memory and I can exfiltrate it. > > Mlock is fine, exfiltration not so much. It's mlock, but a global pool and > > if you didn't munlock then the next mlock from a completely different user > > will alias with your stuff. > > > > Or is there something that prevents that? Oded at least explain that gaudi > > works like a gpu from 20 years ago, single user, no security at all within > > the device. > > -Daniel > > -- > > Daniel Vetter > > Software Engineer, Intel Corporation > > http://blog.ffwll.ch -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch