From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9CF2C433DF for ; Sun, 23 Aug 2020 22:54:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BFACD206BE for ; Sun, 23 Aug 2020 22:54:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="MJ9KBeu+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726631AbgHWWy4 (ORCPT ); Sun, 23 Aug 2020 18:54:56 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:13539 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726057AbgHWWyy (ORCPT ); Sun, 23 Aug 2020 18:54:54 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Sun, 23 Aug 2020 15:53:52 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Sun, 23 Aug 2020 15:54:53 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Sun, 23 Aug 2020 15:54:53 -0700 Received: from [10.2.52.145] (10.124.1.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Sun, 23 Aug 2020 22:54:42 +0000 Subject: Re: [RFC] Experimental DMA-BUF Device Heaps To: Laurent Pinchart CC: Ezequiel Garcia , Brian Starkey , , , Sumit Semwal , "Andrew F . Davis" , Benjamin Gaignard , Liam Mark , Laura Abbott , John Stultz , Daniel Stone , Nicolas Dufresne , Robert Beckett , Tomasz Figa , , References: <20200816172246.69146-1-ezequiel@collabora.com> <20200817151813.wet5faqg4fzlfbsh@DESKTOP-E1NTVVP.localdomain> <746a0bb75bd8388a30b53a5ddc56fb24aea308a8.camel@collabora.com> <4f987e06-7af1-671b-836a-10d3e9623547@nvidia.com> <20200823204637.GF6002@pendragon.ideasonboard.com> From: James Jones X-Nvconfidentiality: public Message-ID: <5a03e7d8-15cf-1323-31c1-74ae78f4f9c6@nvidia.com> Date: Sun, 23 Aug 2020 15:53:50 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20200823204637.GF6002@pendragon.ideasonboard.com> X-Originating-IP: [10.124.1.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1598223232; bh=18BF5ulZDZxLpxEQYQspgFSodbV0L1zev1sX0RHs0xo=; h=X-PGP-Universal:Subject:To:CC:References:From:X-Nvconfidentiality: Message-ID:Date:User-Agent:MIME-Version:In-Reply-To: X-Originating-IP:X-ClientProxiedBy:Content-Type:Content-Language: Content-Transfer-Encoding; b=MJ9KBeu+kyvrG2C9Rzk8U/ELt/ft8+Rt0IxTe1dH7faROiRCwD+f0zVYqe8Fl/GrO YLfUeebv8mltWdbdjosyaahodibvy26R7M0rKyGjGOoDPtVCk/GXpoa7S3Mc5sZRLX zNnnaKN45rNtYa7dxnT6IxlgcviUuXD4exM4pJoOqeRWn/vKgBGIK+K5D8EnSQa3zg 8D1arD91FZfV1cK95gh0v70fsZKSOGIdwjsUqVvWwAWBAvgOYVTpm6dW4dK4xchRWL DC8cERRrZop7hBO38ns5jkvf/vFjW7SZsNyiMqqMzUm+Fr1oJUVDq29F3fyi28+2LS Nj04UD/6UuFkw== Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org On 8/23/20 1:46 PM, Laurent Pinchart wrote: > Hi James, > > On Sun, Aug 23, 2020 at 01:04:43PM -0700, James Jones wrote: >> On 8/20/20 1:15 AM, Ezequiel Garcia wrote: >>> On Mon, 2020-08-17 at 20:49 -0700, James Jones wrote: >>>> On 8/17/20 8:18 AM, Brian Starkey wrote: >>>>> On Sun, Aug 16, 2020 at 02:22:46PM -0300, Ezequiel Garcia wrote: >>>>>> This heap is basically a wrapper around DMA-API dma_alloc_attrs, >>>>>> which will allocate memory suitable for the given device. >>>>>> >>>>>> The implementation is mostly a port of the Contiguous Videobuf2 >>>>>> memory allocator (see videobuf2/videobuf2-dma-contig.c) >>>>>> over to the DMA-BUF Heap interface. >>>>>> >>>>>> The intention of this allocator is to provide applications >>>>>> with a more system-agnostic API: the only thing the application >>>>>> needs to know is which device to get the buffer for. >>>>>> >>>>>> Whether the buffer is backed by CMA, IOMMU or a DMA Pool >>>>>> is unknown to the application. >>>>>> >>>>>> I'm not really expecting this patch to be correct or even >>>>>> a good idea, but just submitting it to start a discussion on DMA-BUF >>>>>> heap discovery and negotiation. >>>>>> >>>>> >>>>> My initial reaction is that I thought dmabuf heaps are meant for use >>>>> to allocate buffers for sharing across devices, which doesn't fit very >>>>> well with having per-device heaps. >>>>> >>>>> For single-device allocations, would using the buffer allocation >>>>> functionality of that device's native API be better in most >>>>> cases? (Some other possibly relevant discussion at [1]) >>>>> >>>>> I can see that this can save some boilerplate for devices that want >>>>> to expose private chunks of memory, but might it also lead to 100 >>>>> aliases for the system's generic coherent memory pool? >>>>> >>>>> I wonder if a set of helpers to allow devices to expose whatever they >>>>> want with minimal effort would be better. >>>> >>>> I'm rather interested on where this goes, as I was toying with using >>>> some sort of heap ID as a basis for a "device-local" constraint in the >>>> memory constraints proposals Simon and I will be discussing at XDC this >>>> year. It would be rather elegant if there was one type of heap ID used >>>> universally throughout the kernel that could provide a unique handle for >>>> the shared system memory heap(s), as well as accelerator-local heaps on >>>> fancy NICs, GPUs, NN accelerators, capture devices, etc. so apps could >>>> negotiate a location among themselves. This patch seems to be a step >>>> towards that in a way, but I agree it would be counterproductive if a >>>> bunch of devices that were using the same underlying system memory ended >>>> up each getting their own heap ID just because they used some SW >>>> framework that worked that way. >>>> >>>> Would appreciate it if you could send along a pointer to your BoF if it >>>> happens! >>> >>> Here is it: >>> >>> https://linuxplumbersconf.org/event/7/contributions/818/ >>> >>> It would be great to see you there and discuss this, >>> given I was hoping we could talk about how to meet a >>> userspace allocator library expectations as well. >> >> Thanks! I hadn't registered for LPC and it looks like it's sold out, >> but I'll try to watch the live stream. >> >> This is very interesting, in that it looks like we're both trying to >> solve roughly the same set of problems but approaching it from different >> angles. From what I gather, your approach is that a "heap" encompasses >> all the allocation constraints a device may have. >> >> The approach Simon Ser and I are tossing around so far is somewhat >> different, but may potentially leverage dma-buf heaps a bit as well. >> >> Our approach looks more like what I described at XDC a few years ago, >> where memory constraints for a given device's usage of an image are >> exposed up to applications, which can then somehow perform boolean >> intersection/union operations on them to arrive at a common set of >> constraints that describe something compatible with all the devices & >> usages desired (or fail to do so, and fall back to copying things around >> presumably). I believe this is more flexible than your initial proposal >> in that devices often support multiple usages (E.g., different formats, >> different proprietary layouts represented by format modifiers, etc.), >> and it avoids adding a combinatorial number of heaps to manage that. >> >> In my view, heaps are more like blobs of memory that can be allocated >> from in various different ways to satisfy constraints. I realize heaps >> mean something specific in the dma-buf heap design (specifically, >> something closer to an association between an "allocation mechanism" and >> "physical memory"), but I hope we don't have massive heap/allocator >> mechanism proliferation due to constraints alone. Perhaps some >> constraints, such as contiguous memory or device-local memory, are >> properly expressed as a specific heap, but consider the proliferation >> implied by even that simple pair of examples: How do you express >> contiguous device-local memory? Do you need to spawn two heaps on the >> underlying device-local memory, one for contiguous allocations and one >> for non-contiguous allocations? Seems excessive. >> >> Of course, our approach also has downsides and is still being worked on. >> For example, it works best in an ideal world where all the allocators >> available understand all the constraints that exist. > > Shouldn't allocators be decoupled of constraints ? In my imagination I > see devices exposing constraints, and allocators exposing parameters, > with a userspace library to reconcile the constraints and produce > allocator parameters from them. Perhaps another level of abstraction would help. I'll have to think about that. However, as far as I can tell, it wouldn't remove the need to communicate a lot of constraints from multiple engines/devices/etc. to the allocator (likely a single allocator. I'd be interested to know if anyone has a design that effectively uses multiple allocators to satisfy a single allocation request, but I haven't come up with a good one) somehow. Either the constraints are directly used as the parameters, or there's a translation/second level of abstraction, but either way much of the information needs to make it to the allocator, or represent the need to use a particular allocator. Simple things like pitch and offset alignment can be done without help from a kernel-level allocator, but others such as cache coherency, physical memory bank placement, or device-local memory will need to make it all the way down to the kernel some how I believe. Thanks, -James >> Dealing with a >> reality where there are probably a handful of allocators, another >> handful of userspace libraries and APIs, and still more applications >> trying to make use of all this is one of the larger remaining challenges >> of the design. >> >> We'll present our work at XDC 2020. Hope you can check that out as well! >> >>>>> 1. https://lore.kernel.org/dri-devel/57062477-30e7-a3de-6723-a50d03a402c4@kapsi.fi/ >>>>> >>>>>> Given Plumbers is just a couple weeks from now, I've submitted >>>>>> a BoF proposal to discuss this, as perhaps it would make >>>>>> sense to discuss this live? >>>>>> >>>>>> Not-signed-off-by: Ezequiel Garcia >