From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 389C1C433FE for ; Wed, 2 Nov 2022 11:07:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hI/DV1pQ6hLbF98vuNR3/QrKwhHUTUgxIo8VmCqxg5c=; b=aQil4m5L8UdoSH AeSXkbKlktbslT1v3eNfgLL07bfOxabwudg+pouAR8GH/abgG4iUAVzpN4gvtJMHRLKyq57G0/6Uk LBR2X961ZXQUvirzSdWIxACisq+zieIiAHJ/XnCJnjT+mSRvlioY/y826X23Oa4rRFEgjjJUjnf5l hLckf3YMj9PngnW7bVANBbv1JnCuCNP8dIXSwuihFZIx4ltbdbdAXKgPq1rxa1iPrWTSETpIGLdJk EoT9Br8pN/3S86iRt9XVnDvV093zNYcPmu/q13aAl2ag9tuDU90ukUdDoQ7BY8TXCgAFly3m15Z/r o0kv+833slC4DU5iWQ6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqBZM-00AfqE-HB; Wed, 02 Nov 2022 11:06:04 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqBZJ-00Afpa-92 for linux-arm-kernel@lists.infradead.org; Wed, 02 Nov 2022 11:06:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A031F617CE; Wed, 2 Nov 2022 11:06:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BDCBCC433D6; Wed, 2 Nov 2022 11:05:57 +0000 (UTC) Date: Wed, 2 Nov 2022 11:05:54 +0000 From: Catalin Marinas To: Isaac Manjarres Cc: Christoph Hellwig , Greg Kroah-Hartman , Linus Torvalds , Arnd Bergmann , Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 2/2] treewide: Add the __GFP_PACKED flag to several non-DMA kmalloc() allocations Message-ID: References: <20221030084718.GC5278@lst.de> <20221030091349.GA5600@lst.de> <20221101105919.GA13872@lst.de> <20221101172416.GB20381@lst.de> <20221101173940.GA20821@lst.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221102_040601_418097_2FDAC3D2 X-CRM114-Status: GOOD ( 40.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Nov 01, 2022 at 12:10:51PM -0700, Isaac Manjarres wrote: > On Tue, Nov 01, 2022 at 06:39:40PM +0100, Christoph Hellwig wrote: > > On Tue, Nov 01, 2022 at 05:32:14PM +0000, Catalin Marinas wrote: > > > There's also the case of low-end phones with all RAM below 4GB and arm64 > > > doesn't allocate the swiotlb. Not sure those vendors would go with a > > > recent kernel anyway. > > > > > > So the need for swiotlb now changes from 32-bit DMA to any DMA > > > (non-coherent but we can't tell upfront when booting, devices may be > > > initialised pretty late). > > Not only low-end phones, but there are other form-factors that can fall > into this category and are also memory constrained (e.g. wearable > devices), so the memory headroom impact from enabling SWIOTLB might be > non-negligible for all of these devices. I also think it's feasible for > those devices to use recent kernels. Another option I had in mind is to disable this bouncing if there's no swiotlb buffer, so kmalloc() will return ARCH_DMA_MINALIGN (or the typically lower cache_line_size()) aligned objects. That's at least until we find a lighter way to do bouncing. Those devices would work as before. > > Yes. The other option would be to use the dma coherent pool for the > > bouncing, which must be present on non-coherent systems anyway. But > > it would require us to write a new set of bounce buffering routines. > > I think in addition to having to write new bounce buffering routines, > this approach still suffers the same problem as SWIOTLB, which is that > the memory for SWIOTLB and/or the dma coherent pool is not reclaimable, > even when it is not used. The dma coherent pool at least it has the advantage that its size can be increased at run-time and we can start with a small one. Not decreased though, but if really needed I guess it can be added. We'd also skip some cache maintenance here since the coherent pool is mapped as non-cacheable already. But to Christoph's point, it does require some reworking of the current bouncing code. > There's not enough context in the DMA mapping routines to know if we need > an atomic allocation, so if we used kmalloc(), instead of SWIOTLB, to > dynamically allocate memory, it would always have to use GFP_ATOMIC. I've seen the expression below in a couple of places in the kernel, though IIUC in_atomic() doesn't always detect atomic contexts: gfpflags = (in_atomic() || irqs_disabled()) ? GFP_ATOMIC : GFP_KERNEL; > But what about having a pool that has a small amount of memory and is > composed of several objects that can be used for small DMA transfers? > If the amount of memory in the pool starts falling below a certain > threshold, there can be a worker thread--so that we don't have to use > GFP_ATOMIC--that can add more memory to the pool? If the rate of allocation is high, it may end up calling a slab allocator directly with GFP_ATOMIC. The main downside of any memory pool is identifying the original pool in dma_unmap_*(). We have a simple is_swiotlb_buffer() check looking just at the bounce buffer boundaries. For the coherent pool we have the more complex dma_free_from_pool(). With a kmem_cache-based allocator (whether it's behind a mempool or not), we'd need something like virt_to_cache() and checking whether it is from our DMA cache. I'm not a big fan of digging into the slab internals for this. An alternative could be some xarray to remember the bounced dma_addr. Anyway, I propose that we try the swiotlb first and look at optimising it from there, initially using the dma coherent pool. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel