From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5605AC4332F for ; Wed, 2 Nov 2022 20:51:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=D0pojtpEJ3OEvJWUw1OR49yuW9m3m5ClfyjgQExFgA8=; b=lZXjC8LztZtZjr /L1xiRnAEwLq2Gpx1jm+fxwyb8uyY4tXmFbZ72fv4p4kmZLN9xZMikliyFz+B7yRu2WGZcfKB5LiT vPbcNbdjgrELYaofBJ9i/5/phrgkBK5UHn8LWO0EZrknK32HFuwYi/1hIahV/BbbbgVW7RpUTs4P0 F0RaB+dThRyYT87BjgLZXcMhPMZ94aazZusAIhG0OQAZMyD/IGXn3CdW4h/tnKTGWpHwQzYr/rHbz 2ahqYdJJNK367O29E473X8Yphh+UdYM35MqtYXMsRETyugzofT4XF1+dhFqdiSOiCHtWtuiYbDObC 81n1DmflN4gXiPDU/jGA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqKh4-00ENWE-6U; Wed, 02 Nov 2022 20:50:38 +0000 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqKh0-00ENUM-B3 for linux-arm-kernel@lists.infradead.org; Wed, 02 Nov 2022 20:50:36 +0000 Received: by mail-pg1-x533.google.com with SMTP id r18so17231254pgr.12 for ; Wed, 02 Nov 2022 13:50:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=s2Fta52Ezq1Jihj5LiWc83ydrfKZa4wMtzdeCdVvZOg=; b=QoYqii6A1WIsfq91Qqfry1YALgjlg5cUP7EFuQ/pp6teI4iPF1pa4+loZATqih6lw6 esp1SUTcldVCRlo03PzKLwYGaDc/FSNT5W3TEo7wQBa4Fnu7o0g/k0TqnE8ctSSLFsYE PIKxc+5D284UajGqBV9sXA3JCt4D70LMtOC1QlSIUwx3Mk1hXPEdmpGzA4LMyDdULsvF mgrLM2lKwq9dcxnFLJ88fr3igVDpaQVJbQNslhITrLjwJngB9xs7WSNBLNQkNutaFPvR T7Em1tDvPvUseHZnYuDQrX1zCQ8+TboAF543WDJz/pnvDSCv8QPfgX0CQhc5xAHatIrA dXgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=s2Fta52Ezq1Jihj5LiWc83ydrfKZa4wMtzdeCdVvZOg=; b=c0eGDrRnLzphesA8dKv7bE42s/94qnrLrhvycawJnhPFIppcnxaxOYGI4+vidLPlq7 8ePDXpbaJ9rd6EzirWUN3bHIlbui82rMWu81YTJ2mPsobA19JvkbY+HSNWHF5sjPH7Rj PPr1b6pNXszQ6TxkDfm8pjTXxoMA1aJk9URHjk5uIuaO2xvZShZlo89Qvbn97WZZDgNu 76OwWMCbtREbXw1OETr99SuJQt+D+Eb1ahu52KwALViWGS1Q1uaYeRpIshBQ/yfzRSsU Gy28MeA9i/Iop8UkusZRLKUC+RSEMFMzzDvAPCuQfcIlBhhg5yAeTLygvvmjbC+GmXOM voWw== X-Gm-Message-State: ACrzQf3XbKoziiTBK8xXc/wsUSXQxXmcQegIF2qCwPPACHy7NDdzUjXb q/5jRM3PWWX1A8bHTJsRQ1X6FA== X-Google-Smtp-Source: AMsMyM4iq8sAfWgMMDo/qq5AIFNURO8NNeNSoKp22ASEgNRhINF26lyaAZtqJs6sJv3wS3jKBa0elw== X-Received: by 2002:a63:1765:0:b0:457:8091:1b6c with SMTP id 37-20020a631765000000b0045780911b6cmr22861825pgx.208.1667422229142; Wed, 02 Nov 2022 13:50:29 -0700 (PDT) Received: from google.com ([2620:15c:2d:3:c83f:bf46:7d3c:579f]) by smtp.gmail.com with ESMTPSA id q22-20020a170902bd9600b00185002f0c6csm8715308pls.134.2022.11.02.13.50.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 13:50:28 -0700 (PDT) Date: Wed, 2 Nov 2022 13:50:23 -0700 From: Isaac Manjarres To: Catalin Marinas Cc: Christoph Hellwig , Greg Kroah-Hartman , Linus Torvalds , Arnd Bergmann , Will Deacon , Marc Zyngier , Andrew Morton , Herbert Xu , Ard Biesheuvel , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH v2 2/2] treewide: Add the __GFP_PACKED flag to several non-DMA kmalloc() allocations Message-ID: References: <20221030091349.GA5600@lst.de> <20221101105919.GA13872@lst.de> <20221101172416.GB20381@lst.de> <20221101173940.GA20821@lst.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221102_135034_402302_BB014EB7 X-CRM114-Status: GOOD ( 51.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Nov 02, 2022 at 11:05:54AM +0000, Catalin Marinas wrote: > On Tue, Nov 01, 2022 at 12:10:51PM -0700, Isaac Manjarres wrote: > > On Tue, Nov 01, 2022 at 06:39:40PM +0100, Christoph Hellwig wrote: > > > On Tue, Nov 01, 2022 at 05:32:14PM +0000, Catalin Marinas wrote: > > > > There's also the case of low-end phones with all RAM below 4GB and arm64 > > > > doesn't allocate the swiotlb. Not sure those vendors would go with a > > > > recent kernel anyway. > > > > > > > > So the need for swiotlb now changes from 32-bit DMA to any DMA > > > > (non-coherent but we can't tell upfront when booting, devices may be > > > > initialised pretty late). > > > > Not only low-end phones, but there are other form-factors that can fall > > into this category and are also memory constrained (e.g. wearable > > devices), so the memory headroom impact from enabling SWIOTLB might be > > non-negligible for all of these devices. I also think it's feasible for > > those devices to use recent kernels. > > Another option I had in mind is to disable this bouncing if there's no > swiotlb buffer, so kmalloc() will return ARCH_DMA_MINALIGN (or the > typically lower cache_line_size()) aligned objects. That's at least > until we find a lighter way to do bouncing. Those devices would work as > before. The SWIOTLB buffer will not be allocated in cases with devices that have low amounts of RAM sitting entirely below 4 GB. Those devices though would still benefit greatly from kmalloc() using a smaller size for objects, so it would be unfortunate to not allow this based on the existence of the SWIOTLB buffer. > > > Yes. The other option would be to use the dma coherent pool for the > > > bouncing, which must be present on non-coherent systems anyway. But > > > it would require us to write a new set of bounce buffering routines. > > > > I think in addition to having to write new bounce buffering routines, > > this approach still suffers the same problem as SWIOTLB, which is that > > the memory for SWIOTLB and/or the dma coherent pool is not reclaimable, > > even when it is not used. > > The dma coherent pool at least it has the advantage that its size can be > increased at run-time and we can start with a small one. Not decreased > though, but if really needed I guess it can be added. > > We'd also skip some cache maintenance here since the coherent pool is > mapped as non-cacheable already. But to Christoph's point, it does > require some reworking of the current bouncing code. Right, I do think it's a good thing that dma coherent pool starts small and can grow. I don't think it would be too difficult to add logic to free the memory back. Perhaps using a shrinker might be sufficient to free back memory when the system is experiencing memory pressure, instead of relying on some threshold? > > I've seen the expression below in a couple of places in the kernel, > though IIUC in_atomic() doesn't always detect atomic contexts: > > gfpflags = (in_atomic() || irqs_disabled()) ? GFP_ATOMIC : GFP_KERNEL; > I'm not too sure about this; I was going more off of how the mapping callbacks in iommu/dma-iommu.c use the atomic variants of iommu_map. > > But what about having a pool that has a small amount of memory and is > > composed of several objects that can be used for small DMA transfers? > > If the amount of memory in the pool starts falling below a certain > > threshold, there can be a worker thread--so that we don't have to use > > GFP_ATOMIC--that can add more memory to the pool? > > If the rate of allocation is high, it may end up calling a slab > allocator directly with GFP_ATOMIC. > > The main downside of any memory pool is identifying the original pool in > dma_unmap_*(). We have a simple is_swiotlb_buffer() check looking just > at the bounce buffer boundaries. For the coherent pool we have the more > complex dma_free_from_pool(). > > With a kmem_cache-based allocator (whether it's behind a mempool or > not), we'd need something like virt_to_cache() and checking whether it > is from our DMA cache. I'm not a big fan of digging into the slab > internals for this. An alternative could be some xarray to remember the > bounced dma_addr. Right. I had actually thought of using something like what is in mm/dma-pool.c and the dma coherent pool, where the pool is backed by the page allocator, and the objects are of a fixed size (for ARM64 for example, it would be align(192, ARCH_DMA_MINALIGN) == 256, though it would be good to have a more generic way of calculating this). Then determining whether an object resides in the pool boils down to scanning the backing pages for the pool, which dma coherent pool does. > Anyway, I propose that we try the swiotlb first and look at optimising > it from there, initially using the dma coherent pool. Except for the freeing logic, which can be added if needed as you pointed out, and Christoph's point about reworking the bouncing code, dma coherent pool doesn't sound like a bad idea. --Isaac _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel