From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6CD80C52D73 for ; Wed, 7 Aug 2024 18:15:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3bb5l39Qsw04T9DXAdATLbo9YT00yFuZtMybZAG8wHs=; b=rrr449S2KGJxMX3ZoOxWfqyyIs EAip+GW+TJ1KAlEAs3vfmGNPxR/6qlxrt8/dukRH0IGXF+ebPL90PJNVZEtlKT1b3NIwGSk4aKNIV nzBCBtUxnePxM4Z/eQhsWIF2L1RMD6JlrI4NPsnV/kopwJBVSGm49GAcWrDlD5eD+UfLTZAlBSXFN AZ3wKGY++XJnCFWsUoxFvJreimoKRxQkI7hnK67JhzGMJBew0ko76qGUIXrhdnVBOKnL0Gi+f9BzK VelmAwhadc+ryNcdzDoBQfNRfcEP9zGkn6hQvMYEpUa7/fmUcnzyNQoxYKbxUIXRXDAVwcSJS+2B8 ywk6I31Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sblCH-00000005uop-3CqF; Wed, 07 Aug 2024 18:15:41 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sblBg-00000005ufu-3DsG for linux-arm-kernel@lists.infradead.org; Wed, 07 Aug 2024 18:15:06 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 090CC61132; Wed, 7 Aug 2024 18:15:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C87EC32781; Wed, 7 Aug 2024 18:15:00 +0000 (UTC) Date: Wed, 7 Aug 2024 19:14:58 +0100 From: Catalin Marinas To: Petr =?utf-8?B?VGVzYcWZw61r?= Cc: Baruch Siach , Christoph Hellwig , Marek Szyprowski , Will Deacon , Robin Murphy , iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Ramon Fried , Elad Nachman Subject: Re: [PATCH v5 2/3] dma: replace zone_dma_bits by zone_dma_limit Message-ID: References: <5821a1b2eb82847ccbac0945da040518d6f6f16b.1722578375.git.baruch@tkos.co.il> <20240807161938.5729b656@mordecai.tesarici.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240807161938.5729b656@mordecai.tesarici.cz> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240807_111504_991420_0079D08A X-CRM114-Status: GOOD ( 31.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 07, 2024 at 04:19:38PM +0200, Petr Tesařík wrote: > On Fri, 2 Aug 2024 10:37:38 +0100 > Catalin Marinas wrote: > > On Fri, Aug 02, 2024 at 09:03:47AM +0300, Baruch Siach wrote: > > > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > > > index 3b4be4ca3b08..62b36fda44c9 100644 > > > --- a/kernel/dma/direct.c > > > +++ b/kernel/dma/direct.c > > > @@ -20,7 +20,7 @@ > > > * it for entirely different regions. In that case the arch code needs to > > > * override the variable below for dma-direct to work properly. > > > */ > > > -unsigned int zone_dma_bits __ro_after_init = 24; > > > +u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24); > > > > u64 here makes sense even if it may be larger than phys_addr_t. It > > matches the phys_limit type in the swiotlb code. The compilers should no > > longer complain. > > FTR I have never quite understood why phys_limit is u64, but u64 was > already used all around the place when I first looked into swiotlb. > > > > diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c > > > index d10613eb0f63..7b04f7575796 100644 > > > --- a/kernel/dma/pool.c > > > +++ b/kernel/dma/pool.c > > > @@ -70,9 +70,9 @@ static bool cma_in_zone(gfp_t gfp) > > > /* CMA can't cross zone boundaries, see cma_activate_area() */ > > > end = cma_get_base(cma) + size - 1; > > > if (IS_ENABLED(CONFIG_ZONE_DMA) && (gfp & GFP_DMA)) > > > - return end <= DMA_BIT_MASK(zone_dma_bits); > > > + return end <= zone_dma_limit; > > > if (IS_ENABLED(CONFIG_ZONE_DMA32) && (gfp & GFP_DMA32)) > > > - return end <= DMA_BIT_MASK(32); > > > + return end <= max(DMA_BIT_MASK(32), zone_dma_limit); > > > return true; > > > } > > > > > > diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c > > > index 043b0ecd3e8d..bb51bd5335ad 100644 > > > --- a/kernel/dma/swiotlb.c > > > +++ b/kernel/dma/swiotlb.c > > > @@ -450,9 +450,9 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, > > > if (!remap) > > > io_tlb_default_mem.can_grow = true; > > > if (IS_ENABLED(CONFIG_ZONE_DMA) && (gfp_mask & __GFP_DMA)) > > > - io_tlb_default_mem.phys_limit = DMA_BIT_MASK(zone_dma_bits); > > > + io_tlb_default_mem.phys_limit = zone_dma_limit; > > > else if (IS_ENABLED(CONFIG_ZONE_DMA32) && (gfp_mask & __GFP_DMA32)) > > > - io_tlb_default_mem.phys_limit = DMA_BIT_MASK(32); > > > + io_tlb_default_mem.phys_limit = max(DMA_BIT_MASK(32), zone_dma_limit); > > > else > > > io_tlb_default_mem.phys_limit = virt_to_phys(high_memory - 1); > > > #endif > > > > These two look correct to me now and it's the least intrusive (the > > alternative would have been a zone_dma32_limit). The arch code, however, > > needs to ensure that zone_dma_limit can always support 32-bit devices > > even if it is above 4GB (with the relevant dma offsets in place for such > > devices). > > Just to make sure, the DMA zone (if present) must map to at most 32-bit > bus address space (possibly behind a bridge). Is that what you're > saying? No exactly. What I'm trying to say is that on arm64 zone_dma_limit can go beyond DMA_BIT_MASK(32) when the latter is treated as a CPU address. In such cases, ZONE_DMA32 is empty. TBH, this code is confusing and not entirely suitable for a system where the CPU address offsets are not 0. The device::dma_coherent_mask is about the bus address range and phys_limit is calculated correctly in functions like dma_direct_optimal_gfp_mask(). But that's about it w.r.t. DMA bit masks because zone_dma_bits and DMA_BIT_MASK(32) are assumed to be about the CPU address ranges in some cases (in other cases DMA_BIT_MASK() is used to initialise dma_coherent_mask, so more of a bus address). On the platform Baruch is trying to fix, RAM starts at 32GB and ZONE_DMA should end at 33GB. That's 30-bit mask in bus address terms but something not a power of two for the CPU address, hence the zone_dma_limit introduced here. With ZONE_DMA32, since all the DMA code assumes that ZONE_DMA32 ends at 4GB CPU address, it doesn't really work for such platforms. If there are 32-bit devices with a corresponding CPU address offset, ZONE_DMA32 should end at 36GB on Baruch's platform. But to simplify things, we just ignore this on arm64 and make ZONE_DMA32 empty. In some cases where we have the device structure we could instead do a dma_to_phys(DMA_BIT_MASK(32)) but not in the two cases above. I guess if we really want to address this properly, we'd need to introduce a zone_dma32_limit that's initialised by the arch code. For arm64, I'm happy with just having an empty ZONE_DMA32 on such platforms. -- Catalin