From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8711E606A33A for ; Wed, 11 Nov 2026 11:53:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fujitsu.com header.i=@fujitsu.com header.a=rsa-sha256 header.s=fj2 header.b=UpmG4/Pt Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4jPhk16DPHz2y8W; Wed, 11 Nov 2026 22:53:29 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip=68.232.139.117 ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1794398009; cv=none; b=IKj+Ad+VQmqSJDdaR/XoUIJrQPn2dUNF4thorP6mfpLBzLkBfEOb2L3Sp1Xt+iqgZhD8lYPmki1XLZUz/twiIynMvlyPjyn+0jew1sTrEXUvl/x+HAB4mohdqBdJF6KKbKLktH3GrprAPpsA8tlaIWJczlJFwouLoqVIVG9lDBxmt0vtGXgc13O6vIrySPAgzHPMDaMpIqPmOVOWcRsn5dOypxROnyRk398y2dAibvn3JV20gvQ4oz9N2BVWzvJkXq4FpTL/Dxkm3XCaYcg+zav07iMIFi+O/Jdgm2HqJAxIShK/qwpg9AY580/hJHWkXEEIvLAwsVBastXawtCg1A== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1794398009; c=relaxed/relaxed; bh=LHx3sOu0nRc26uMaGdX5eIGUhuQ24/CQDwrvILlx8g0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=k8wmNTb+iv4AX9oAkJR3ZgpAG2TbBnuU5kReBb9PFeetBlp4aPTOL/l1yxPwl3H3IBEcrBLby7yZmevjIRZhR0uSPhkVJYFOCvoVEB1V87rRjSnA7fyAR5ZFGBGHWXg/uj8j6JDY8rI9JWOcZ9Wlbb7G6LrpVGOrRerJlZrTAj2mSDjw7n4Rq94F13ocaDrhyp358Lt24w5VhywRc6BLuTxKITK5rDof2aTBCEgtS4w734znmLdPSrsUvglkQjhpGCC0zdUJQjYRFiAnJGIE52+EKvObPeuZ4f0RKOrCE7WBvUr0HSVMcp5ucN6sF0FzdNur1ubpEI8hlZfEnncvdA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=reject dis=none) header.from=fujitsu.com; dkim=pass (2048-bit key; unprotected) header.d=fujitsu.com header.i=@fujitsu.com header.a=rsa-sha256 header.s=fj2 header.b=UpmG4/Pt; dkim-atps=neutral; spf=pass (client-ip=68.232.139.117; helo=esa4.hc1455-7.c3s2.iphmx.com; envelope-from=enju.kohei@fujitsu.com; receiver=lists.ozlabs.org) smtp.mailfrom=fujitsu.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=reject dis=none) header.from=fujitsu.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=fujitsu.com header.i=@fujitsu.com header.a=rsa-sha256 header.s=fj2 header.b=UpmG4/Pt; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=fujitsu.com (client-ip=68.232.139.117; helo=esa4.hc1455-7.c3s2.iphmx.com; envelope-from=enju.kohei@fujitsu.com; receiver=lists.ozlabs.org) Received: from esa4.hc1455-7.c3s2.iphmx.com (esa4.hc1455-7.c3s2.iphmx.com [68.232.139.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4jPhjz6McDz2y2h for ; Wed, 11 Nov 2026 22:53:27 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=fujitsu.com; i=@fujitsu.com; q=dns/txt; s=fj2; t=1794398008; x=1825934008; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=HCCipxtjnPjwYW8DIXAeMVHDqScHm74Hg2zPPlBSipM=; b=UpmG4/Pt6qDpsxip0swbDY7aLmTQEqaRE6zbHWo2WDvgwpMRi898jwz2 OheEz/RvMgl/wf7PTIRYCte8D2L8GfBnLDvxX7kLIkxhkjZhpogvPQF7m 5CKZvMCiNufymQc1tL7MDndSXbZTYXaJSpLMlJ/rif/5VGNuwmmJzTjDn xsUYCuP+MkURD7tHrekauosvTP3YjJZqWa323oj/Ta/RIXEhqxNfiq3fu bQHh2SJVH1fBG3JXGqdlQtNH9JefcdckYY3atL/aUczRjzV3b3MiQ1jgM 0SMtPiiQPR3lrx+IeiXCuW4n4Q3PbAcTAavCvT2FLaurhUSpfZg7soZIB Q==; X-CSE-ConnectionGUID: ikC+KU1IQWC6wOnSmViT8A== X-CSE-MsgGUID: Kl8hS0gZSQCDcNhPeUaynA== X-IronPort-AV: E=McAfee;i="6800,10657,11962"; a="260954464" X-IronPort-AV: E=Sophos;i="6.27,207,1786978800"; d="scan'208";a="260954464" Received: from gmgwuk01.global.fujitsu.com ([172.187.114.235]) by esa4.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Nov 2026 20:53:22 +0900 Received: from az2uksmgm2.o.css.fujitsu.com (unknown [10.151.22.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by gmgwuk01.global.fujitsu.com (Postfix) with ESMTPS id A090BC05E15 for ; Wed, 11 Nov 2026 11:53:22 +0000 (UTC) Received: from az2nlsmom3.fujitsu.com (unknown [10.150.26.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by az2uksmgm2.o.css.fujitsu.com (Postfix) with ESMTPS id 4F59818002D0 for ; Wed, 11 Nov 2026 11:53:22 +0000 (UTC) Received: from FCCLS0092175.localdomain (unknown [10.9.86.149]) by az2nlsmom3.fujitsu.com (Postfix) with SMTP id 52B3F101BB7F; Wed, 11 Nov 2026 11:53:02 +0000 (UTC) Date: Wed, 11 Nov 2026 20:53:00 +0900 From: Kohei Enju To: "Aneesh Kumar K.V" Cc: Catalin Marinas , Will Deacon , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Robin Murphy , Marek Szyprowski , Jonathan Corbet , Shuah Khan , Randy Dunlap , Mark Rutland , Marc Zyngier , Steven Price , Suzuki K Poulose , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , "Ritesh Harjani (IBM)" , Shrikanth Hegde , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , Stefano Stabellini , Russell King , Huacai Chen , WANG Xuerui , Thomas Bogendoerfer , Jiaxun Yang , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, x86@kernel.org Subject: Re: [PATCH v6 6/8] dma: swiotlb: Centralize memory-encryption pool sizing Message-ID: References: <20260924060756.1325156-7-aneesh.kumar@kernel.org> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , Precedence: list X-Archived-By: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=4.0.1 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on aws-us-west-2-korg-listserv-alma10-1 Hi Aneesh, On 11/01 09:47, Aneesh Kumar K.V wrote: [...] > > > > Hi Aneesh, > > > > I can reproduce the CCA regression discussed in this thread: > > https://lore.kernel.org/all/yq5azeyxyfol.fsf@kernel.org/#t > > > > The guest runs out of SWIOTLB slots, causing disk I/O to fail and > > preventing the guest from booting. > > > > I would like to test the fix on real CCA hardware. Could you please Cc > > me when you respin the series? I follow LKML, but I may still miss the > > posting by chance. > > > > The conclusion was to either use CONFIG_SWIOTLB_DYNAMIC or swiotlb kernel command line > option Thanks for the clarification. After reading this whole thread, I agree that dynamic sizing would be appropriate. However, currently some distributions do not enable CONFIG_SWIOTLB_DYNAMIC. Users of such distributions would therefore need to rebuild their kernels with the option enabled. This is not a practical solution for many users, particularly cloud users who use kernels provided by their distributions or cloud providers. Specifying swiotlb= on the kernel command line is also not particularly user-friendly. It requires users or cloud providers to choose a fixed value in advance, without automatically taking into account properties of the VM, such as its memory size or whether it is a confidential guest. I do not think the existing heuristic used for x86 CoCo guests is necessarily optimal, but a fixed swiotlb= value chosen by each user may be even less appropriate. Overall, I would prefer the dynamic sizing approach. However, I believe the initial pool size still matters. Also, as Catalin pointed out earlier in the following message: https://lore.kernel.org/all/ann6noRzoPLL9kDY@arm.com/ the arm64 adjustment also reduces the target size used for dynamic pool allocation, since both the initial pool and dynamic allocation currently use default_nslabs. I am testing the patch below to decouple the initial pool size from the dynamic allocation target. It adds swiotlb_adjust_initial_size() for callers that only want to reduce the initial allocation, while keeping the existing swiotlb_adjust_size() semantics for callers that want to adjust both sizes. Thanks, Kohei --->8--- diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 4e7682dfe366..80a2c171895f 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -349,7 +349,7 @@ void __init arch_mm_preinit(void) unsigned long size = DIV_ROUND_UP(memblock_phys_mem_size(), 1024); - swiotlb_adjust_size(min(swiotlb_default_pool_size(), size)); + swiotlb_adjust_initial_size(min(swiotlb_default_pool_size(), size)); } swiotlb_init(true, flags); diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 070ee69eabb3..7dd9fcec90a5 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -189,7 +189,7 @@ void __init arch_mm_preinit(void) */ unsigned long size = DIV_ROUND_UP(memblock_phys_mem_size(), 1024); - swiotlb_adjust_size(min(swiotlb_default_pool_size(), size)); + swiotlb_adjust_initial_size(min(swiotlb_default_pool_size(), size)); swiotlb = true; swiotlb_flags |= SWIOTLB_ANY; } diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f24edf47b68..c1984ee12355 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -190,6 +190,7 @@ size_t swiotlb_max_mapping_size(struct device *dev); bool is_swiotlb_allocated(void); bool is_swiotlb_active(struct device *dev); void __init swiotlb_adjust_size(unsigned long size); +void __init swiotlb_adjust_initial_size(unsigned long size); phys_addr_t default_swiotlb_base(void); phys_addr_t default_swiotlb_limit(void); #else @@ -232,6 +233,10 @@ static inline void swiotlb_adjust_size(unsigned long size) { } +static inline void swiotlb_adjust_initial_size(unsigned long size) +{ +} + static inline phys_addr_t default_swiotlb_base(void) { return 0; diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 2c57b49ac7c0..211fefb32fb5 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -99,6 +99,7 @@ static struct io_tlb_mem io_tlb_default_mem; #endif /* CONFIG_SWIOTLB_DYNAMIC */ static unsigned long default_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT; +static unsigned long dynamic_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT; static unsigned long default_nareas; static bool swiotlb_cmdline_size_set __initdata; @@ -152,6 +153,8 @@ static void swiotlb_adjust_nareas(unsigned int nareas) nareas = roundup_pow_of_two(nareas); default_nareas = nareas; + dynamic_nslabs = + swiotlb_calc_nslabs(dynamic_nslabs << IO_TLB_SHIFT, nareas); pr_info("area num %d.\n", nareas); nslabs = swiotlb_calc_nslabs(default_nslabs << IO_TLB_SHIFT, nareas); @@ -254,6 +257,7 @@ setup_io_tlb_npages(char *str) /* avoid tail segment of size < IO_TLB_SEGSIZE */ default_nslabs = ALIGN(simple_strtoul(str, &str, 0), IO_TLB_SEGSIZE); + dynamic_nslabs = default_nslabs; swiotlb_cmdline_size_set = true; } if (*str == ',') @@ -290,7 +294,7 @@ unsigned long swiotlb_default_pool_size(void) return default_nslabs << IO_TLB_SHIFT; } -void __init swiotlb_adjust_size(unsigned long size) +void __init swiotlb_adjust_initial_size(unsigned long size) { /* * If swiotlb parameter has not been specified, give a chance to @@ -305,6 +309,12 @@ void __init swiotlb_adjust_size(unsigned long size) pr_info("SWIOTLB bounce buffer size adjusted to %luMB", size >> 20); } +void __init swiotlb_adjust_size(unsigned long size) +{ + swiotlb_adjust_initial_size(size); + dynamic_nslabs = default_nslabs; +} + void swiotlb_print_info(void) { struct io_tlb_pool *mem = &io_tlb_default_mem.defpool; @@ -507,6 +517,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, pr_info("SWIOTLB bounce buffer size adjusted %lu -> %lu slabs", default_nslabs, nslabs); default_nslabs = nslabs; + dynamic_nslabs = default_nslabs; } alloc_size = PAGE_ALIGN(array_size(sizeof(*mem->slots), nslabs)); @@ -889,7 +900,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work) container_of(work, struct io_tlb_mem, dyn_alloc); struct io_tlb_pool *pool; - pool = swiotlb_alloc_pool(NULL, mem, IO_TLB_MIN_SLABS, default_nslabs, + pool = swiotlb_alloc_pool(NULL, mem, IO_TLB_MIN_SLABS, dynamic_nslabs, default_nareas, mem->phys_limit, GFP_KERNEL); if (!pool) { pr_warn_ratelimited("Failed to allocate new pool");