From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E2C6CCA473 for ; Wed, 1 Jun 2022 18:21:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356598AbiFASVv (ORCPT ); Wed, 1 Jun 2022 14:21:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356590AbiFASVu (ORCPT ); Wed, 1 Jun 2022 14:21:50 -0400 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BAFC65ED; Wed, 1 Jun 2022 11:21:47 -0700 (PDT) Received: by verein.lst.de (Postfix, from userid 2407) id 0E98968AA6; Wed, 1 Jun 2022 20:21:42 +0200 (CEST) Date: Wed, 1 Jun 2022 20:21:41 +0200 From: Christoph Hellwig To: Nathan Chancellor Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Anshuman Khandual , Tom Lendacky , Konrad Rzeszutek Wilk , Stefano Stabellini , Boris Ostrovsky , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-ID: <20220601182141.GA28309@lst.de> References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Precedence: bulk List-ID: X-Mailing-List: linux-hyperv@vger.kernel.org On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Date: Wed, 01 Jun 2022 18:21:41 +0000 Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-Id: <20220601182141.GA28309@lst.de> List-Id: References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Nathan Chancellor Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Anshuman Khandual , Tom Lendacky , Konrad Rzeszutek Wilk , Stefano Stabellini , Boris Ostrovsky , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35A41C433EF for ; Wed, 1 Jun 2022 18:21:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C9CE941856; Wed, 1 Jun 2022 18:21:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7ufB1DI5_z2Q; Wed, 1 Jun 2022 18:21:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 35A2D416B5; Wed, 1 Jun 2022 18:21:49 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 092BBC0032; Wed, 1 Jun 2022 18:21:49 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 13443C002D for ; Wed, 1 Jun 2022 18:21:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D5CE961052 for ; Wed, 1 Jun 2022 18:21:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FMfmKepyl67U for ; Wed, 1 Jun 2022 18:21:47 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by smtp3.osuosl.org (Postfix) with ESMTPS id E17A8607A1 for ; Wed, 1 Jun 2022 18:21:46 +0000 (UTC) Received: by verein.lst.de (Postfix, from userid 2407) id 0E98968AA6; Wed, 1 Jun 2022 20:21:42 +0200 (CEST) Date: Wed, 1 Jun 2022 20:21:41 +0200 From: Christoph Hellwig To: Nathan Chancellor Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-ID: <20220601182141.GA28309@lst.de> References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) Cc: linux-hyperv@vger.kernel.org, x86@kernel.org, linux-ia64@vger.kernel.org, linux-pci@vger.kernel.org, linux-riscv@lists.infradead.org, Christoph Hellwig , linux-s390@vger.kernel.org, Stefano Stabellini , Konrad Rzeszutek Wilk , tboot-devel@lists.sourceforge.net, xen-devel@lists.xenproject.org, David Woodhouse , Tom Lendacky , Anshuman Khandual , Boris Ostrovsky , linux-arm-kernel@lists.infradead.org, Juergen Gross , linuxppc-dev@lists.ozlabs.org, linux-mips@vger.kernel.org, iommu@lists.linux-foundation.org, Robin Murphy X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2ADFEC43334 for ; Wed, 1 Jun 2022 18:22:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mE62oD4YEQXEuMJls4F/7rSeQSLHSmSzF0vhBfk7ctw=; b=gM6bd1e3S8po4I XMqnyI1APOs7LQxv6cLRHviWyJU8R47h/OT8SHpQ2i23LVVWhFWGu8EwaMmOTwd4r5HseqvzoqKR5 Vb8cPIoipnLJN4RPAB8DcibsJ3cHQeZv+b+md6rcnE8Ejw3HEvGC6WJ8raAcEDmKoqAJJhJ1hss7c 9Y2L0ir/RaYmWFN/pkDWqOVb80GveK3TG8X/DRw3I7Kp6JA6Ik9QnqEAIS9g+YweNL4bcAigLX3HI cohb/hR3QE6N7PTcJBwpbaG4sWWxlm+KsuSD/tjuM4n1VVHFkclUZwn69/uS1mZycojfyjoJ7+frm 7W/hTSJ9r0BAr6dWwa8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwSym-00HXtb-3b; Wed, 01 Jun 2022 18:22:00 +0000 Received: from verein.lst.de ([213.95.11.211]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwSya-00HXhc-Ui; Wed, 01 Jun 2022 18:21:50 +0000 Received: by verein.lst.de (Postfix, from userid 2407) id 0E98968AA6; Wed, 1 Jun 2022 20:21:42 +0200 (CEST) Date: Wed, 1 Jun 2022 20:21:41 +0200 From: Christoph Hellwig To: Nathan Chancellor Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Anshuman Khandual , Tom Lendacky , Konrad Rzeszutek Wilk , Stefano Stabellini , Boris Ostrovsky , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-ID: <20220601182141.GA28309@lst.de> References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220601_112149_328786_52F68C63 X-CRM114-Status: GOOD ( 25.41 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0273C43334 for ; Wed, 1 Jun 2022 18:22:13 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4LCyD43mM9z3cfy for ; Thu, 2 Jun 2022 04:22:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lst.de (client-ip=213.95.11.211; helo=verein.lst.de; envelope-from=hch@lst.de; receiver=) Received: from verein.lst.de (verein.lst.de [213.95.11.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4LCyCc3cgdz3bqf for ; Thu, 2 Jun 2022 04:21:48 +1000 (AEST) Received: by verein.lst.de (Postfix, from userid 2407) id 0E98968AA6; Wed, 1 Jun 2022 20:21:42 +0200 (CEST) Date: Wed, 1 Jun 2022 20:21:41 +0200 From: Christoph Hellwig To: Nathan Chancellor Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-ID: <20220601182141.GA28309@lst.de> References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-hyperv@vger.kernel.org, x86@kernel.org, linux-ia64@vger.kernel.org, linux-pci@vger.kernel.org, linux-riscv@lists.infradead.org, Christoph Hellwig , linux-s390@vger.kernel.org, Stefano Stabellini , Joerg Roedel , Konrad Rzeszutek Wilk , tboot-devel@lists.sourceforge.net, xen-devel@lists.xenproject.org, David Woodhouse , Tom Lendacky , Anshuman Khandual , Boris Ostrovsky , linux-arm-kernel@lists.infradead.org, Juergen Gross , linuxppc-dev@lists.ozlabs.org, linux-mips@vger.kernel.org, iommu@lists.linux-foundation.org, Robin Murphy , Lu Baolu Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DB5E5C43334 for ; Wed, 1 Jun 2022 18:22:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lQ2RTBki8TET9YzDNFiy5h8Ng7Fd+du65SrgENrSL0Y=; b=a+0ozZm+kpEjGd BpdskveKZSJ1O3mInXNOlsdVjZmTBXbcvSKJysCt04XlstNhla05gvrKe1zyM+uL3nzSspX4hyaWg OAEequcqTGjosHIZm3ETkU6jY9oOKmBCUTt0jYYIaHR/YIWwH8WMC0S+Bd1MHKe1lnsRfOIq6xo1F XPQdHrIno0r8+M3foPOx4PkgVVKEvV5ddjqcpdihlhtvdx27a2TLXSAmFM3cfyd7QM4kcc5tpX7Em 3uAIrxwTkqmbPVnW7U09RGHFaDzG2vBVZz+bZyOM1LJLaVcj9o5tsB23W36bInBJHgPgPjzTKtYCu BmTZH/S5qZ8G9EG5nYMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwSye-00HXrC-6C; Wed, 01 Jun 2022 18:21:52 +0000 Received: from verein.lst.de ([213.95.11.211]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwSya-00HXhc-Ui; Wed, 01 Jun 2022 18:21:50 +0000 Received: by verein.lst.de (Postfix, from userid 2407) id 0E98968AA6; Wed, 1 Jun 2022 20:21:42 +0200 (CEST) Date: Wed, 1 Jun 2022 20:21:41 +0200 From: Christoph Hellwig To: Nathan Chancellor Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Anshuman Khandual , Tom Lendacky , Konrad Rzeszutek Wilk , Stefano Stabellini , Boris Ostrovsky , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: [PATCH 09/15] swiotlb: make the swiotlb_init interface more useful Message-ID: <20220601182141.GA28309@lst.de> References: <20220404050559.132378-1-hch@lst.de> <20220404050559.132378-10-hch@lst.de> <20220601173441.GB27582@lst.de> <20220601175743.GA28082@lst.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220601_112149_328786_52F68C63 X-CRM114-Status: GOOD ( 25.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jun 01, 2022 at 11:11:57AM -0700, Nathan Chancellor wrote: > On Wed, Jun 01, 2022 at 07:57:43PM +0200, Christoph Hellwig wrote: > > On Wed, Jun 01, 2022 at 10:46:54AM -0700, Nathan Chancellor wrote: > > > On Wed, Jun 01, 2022 at 07:34:41PM +0200, Christoph Hellwig wrote: > > > > Can you send me the full dmesg and the content of > > > > /sys/kernel/debug/swiotlb/io_tlb_nslabs for a good and a bad boot? > > > > > > Sure thing, they are attached! If there is anything else I can provide > > > or test, I am more than happy to do so. > > > > Nothing interesting. But the performance numbers almost look like > > swiotlb=force got ignored before (even if I can't explain why). > > I was able to get my performance back with this diff but I don't know if > this is a hack or a proper fix in the context of the series. This looks good, but needs a little tweak. I'd go for this variant of it: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dfa1de89dc944..cb50f8d383606 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -192,7 +192,7 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc) + unsigned long nslabs, unsigned int flags, bool late_alloc) { void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; @@ -203,8 +203,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->index = 0; mem->late_alloc = late_alloc; - if (swiotlb_force_bounce) - mem->force_bounce = true; + mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); spin_lock_init(&mem->lock); for (i = 0; i < mem->nslabs; i++) { @@ -275,8 +274,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, panic("%s: Failed to allocate %zu bytes align=0x%lx\n", __func__, alloc_size, PAGE_SIZE); - swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, false); - mem->force_bounce = flags & SWIOTLB_FORCE; + swiotlb_init_io_tlb_mem(mem, __pa(tlb), nslabs, flags, false); if (flags & SWIOTLB_VERBOSE) swiotlb_print_info(); @@ -348,7 +346,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, true); + swiotlb_init_io_tlb_mem(mem, virt_to_phys(vstart), nslabs, 0, true); swiotlb_print_info(); return 0; @@ -835,8 +833,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, false); - mem->force_bounce = true; + swiotlb_init_io_tlb_mem(mem, rmem->base, nslabs, SWIOTLB_FORCE, + false); mem->for_alloc = true; rmem->priv = mem; _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel