From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FC4BC433F5 for ; Fri, 25 Feb 2022 08:48:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238591AbiBYIsf (ORCPT ); Fri, 25 Feb 2022 03:48:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236360AbiBYIse (ORCPT ); Fri, 25 Feb 2022 03:48:34 -0500 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6BBA169233; Fri, 25 Feb 2022 00:48:01 -0800 (PST) Received: by verein.lst.de (Postfix, from userid 2407) id 364BF68AA6; Fri, 25 Feb 2022 09:47:56 +0100 (CET) Date: Fri, 25 Feb 2022 09:47:55 +0100 From: Christoph Hellwig To: Boris Ostrovsky Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Stefano Stabellini , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: cleanup swiotlb initialization Message-ID: <20220225084755.GA29630@lst.de> References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> User-Agent: Mutt/1.5.17 (2007-11-01) Precedence: bulk List-ID: X-Mailing-List: linux-hyperv@vger.kernel.org On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Date: Fri, 25 Feb 2022 08:47:55 +0000 Subject: Re: cleanup swiotlb initialization Message-Id: <20220225084755.GA29630@lst.de> List-Id: References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Boris Ostrovsky Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Stefano Stabellini , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4026C433EF for ; Fri, 25 Feb 2022 08:48:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4E94741705; Fri, 25 Feb 2022 08:48:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 19ZxnDMTbVko; Fri, 25 Feb 2022 08:48:08 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 9D00241719; Fri, 25 Feb 2022 08:48:07 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A39CEC007A; Fri, 25 Feb 2022 08:48:05 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1C629C001A for ; Fri, 25 Feb 2022 08:48:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id DD48340211 for ; Fri, 25 Feb 2022 08:48:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bome4AsNN5G7 for ; Fri, 25 Feb 2022 08:48:03 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by smtp4.osuosl.org (Postfix) with ESMTPS id C336940265 for ; Fri, 25 Feb 2022 08:48:02 +0000 (UTC) Received: by verein.lst.de (Postfix, from userid 2407) id 364BF68AA6; Fri, 25 Feb 2022 09:47:56 +0100 (CET) Date: Fri, 25 Feb 2022 09:47:55 +0100 From: Christoph Hellwig To: Boris Ostrovsky Subject: Re: cleanup swiotlb initialization Message-ID: <20220225084755.GA29630@lst.de> References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Juergen Gross , linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, Stefano Stabellini , linux-ia64@vger.kernel.org, Robin Murphy , x86@kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, iommu@lists.linux-foundation.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, linux-riscv@lists.infradead.org, David Woodhouse , Christoph Hellwig , linux-arm-kernel@lists.infradead.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 81FCFC433F5 for ; Fri, 25 Feb 2022 08:48:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r+qa/k6Vx32DoqBoAMQ+Z82yNQ0w1jFIJWCLDla12n8=; b=xuGQC90jTY7z7E whcVBONnEl+lZHfqrZHlHvzzqH4TvCKac4qPEFq0le3v16A2lq6tpr4Gy7HyM66/w/4JsXllK+WIa nUvy6netFf0kKuO9anAgOrdIBV6TP2VM4EUXSsZ4Xuh1XAyKprwsppBjhrilSH4dQkrAikJbbwIZo AJqEMHLKfw3fwHSWD/pBPA/epkC7V6hxp5C23owx9EeG/Y7MqUi5Id1TdQosaCyo0bajKHH2tIF2U x2cOqWBrdqY4b6gKtFojyxy9Spw6yZEu3xKisLdZHBTjNf2m5CIklDQKT35UML2Tf0K2KbJwsESzb 12Bekfnz30PmNaoqfnww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNWGx-00410X-9e; Fri, 25 Feb 2022 08:48:19 +0000 Received: from verein.lst.de ([213.95.11.211]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNWGj-0040wu-8L; Fri, 25 Feb 2022 08:48:08 +0000 Received: by verein.lst.de (Postfix, from userid 2407) id 364BF68AA6; Fri, 25 Feb 2022 09:47:56 +0100 (CET) Date: Fri, 25 Feb 2022 09:47:55 +0100 From: Christoph Hellwig To: Boris Ostrovsky Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Stefano Stabellini , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: cleanup swiotlb initialization Message-ID: <20220225084755.GA29630@lst.de> References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> User-Agent: Mutt/1.5.17 (2007-11-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220225_004805_469497_D7319F2D X-CRM114-Status: GOOD ( 16.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C82C3C433EF for ; Fri, 25 Feb 2022 08:48:28 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4K4k2L3c47z3cFh for ; Fri, 25 Feb 2022 19:48:26 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lst.de (client-ip=213.95.11.211; helo=verein.lst.de; envelope-from=hch@lst.de; receiver=) Received: from verein.lst.de (verein.lst.de [213.95.11.211]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4K4k1t3Nmgz30gg for ; Fri, 25 Feb 2022 19:48:01 +1100 (AEDT) Received: by verein.lst.de (Postfix, from userid 2407) id 364BF68AA6; Fri, 25 Feb 2022 09:47:56 +0100 (CET) Date: Fri, 25 Feb 2022 09:47:55 +0100 From: Christoph Hellwig To: Boris Ostrovsky Subject: Re: cleanup swiotlb initialization Message-ID: <20220225084755.GA29630@lst.de> References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> User-Agent: Mutt/1.5.17 (2007-11-01) X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, Stefano Stabellini , linux-ia64@vger.kernel.org, Robin Murphy , Joerg Roedel , x86@kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, iommu@lists.linux-foundation.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org, xen-devel@lists.xenproject.org, linux-riscv@lists.infradead.org, David Woodhouse , Christoph Hellwig , linux-arm-kernel@lists.infradead.org, Lu Baolu Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA9A2C433EF for ; Fri, 25 Feb 2022 08:49:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QKSDCjz/GSESdyd5/fIwFdRgVMuWHOJ8RQutTm/EqBM=; b=0caA+xaFtJ1TZn vxRYWibuJiw+q//em63HfsjyC326YQ7VyA3OL4iT3qQa+K6fqm1vpCxguyX77PelkPsTYyVoEtvly /hkneIaW49zPSGDCXuHi7N3zwqyiV6GZUJSlQJZ3I/QgOUGvgXnadVFU5Ddy3tM16KQbHc0WJalpp 4fedEaqgsifd01zspdja+7B2dGEdW2V15U7nLCU7qldu3tRD8HIkVQw3gmWZPYxtT+lOXbtg/iO1G 4rA56avGvb+dJFKgIPfheW9BMk3nOFFJ97semtRRtzO2mMxzJggnI8S009cymL9z8H4iXNOEw4C77 Oxgd6DW2qnj/89myMa8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNWGo-0040yC-NG; Fri, 25 Feb 2022 08:48:10 +0000 Received: from verein.lst.de ([213.95.11.211]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNWGj-0040wu-8L; Fri, 25 Feb 2022 08:48:08 +0000 Received: by verein.lst.de (Postfix, from userid 2407) id 364BF68AA6; Fri, 25 Feb 2022 09:47:56 +0100 (CET) Date: Fri, 25 Feb 2022 09:47:55 +0100 From: Christoph Hellwig To: Boris Ostrovsky Cc: Christoph Hellwig , iommu@lists.linux-foundation.org, x86@kernel.org, Stefano Stabellini , Juergen Gross , Joerg Roedel , David Woodhouse , Lu Baolu , Robin Murphy , linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-hyperv@vger.kernel.org, tboot-devel@lists.sourceforge.net, linux-pci@vger.kernel.org Subject: Re: cleanup swiotlb initialization Message-ID: <20220225084755.GA29630@lst.de> References: <20220222153514.593231-1-hch@lst.de> <09cb4ad3-88e7-3744-b4b8-a6d745ecea9e@oracle.com> <20220224155854.GA30938@lst.de> <206ba6a3-770a-70ad-96bc-76c6380da988@oracle.com> <20220224163943.GA32088@lst.de> <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <8ffd8587-7eb3-d5b6-eab0-b86df5c0ebbd@oracle.com> User-Agent: Mutt/1.5.17 (2007-11-01) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220225_004805_469497_D7319F2D X-CRM114-Status: GOOD ( 16.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: >>> Just to be clear: this crashes only as dom0. Boots fine as baremetal. >> Ah. I can gues what this might be. On Xen the hypervisor controls the >> IOMMU and we should never end up initializing it in Linux, right? > > > Right, we shouldn't be in that code path. Can you try the patch below on top of the series? diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index b849f11a756d0..f88c9a4a5ed12 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags; */ static void __init pci_swiotlb_detect_4gb(void) { -#ifdef CONFIG_SWIOTLB_XEN - if (xen_pv_domain()) { - if (xen_initial_domain()) - x86_swiotlb_enable = true; - - if (x86_swiotlb_enable) { - dma_ops = &xen_swiotlb_dma_ops; -#ifdef CONFIG_PCI - /* Make sure ACS will be enabled */ - pci_request_acs(); -#endif - } - return; - } -#endif /* CONFIG_SWIOTLB_XEN */ - /* don't initialize swiotlb if iommu=off (no_iommu=1) */ if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN) x86_swiotlb_enable = true; @@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void) } #endif /* CONFIG_SWIOTLB */ +#ifdef CONFIG_SWIOTLB_XEN +static void __init xen_swiotlb_init(void) +{ + if (!xen_initial_domain() && !x86_swiotlb_enable) + return; + x86_swiotlb_enable = true; + dma_ops = &xen_swiotlb_dma_ops; +#ifdef CONFIG_PCI + /* Make sure ACS will be enabled */ + pci_request_acs(); +#endif + swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup); +} +#else +static inline void __init xen_swiotlb_init(void) +{ +} +#endif /* CONFIG_SWIOTLB_XEN */ + void __init pci_iommu_alloc(void) { + if (xen_pv_domain()) { + xen_swiotlb_init(); + return; + } pci_swiotlb_detect_4gb(); gart_iommu_hole_init(); amd_iommu_detect(); detect_intel_iommu(); -#ifdef CONFIG_SWIOTLB - swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags, - xen_pv_domain() ? xen_swiotlb_fixup : NULL); -#endif + swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags); } /* _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel