From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9D3ACE8E6A for ; Thu, 24 Oct 2024 17:39:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=psnlZOaJ47vx5nu3ohbujGpbdy3SwZ6aBUbcSi16HXc=; b=yBE5Qd/m9Cqu+1JlBYNn2mlVXs bNkYtrlJhxG8Kn1/X0Lvdx60pVxfsQSPygrc3UM+XZmdy2krA6uO0F5Fl/i5zqKbF/JddO+AmQhvr N3RLqFtrtuw5oGhpcxQi9GSYLan7K8JROyurq0Ss3WEjQ+Kr3G22jEEDzHURKFn0gpbVE61dYJxdO zaJ/EzKIwB237kABtK7v6Nm/ir9ieQWDpg6wgR+ODIiU3cqjkW7i+iN7/vO4OweJ3nesmxCwfWadu pExyZfgwhOprBJvUcUY8o/1ZNCsSPF1SfcINeUMAghzICop+5kGV8uLinUZnQDdCbwTAXk1F8FrN2 4Oi5Bk2g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t41oE-00000001Jnu-2gSO; Thu, 24 Oct 2024 17:39:42 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t41QO-00000001F50-1yxt for linux-arm-kernel@lists.infradead.org; Thu, 24 Oct 2024 17:15:05 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 6485BA455C1; Thu, 24 Oct 2024 17:14:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 459A7C4CEC7; Thu, 24 Oct 2024 17:15:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729790103; bh=qKUU2cq8v+5SaeVu6RRbmqKGY6u205flt7+P+PAhKt8=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=XLXXrzHUAaUf/MCzx4vivPosRtuWGRFNg2bv7Lc6HMcc9UvSy3QscmxKbNY5GuJlE XX/qG+NPu/18qDb8rzdBeQrZr4WySMdwRdjcHplzCvcmA0CnRsVCKCENzgvR2x86sI yUnPDdsMjKr1iR5LnCTAs4ZrZ2+BH1O5jOX7zbFkoBV56fXCyXEFDcEv/zYOxPc+4x Hc2kPzHV9iJkUgu90yGuzHCsH43E0Z1pGEiKQRXkRNJ2O/ajK4U8OznO1g5nYCTasW qw3XBAe+p7izv6WqvgO258kcfLLjp3Kb23hqkqz0UdncqEeJm+i1Xiw3ogpDzSfB2j 8d14lsWaswaBQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1t41QL-006X9S-0p; Thu, 24 Oct 2024 18:15:01 +0100 Date: Thu, 24 Oct 2024 18:15:00 +0100 Message-ID: <86msit2zjf.wl-maz@kernel.org> From: Marc Zyngier To: Johan Hovold Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, Sibi Sankar , Konrad Dybcio , Abel Vesa , Johan Hovold Subject: Re: [PATCH] arm64: dts: qcom: x1e80100: Route pcie5 MSIs to the GIC ITS In-Reply-To: References: <20241024161814.1827514-1-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.4 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: johan@kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, quic_sibis@quicinc.com, konradybcio@kernel.org, abel.vesa@linaro.org, johan+linaro@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241024_101504_668435_79F523CB X-CRM114-Status: GOOD ( 33.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 24 Oct 2024 17:25:25 +0100, Johan Hovold wrote: > > On Thu, Oct 24, 2024 at 05:18:14PM +0100, Marc Zyngier wrote: > > There is no reason to use the PCIe root port widget for MSIs for > > pcie5 while both pcie4 and pcie6a are enjoying the ITS. > > > > This is specially useful when booting the kernel at EL2, as KVM > > can then configure the ITS to have MSIs directly injected in guests > > (since this machine has a GICv4.1 implementation). > > > > Tested on a x1e001de devkit. > > > > Signed-off-by: Marc Zyngier > > Cc: Sibi Sankar > > Cc: Konrad Dybcio > > Cc: Abel Vesa > > Cc: Johan Hovold > > --- > > arch/arm64/boot/dts/qcom/x1e80100.dtsi | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > index 3441d167a5cc..48f0ebd66863 100644 > > --- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > +++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > @@ -3281,6 +3281,8 @@ pcie5: pci@1c00000 { > > linux,pci-domain = <5>; > > num-lanes = <2>; > > > > + msi-map = <0x0 &gic_its 0xd0000 0x10000>; > > As I just mentioned in another thread, and in the commit message of > 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe") > this was done on purpose as > > PCIe5 (and PCIe3) can currently only be used with the internal > MSI controller due to a platform (firmware) limitation > > Did you try this when booting in EL1? If so we may need to enable this > per board. Nah, you are absolutely correct: when booted at EL1, the ITS driver reports that the ITS queue is no longer making forward progress as soon as we are trying to map something in that range: [ 5.068749] ITS queue timeout (9984 9921) [ 5.072871] ITS cmd its_build_mapd_cmd failed I suspect it trips over itself trying to interpret the command, and that the other PCIe ports work by pure luck (maybe thanks to having a even number?). Comparing the logs, it is obvious that the hypervisor is not showing us the actual HW topology: the ITS supports 64kB pages, which we use when booted at EL2, while we only see 4kB support at EL1. And the boot really is hilarious: [ 0.000000] ITS@0x0000000017040000: Devices Table too large, reduce ids 32->19 19 bits is the maximum the kernel can allocate with a 4kB page size. I would like to see the face of a HW person if they had to design a system with 32bit worth of DeviceID... [ 0.000000] ITS@0x0000000017040000: Devices too large, reduce ITS pages 1024->256 and 256 pages is the maximum we can describe to the ITS... Obviously, this emulation was never really tested, since Windows replaces it at boot time. Oh well. I'll stash this patch as part of my "make EL2 great again" branch! ;-) Thanks, M. -- Without deviation from the norm, progress is not possible.