From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7B2E1FAEE5; Thu, 24 Oct 2024 17:15:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729790103; cv=none; b=jiOAENDLtx2oJBRPaS235yWh+ZGvPEfP7MAVrK0SOEq+s44tSEumpcQLRD/eb5Ox3AotCPqZCyJy6cPyp0o7DN/ubc3ay2t1vgzR3u+kVVlMwMjCM+A36Fc6uc1zyCkw6t9hrnz/AW7NFeew9ZAmSa3DQXaWQWfc151B4Rm3Do8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729790103; c=relaxed/simple; bh=qKUU2cq8v+5SaeVu6RRbmqKGY6u205flt7+P+PAhKt8=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=AhmGVqoBujLfgTP9IyjbHmfUh8G3N39qHXs+I07U4ac4VrO+wizCbHdc3zTf53ydIeoI8WGPQNBqVBKSsNCKRtMWDWQfLXeKQmTZv3Gl96pLRFXXqGgv6boqWJyokj67+WA5aJ3O3Dw2b13BsjYPmkppytDS+I3UCrFwwDEjgp4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XLXXrzHU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XLXXrzHU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 459A7C4CEC7; Thu, 24 Oct 2024 17:15:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729790103; bh=qKUU2cq8v+5SaeVu6RRbmqKGY6u205flt7+P+PAhKt8=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=XLXXrzHUAaUf/MCzx4vivPosRtuWGRFNg2bv7Lc6HMcc9UvSy3QscmxKbNY5GuJlE XX/qG+NPu/18qDb8rzdBeQrZr4WySMdwRdjcHplzCvcmA0CnRsVCKCENzgvR2x86sI yUnPDdsMjKr1iR5LnCTAs4ZrZ2+BH1O5jOX7zbFkoBV56fXCyXEFDcEv/zYOxPc+4x Hc2kPzHV9iJkUgu90yGuzHCsH43E0Z1pGEiKQRXkRNJ2O/ajK4U8OznO1g5nYCTasW qw3XBAe+p7izv6WqvgO258kcfLLjp3Kb23hqkqz0UdncqEeJm+i1Xiw3ogpDzSfB2j 8d14lsWaswaBQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1t41QL-006X9S-0p; Thu, 24 Oct 2024 18:15:01 +0100 Date: Thu, 24 Oct 2024 18:15:00 +0100 Message-ID: <86msit2zjf.wl-maz@kernel.org> From: Marc Zyngier To: Johan Hovold Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, Sibi Sankar , Konrad Dybcio , Abel Vesa , Johan Hovold Subject: Re: [PATCH] arm64: dts: qcom: x1e80100: Route pcie5 MSIs to the GIC ITS In-Reply-To: References: <20241024161814.1827514-1-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.4 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: johan@kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, devicetree@vger.kernel.org, quic_sibis@quicinc.com, konradybcio@kernel.org, abel.vesa@linaro.org, johan+linaro@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Thu, 24 Oct 2024 17:25:25 +0100, Johan Hovold wrote: > > On Thu, Oct 24, 2024 at 05:18:14PM +0100, Marc Zyngier wrote: > > There is no reason to use the PCIe root port widget for MSIs for > > pcie5 while both pcie4 and pcie6a are enjoying the ITS. > > > > This is specially useful when booting the kernel at EL2, as KVM > > can then configure the ITS to have MSIs directly injected in guests > > (since this machine has a GICv4.1 implementation). > > > > Tested on a x1e001de devkit. > > > > Signed-off-by: Marc Zyngier > > Cc: Sibi Sankar > > Cc: Konrad Dybcio > > Cc: Abel Vesa > > Cc: Johan Hovold > > --- > > arch/arm64/boot/dts/qcom/x1e80100.dtsi | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > index 3441d167a5cc..48f0ebd66863 100644 > > --- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > +++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi > > @@ -3281,6 +3281,8 @@ pcie5: pci@1c00000 { > > linux,pci-domain = <5>; > > num-lanes = <2>; > > > > + msi-map = <0x0 &gic_its 0xd0000 0x10000>; > > As I just mentioned in another thread, and in the commit message of > 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe") > this was done on purpose as > > PCIe5 (and PCIe3) can currently only be used with the internal > MSI controller due to a platform (firmware) limitation > > Did you try this when booting in EL1? If so we may need to enable this > per board. Nah, you are absolutely correct: when booted at EL1, the ITS driver reports that the ITS queue is no longer making forward progress as soon as we are trying to map something in that range: [ 5.068749] ITS queue timeout (9984 9921) [ 5.072871] ITS cmd its_build_mapd_cmd failed I suspect it trips over itself trying to interpret the command, and that the other PCIe ports work by pure luck (maybe thanks to having a even number?). Comparing the logs, it is obvious that the hypervisor is not showing us the actual HW topology: the ITS supports 64kB pages, which we use when booted at EL2, while we only see 4kB support at EL1. And the boot really is hilarious: [ 0.000000] ITS@0x0000000017040000: Devices Table too large, reduce ids 32->19 19 bits is the maximum the kernel can allocate with a 4kB page size. I would like to see the face of a HW person if they had to design a system with 32bit worth of DeviceID... [ 0.000000] ITS@0x0000000017040000: Devices too large, reduce ITS pages 1024->256 and 256 pages is the maximum we can describe to the ITS... Obviously, this emulation was never really tested, since Windows replaces it at boot time. Oh well. I'll stash this patch as part of my "make EL2 great again" branch! ;-) Thanks, M. -- Without deviation from the norm, progress is not possible.