From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9622EC433ED for ; Tue, 13 Apr 2021 09:23:41 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1DFA96120E for ; Tue, 13 Apr 2021 09:23:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1DFA96120E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-rockchip-bounces+linux-rockchip=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SiOeeYfje8ONYmyI8RMx1fh1ts7qcDGjHvCmGcy8eGs=; b=Vi9gtgwbRbFEbamCPKvisGo8Y 6NloNGScooyxea4O3gfj0+xRBevdWREfwRrGLH2BEY3dSlCZ9PZMY6f7A3WTHoTFbcNkhAWpOUU52 Tz04be28AjIAhWd7J+kRNXZpAW9z3EYw+gY+LH5AEPpFres6qnDYx8TzYWvrlIY5jVQt9O8p8wi6X QC/aQRP0mlmU5+/xOO1jp0N7XNkBR1ijsv+RuuFJN9oBoRyTtZzT+Td7OQNNoVHt9ZS7vK+Qo8Uff SRHO5qKBL73UBq0Xt7UnDtVi5QZru4OJJq5vo/fdWTumoG+qEcdcWixF+9YRUOXPe/STCuYtuHsVT CkUYxqsHQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lWFGg-008hG8-Se; Tue, 13 Apr 2021 09:23:35 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lWFGQ-008hCs-DL for linux-rockchip@desiato.infradead.org; Tue, 13 Apr 2021 09:23:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=iBUsfzP+/jwaMYQ/2t33mGzdXWhKAbF5SXuS16riXxk=; b=naTVv/i8U9y0DvnfWkpToRFzjT z6klp64RYetUt57gGanNojpZsEikeTtIE0mo/uVsjWAAFV4sIto/HoNeQLJceDAUGGUzn1r5qJejZ WWOgV5l/80+bHYWbDFVBdaQE/t0Tn0C/Qza3erYN6dp5f5Ngr5okE7648hxx2juEwJaQDN7Fx8lDG 8n5UJseBGgv94/5tFbZtjQ7A/XJhwlfpoAdXrs6HEWr+bZRvabujFTivtRl8UoAXDjcZbJjS69GAX 54kqKKVM5WbQ1fOWih9snzsfUy6bMSbIesP9eFoyVx5FhpRLtmPMI6Mwict3FZ/DTpZeKw5VTqSqZ K2L9jEiA==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lWFGN-006sFT-P3 for linux-rockchip@lists.infradead.org; Tue, 13 Apr 2021 09:23:17 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 72D4A611F2; Tue, 13 Apr 2021 09:23:15 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lWFGL-007CIx-7A; Tue, 13 Apr 2021 10:23:13 +0100 Date: Tue, 13 Apr 2021 10:23:12 +0100 Message-ID: <871rbeo7wf.wl-maz@kernel.org> From: Marc Zyngier To: Peter Geis Cc: Thomas Gleixner , "open list:ARM/Rockchip SoC..." , linux-kernel@vger.kernel.org Subject: Re: [RFC] ITS fails to allocate on rk3568/rk3566 In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: pgwipeout@gmail.com, tglx@linutronix.de, linux-rockchip@lists.infradead.org, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210413_022315_875556_432799A3 X-CRM114-Status: GOOD ( 29.56 ) X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+linux-rockchip=archiver.kernel.org@lists.infradead.org Hi Peter, On Mon, 12 Apr 2021 21:49:59 +0100, Peter Geis wrote: > > Good Afternoon, > > I am assisting with early bringup of the rk3566 based quartz64 > development board for mainline linux. > I've encountered a few issues with allocating ITS on their version of > the GIC-V3. > The first issue is the ITS controller can only use 32bit addresses. > This leads to the following error: > [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode > [ 0.000000] GICv3: 320 SPIs implemented > [ 0.000000] GICv3: 0 Extended SPIs implemented > [ 0.000000] GICv3: Distributor has no Range Selector support > [ 0.000000] Root IRQ handler: gic_handle_irq > [ 0.000000] GICv3: 16 PPIs implemented > [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000 > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: Devices doesn't stick: > f907000100190600 f907000000190600 Ouch. That looks pretty bad. Bit 32 of the register doesn't stick, and that's right in the middle of the address. The register should be fully writable as far as the address field is concerned. Please dump the distributor and ITS IIDR registers so that I can find the TRM for the exact IP. > [ 0.000000] ITS@0x00000000fd440000: failed probing (-6) > [ 0.000000] ITS: No ITS available, not enabling LPIs > > Downstream fixed this by adding the GFP_DMA32 flag to the memory > allocations. Urgh... this really looks like broken silicon to me. > They also force clear the GITS_BASER_SHAREABILITY_MASK. Why? Does this also apply to the command queue? Are they forcing cache flushing? > Unfortunately while this allowed ITS to allocate on downstream, as > soon as MSIs attempted to use it all interrupts would time out. > > On upstream, we observe this during allocation: > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @3810000 > (indirect, esz 8, psz 64K, shr 1) > [ 0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt > Collections @3820000 (flat, esz 2, psz 64K, shr 1) > [ 0.000000] GICv3: using LPI property table @0x0000000003830000 > [ 0.000000] GICv3: CPU0: using allocated LPI pending table > @0x0000000003840000 > [ 0.000000] ITS queue timeout (64 1) > [ 0.000000] ITS cmd its_build_mapc_cmd failed > [ 0.000000] ITS queue timeout (96 1) > [ 0.000000] ITS cmd its_build_invall_cmd failed > So the command queue is not making forward progress. Either because the ITS cannot access the commands, or because it cannot use the memory it has been allocated. Please dump GITS_CBASER (and the value that has been written to it), just in case it shows the same brokenness as the GITS_BASER registers... [...] > Any assistance you can provide would be greatly appreciated. I'm not sure there is much we can do without a lot more details about the HW. We need to know the exact GIC implementation they are using (ARM has two versions of the GICv3 IP), and we also need to find out *how* this has been integrated. Only Rockchip can tell you that. Once we know which version they are using, and how it is wired, we can start looking at some IMPDEF debug registers. The Linux driver assumes that the ITS is able to access the whole memory. If there are restrictions on what memory ranges can be used, it is going to be a pain to support :-(. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ Linux-rockchip mailing list Linux-rockchip@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-rockchip From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D14B6C433B4 for ; Tue, 13 Apr 2021 09:23:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B3AF1613BA for ; Tue, 13 Apr 2021 09:23:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245638AbhDMJXf (ORCPT ); Tue, 13 Apr 2021 05:23:35 -0400 Received: from mail.kernel.org ([198.145.29.99]:47394 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237403AbhDMJXe (ORCPT ); Tue, 13 Apr 2021 05:23:34 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 72D4A611F2; Tue, 13 Apr 2021 09:23:15 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lWFGL-007CIx-7A; Tue, 13 Apr 2021 10:23:13 +0100 Date: Tue, 13 Apr 2021 10:23:12 +0100 Message-ID: <871rbeo7wf.wl-maz@kernel.org> From: Marc Zyngier To: Peter Geis Cc: Thomas Gleixner , "open list:ARM/Rockchip SoC..." , linux-kernel@vger.kernel.org Subject: Re: [RFC] ITS fails to allocate on rk3568/rk3566 In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: pgwipeout@gmail.com, tglx@linutronix.de, linux-rockchip@lists.infradead.org, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Peter, On Mon, 12 Apr 2021 21:49:59 +0100, Peter Geis wrote: > > Good Afternoon, > > I am assisting with early bringup of the rk3566 based quartz64 > development board for mainline linux. > I've encountered a few issues with allocating ITS on their version of > the GIC-V3. > The first issue is the ITS controller can only use 32bit addresses. > This leads to the following error: > [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode > [ 0.000000] GICv3: 320 SPIs implemented > [ 0.000000] GICv3: 0 Extended SPIs implemented > [ 0.000000] GICv3: Distributor has no Range Selector support > [ 0.000000] Root IRQ handler: gic_handle_irq > [ 0.000000] GICv3: 16 PPIs implemented > [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000 > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: Devices doesn't stick: > f907000100190600 f907000000190600 Ouch. That looks pretty bad. Bit 32 of the register doesn't stick, and that's right in the middle of the address. The register should be fully writable as far as the address field is concerned. Please dump the distributor and ITS IIDR registers so that I can find the TRM for the exact IP. > [ 0.000000] ITS@0x00000000fd440000: failed probing (-6) > [ 0.000000] ITS: No ITS available, not enabling LPIs > > Downstream fixed this by adding the GFP_DMA32 flag to the memory > allocations. Urgh... this really looks like broken silicon to me. > They also force clear the GITS_BASER_SHAREABILITY_MASK. Why? Does this also apply to the command queue? Are they forcing cache flushing? > Unfortunately while this allowed ITS to allocate on downstream, as > soon as MSIs attempted to use it all interrupts would time out. > > On upstream, we observe this during allocation: > [ 0.000000] ITS [mem 0xfd440000-0xfd45ffff] > [ 0.000000] ITS@0x00000000fd440000: allocated 8192 Devices @3810000 > (indirect, esz 8, psz 64K, shr 1) > [ 0.000000] ITS@0x00000000fd440000: allocated 32768 Interrupt > Collections @3820000 (flat, esz 2, psz 64K, shr 1) > [ 0.000000] GICv3: using LPI property table @0x0000000003830000 > [ 0.000000] GICv3: CPU0: using allocated LPI pending table > @0x0000000003840000 > [ 0.000000] ITS queue timeout (64 1) > [ 0.000000] ITS cmd its_build_mapc_cmd failed > [ 0.000000] ITS queue timeout (96 1) > [ 0.000000] ITS cmd its_build_invall_cmd failed > So the command queue is not making forward progress. Either because the ITS cannot access the commands, or because it cannot use the memory it has been allocated. Please dump GITS_CBASER (and the value that has been written to it), just in case it shows the same brokenness as the GITS_BASER registers... [...] > Any assistance you can provide would be greatly appreciated. I'm not sure there is much we can do without a lot more details about the HW. We need to know the exact GIC implementation they are using (ARM has two versions of the GICv3 IP), and we also need to find out *how* this has been integrated. Only Rockchip can tell you that. Once we know which version they are using, and how it is wired, we can start looking at some IMPDEF debug registers. The Linux driver assumes that the ITS is able to access the whole memory. If there are restrictions on what memory ranges can be used, it is going to be a pain to support :-(. M. -- Without deviation from the norm, progress is not possible.