From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 90105C433FE for ; Thu, 10 Feb 2022 22:44:02 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 6017480F68; Thu, 10 Feb 2022 23:44:00 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: by phobos.denx.de (Postfix, from userid 109) id 800B2836A3; Thu, 10 Feb 2022 23:43:58 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by phobos.denx.de (Postfix) with ESMTP id E636F80615 for ; Thu, 10 Feb 2022 23:43:54 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=andre.przywara@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4C7D0D6E; Thu, 10 Feb 2022 14:43:54 -0800 (PST) Received: from slackpad.lan (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B9D5E3F70D; Thu, 10 Feb 2022 14:43:53 -0800 (PST) Date: Thu, 10 Feb 2022 22:43:18 +0000 From: Andre Przywara To: Joakim Tjernlund Cc: "u-boot@lists.denx.de" Subject: Re: ARM A53 and initial MMU mapping for EL0/1/2/3 ? Message-ID: <20220210224318.1d9e11b3@slackpad.lan> In-Reply-To: <3902a00ebedaa54e71af4d10726e195ed557f1db.camel@infinera.com> References: <52270ef295b438af775d6f00e3c6e97f6482d74c.camel@infinera.com> <20220209003308.015c087d@slackpad.lan> <70ea450b3c2005c594488dc717c0576e721f6bd4.camel@infinera.com> <20220209104514.5f9bb2bd@donnerap.cambridge.arm.com> <20220210102248.365f900e@donnerap.cambridge.arm.com> <3902a00ebedaa54e71af4d10726e195ed557f1db.camel@infinera.com> Organization: Arm Ltd. X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.31; x86_64-slackware-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean On Thu, 10 Feb 2022 21:58:30 +0000 Joakim Tjernlund wrote: Hi, > On Thu, 2022-02-10 at 10:22 +0000, Andre Przywara wrote: > > On Wed, 9 Feb 2022 12:03:47 +0000 > > Joakim Tjernlund wrote: > > > > Hi, > > > > > On Wed, 2022-02-09 at 10:45 +0000, Andre Przywara wrote: > > > > On Wed, 9 Feb 2022 08:35:04 +0000 > > > > Joakim Tjernlund wrote: > > > > > > > > Hi, > > > > > > > > > On Wed, 2022-02-09 at 00:33 +0000, Andre Przywara wrote: > > > > > > On Tue, 8 Feb 2022 22:05:00 +0000 > > > > > > Joakim Tjernlund wrote: > > > > > > > > > > > > Hi Joakim, > > > > > > > > > > > > > Trying to figure out how I should map the MMU for normal RAM so it acessible > > > > > > > from all ELx security states. > > > > > > > > > > > > ^^^^^^^ > > > > > > > > > > > > This does not make much sense. U-Boot is typically running in one > > > > > > exception level only, and sets up the page table for exactly that EL. > > > > > > Each EL uses a separate translation regime (with some twists for stage > > > > > > 2 EL2 and combined EL1/0, plus VHE). If you map your memory in EL3, then > > > > > > drop to EL2, the EL3 page tables become irrelevant. > > > > > > > > > > > > So in U-Boot we just set up the page tables for the EL we are running > > > > > > in, and leave the paging for the lower exception levels to be set up at > > > > > > the discretion of our payloads (kernels, hypervisors). > > > > > > > > > > > > Please not that *secure* memory is a separate concept, and handled by > > > > > > external hardware, typically using regions, not page tables. > > > > > > > > > > I am a beginner w.r.t ARM and Secure/Non secure so thank you for above. > > > > > > > > > > The problem I have is that I boot a custom SOC into u-boot and when u-boot tries > > > > > to boot linux I get an error exception when u-boot calls armv8_switch_to_el2 to enter linux. > > > > > > > > So that means that U-Boot runs in EL3, is that the first and only firmware > > > > that you run? I think the EL3 part of U-Boot is not widely used and tested > > > > beyond the very few platforms that use it. > > > > > > Yes, u-boot is first firmware and runs in EL3(ATM, may change once initial bringup is complete) > > > Maybe u-boot then lacks some critical init? Do you have an example of a board in u-boot > > > that starts in EL3(from reset) using an A53 cpu? > > > > As you have probably figured out by now, the whole Layerscape family uses > > that approach. However most other platforms go with Trusted-Firmware as the > > EL3 setup and secure runtime service provider, so the U-Boot EL3 code in > > here is not well tested or looked after. For initial bringup it might be > > OK, but maybe the problems you run into are due to issues in this code. > > > > > > Do you have the exact address that fails? That should be in ELR, it would > > > > be great if you can pinpoint the exact instruction in macro.h that fails. > > > > > > Yes, the address is the first address where kernel is loaded and you can branch there without problems. > > > > You mean if you load the kernel and branch to the entry point, it starts > > running, but crashes as soon as it realises that in runs in EL3? > > > > > It is the eret instruction(last insn in macro armv8_switch_to_el2_m) that fails. > > > > Interesting. Maybe there is something missing in the EL2 setup, but my > > understanding is that this is the part that is actually used by > > Layerscape, for instance. > > > > > > > I think the exception means "Instruction Abort taken without a change in Exception level." > > > > > I was thinking it could be some privilege missing in MMU map. > > > > > > > > Could be. One thing that made me wonder is your rather miserly mapping of > > > > only 32MB, which sounds a bit on the small side. Typically we just map the > > > > > > We only have 32 MB ATM :( a bit small but it may increase to 64MB > > > > That sounds very miserly. Can you actually run an arm64 Linux kernel with > > that little RAM? IIRC for QEMU we need at least 128 MB, and I haven't seen > > an ARMv8 hardware platform with less than 512MB (maybe 256MB) DRAM yet. > > > > > > whole first DRAM bank, regardless of whether you actually have memory > > > > there or not. U-Boot should know how much DRAM you have, so will not go > > > > beyond that. Having page tables covering more address space does not > > > > really hurt, but avoids all kind of problems. > > > > And please note that U-Boot loves to move things around: itself from the > > > > load address to the end of DRAM (that it knows of); possibly the kernel, > > > > when the alignment is not right, or the DT and initrd if it sees fit. > > > > So there is little point in mapping just portions of the memory. > > > > > > U-boot moves around a lot, I know :) In this case u-boot lives > > > in is own 4MB SRAM but kernel lives in a 32MB HyperRAM. > > > > Interesting. I wonder if this works well with U-Boot's memory management, > > which assumes it has quite some DRAM to play with. > > Found it, all memory spaces were set to secure mode, the req. spec does not agree :( Ah, yes, if the DRAM is configured as secure only, running in EL2 (always non-secure on the A53) will not end well. > Anyhow, now kernel enters into EL2 then EL1 to EL0, all is well until kernel tries > to do simple cache ops like dc ivac, x0 or mrs x3,ctr_el0 when I again just get an error exception: > EXC [0x400] Synchronous Lower EL using AArch64 Was this with Linux, or some other kernel? IIRC cache maintenance instructions in EL0 need to be enabled in SCTLR_EL1 (.UCI and .DZE, for instance, plus maybe more registers), and those and other operations should not be trapped to EL2 as well. > so there is something missing here but I cannot figure out what. > u-boot really don't know about the 32 MB RAM kernel is using, But you did map this RAM, and probably loaded the kernel in there? So U-Boot would need to touch this memory. Maybe it left something behind? PSCI handlers? UEFI runtime services? > it booted just its own 4MB > and I guessing u-boot then missed to configure something for the 32 MB RAM but what ? I really don't understand enough of your setup to be of help here. Cheers, Andre