From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7CC71C433F5 for ; Thu, 19 May 2022 11:38:39 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id DC92283BDB; Thu, 19 May 2022 13:38:28 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="tW4obSAk"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 20F9682105; Thu, 19 May 2022 10:17:22 +0200 (CEST) Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 2659783BDB for ; Thu, 19 May 2022 10:17:17 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=lee.jones@linaro.org Received: by mail-wm1-x32e.google.com with SMTP id k126so2438205wme.2 for ; Thu, 19 May 2022 01:17:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=PY3yEWezghsc9BVKAayrOuRVez45pfCQmG85ye1K2UU=; b=tW4obSAky1ewXrLKuzY4mlu2bz92OOAHlG2V3q+W5agxJOfYd0Q2UH6KEOICZ8j8AU gfx0NdciZ11DMkbA5FKeYMAkC+zGLgjSZsV5dZxZCw5kOlEkbx0g/h2ryXCF0kspfRCI ag7T8Z9p+Vd3bapHLFCUyUISj3TY5NbK9SzpQumTjHm6+fI0Irm7zhX8R6CYxR0cQRsA a00WF5WxwO3E1DvYCqb8zd/Y2AOkqTWVMxIJmuY9xS8eat8bGFIQKcPCCG9UoacaBAKm TiwCXbXKTdLDy8UcYDpXGT/UgNbGDjiHvSYiTy63msuAc/Lc71lYrIP6Hi1HmMjx/J+L Aqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=PY3yEWezghsc9BVKAayrOuRVez45pfCQmG85ye1K2UU=; b=vxf15bSbokYQgKWUJw368ZYELKyiyELu4Vub2ccwyIML6TDNh5efbjiJhUOAc+Pq2V I/OFzneanxxumGUqQtnAbedgfchmTGpFZylHedVfnnI5yGx5ZjuDWs1y0+VQOf/nQw9i MwmspGbG1NP9df+psE6OZryWyBVd+bk2Aa+wrN9WKFJ4T5TAKLQUsShOStPyu9SGQQ5S hw1D0A4p7QyUzmQjNvi9ygGWPyrlxldptmW53IpQWPdRF3oUato3o9qpC2ei0z9iITw8 V1a1GC8JBh8IP4HSG9rMJyeuRxRrtMiUYi0uuNjxfaiXPsCP9eDlYBoKk3rgdvTj2wYw /pVQ== X-Gm-Message-State: AOAM532Qq8sdljKCiWw0lZE2bg8GrGYTXQt9ouXeBwspIJZb57PFUmqN fhCf2roTP6CdDueOARfGckE/Jg== X-Google-Smtp-Source: ABdhPJy7+yiTaI0B9Mm3q+yQIRKOsLD6jBHRbS/CcJfFhXF2MyqH61oOryX7MJrMnK8Hsp7suKQtpQ== X-Received: by 2002:a05:600c:a42:b0:393:d831:bf05 with SMTP id c2-20020a05600c0a4200b00393d831bf05mr2561045wmq.187.1652948236658; Thu, 19 May 2022 01:17:16 -0700 (PDT) Received: from google.com (cpc155339-bagu17-2-0-cust87.1-3.cable.virginm.net. [86.27.177.88]) by smtp.gmail.com with ESMTPSA id l16-20020a05600c1d1000b0039444973258sm8013830wms.0.2022.05.19.01.17.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 01:17:16 -0700 (PDT) Date: Thu, 19 May 2022 09:17:14 +0100 From: Lee Jones To: Peter Geis Cc: Hugh Cole-Baker , U-Boot Mailing List , Caesar Wang , Vasily Khoruzhick , Kever Yang , Heiko Stuebner , Elaine Zhang , Brian Norris , "open list:ARM/Rockchip SoC..." , arm-mail-list Subject: Re: [U-Boot] [PATCH 2/2] rockchip: rk3399: rockpro64: enable force power on reset workaround Message-ID: References: <20191128061433.1952869-1-anarsoul@gmail.com> <20191128061433.1952869-2-anarsoul@gmail.com> <1ab40490-a232-b5fd-6180-17867aa6a053@rock-chips.com> <6C83A835-37D3-4E48-BB08-024BB2703FB8@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Mailman-Approved-At: Thu, 19 May 2022 13:38:22 +0200 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean On Wed, 18 May 2022, Peter Geis wrote: > On Wed, May 18, 2022 at 7:56 AM Lee Jones wrote: > > > > Looping int a few relevant/active kernel people/lists for full coverage. > > > > On Sun, 01 Dec 2019, Hugh Cole-Baker wrote: > > > > On 29 Nov 2019, at 01:06, Vasily Khoruzhick wrote: > > > > On Thu, Nov 28, 2019 at 4:59 PM Kever Yang wrote: > > > >> > > > >> Hi Vasily, > > > >> > > > >> On 2019/11/28 下午11:51, Vasily Khoruzhick wrote: > > > >>> On Thu, Nov 28, 2019 at 1:23 AM Kever Yang wrote: > > > >>>> Hi Vasily, > > > >>>> > > > >>>> I think this should not be needed, see comments below. > > > >>> Hi Kever, > > > >>> > > > >>> I've spent 2 weeks of my evenings debugging this issue but > > > >> > > > >> I can understand you work pretty hard on make it work, it's not so easy > > > >> to identify the root cause > > > >> > > > >> some times, thanks very much for working on this. > > > >> > > > >>> unfortunately I don't have a proper fix. This is the only solution > > > >>> that makes my rockpro64 reboot reliably with mainline u-boot and ATF. > > > >>> See my comments below. > > > > > > I also had a problem where Linux would hang or panic after rebooting, with > > > mainline u-boot and ATF on a rockpro64. This patch does fix the issue for me, > > > I have tested it by performing 10 reboots from Linux in a row and I've seen > > > no hangs or panics. > > > > > > I noticed the Armbian project have recently included a patch to ATF [1] which > > > switches all power domains on before ATF performs a soft reset. I have also > > > tested using u-boot mainline, without any patches to u-boot, but including ATF > > > patched with your reset fix [2] and the Armbian power domains patch [1]. This > > > also fixes the same hanging on reboot issue for me without modifications to > > > u-boot, I've also tested 10 reboots in a row with this ATF and seen no hangs. > > > > > > So this u-boot patch may not be needed if ATF is patched instead to switch > > > power domains on before soft reset. > > > > > > FWIW, when I was able to see panic messages from Linux when it panicked on > > > boot, the call trace always seemed to include rockchip_pd_power_off() [3]. > > > > > > [1] https://github.com/armbian/build/blob/master/patch/atf/atf-rk3399/switch-power-domains-on-before-reset.patch > > > [2] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/2512 > > > [3] https://gist.github.com/sigmaris/c0e155c8cb0a325d84f549185f9a568c > > > > This last paste looks remarkably similar to an issue currently seen on > > the Radxa ROCK Pi 4B (RK3399) during power-up after a soft reboot > > (`sudo reboot`) is issued. We're presently running v5.15.35 [0]. > > Good Evening, Hi, Peter, Thank you so much for your reply. > That's definitely not stock v5.15.35, it's been tagged as an android kernel. > 5.15.35-android13-5-00092-g525d77310a20 It's not stock, no. Although the differences from RockPi's perspective are minimal. The main difference is the way the kernel is configured. It's GKI: https://android.googlesource.com/kernel/common/+/refs/heads/android13-5.15/arch/arm64/configs/gki_defconfig Plus a few non-GKI specifics: https://android.googlesource.com/kernel/common/+/refs/heads/android13-5.15/arch/arm64/configs/rockpi4_gki.fragment > > It's not clear how this issue (present 3 years ago) was finally > > resolved. From the thread, it looks as if the fix might have made its > > way into ATF, but I'm 87.6% sure ATF is not running on this platform > > (yet). > > The rk3399 SoC has a hardware bug where the power domains are not > reset upon a soft reset. This leads to situations like this one where > power domains are shut down during shutdown but aren't restored on > reboot. I assume this isn't something we can patch in the kernel driver? > Mainline TF-A was patched to force all power domains online > when a soft reboot is triggered, which solved that issue. Okay, this is what I figured. > What particular issues are you having initializing modern u-boot on > this device? This is the output: https://pastebin.ubuntu.com/p/d5DmsSBnrR/ Speaking with one of the guys who supports RockPi 4 in AOSP, he suspects the DDR settings. Apparently settings for older SoCs sometimes get clobbered when support for newer SoCs is added. I am yet to investigate the u-boot story in any detail. It's on my TODO list for today. > Is there a particular reason it isn't using Mainline TF-A? We're not using Trusted Firmware yet. Although I'm starting to think this should be re-prioritised. > I've also run into issues on rk356x where the regulator powering a > power domain isn't powered due to a soft reset, which also causes > faults like this. Set your main regulators to always-on and see if it > helps with the issue. I'll do that. Thanks for the tip. Our main issue currently is an RCU-lock-up, again on soft reboot: [ 21.226951][ C0] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 21.227637][ C0] rcu: 5-...!: (1 GPs behind) idle=3de/1/0x4000000000000000 softirq=9/10 fqs=3 last_accelerate: 0000/efb9 dyntick_enabled: 0 [ 21.228890][ C0] (detected by 0, t=5252 jiffies, g=-1167, q=46) Do you think these issues could all be related? Thanks ever so much for your reply Peter. You've potentially saved us hours and hours of debugging. Kind regards, Lee > > Note that the u-boot we're using is also quite old: > > > > U-Boot 2019.10-09248-g8511c75bb4 (Jan 08 2020 - 17:13:03 -0800) > > > > ... so this could easily be the root cause. The current plan is to > > try to update this ASAP. However early attempts are yet to result in > > a successful boot. > > > > I see that Brian recently added a few patches related to PD/DVFS, but > > again, these appear to be ATF related. > > > > Would anyone be able to shed some light onto this for me please? > > > > As always, any help would be gratefully received. > > > > Kind regards, > > Lee > > > > [0] > > Full reboot log can be seen at: https://pastebin.ubuntu.com/p/MjZP2V6kQ3/ > > > > [ 0.699736][ T1] initcall __initstub__kmod_iommu__362_155_iommu_subsys_init4+0x0/0x8 returned 0 after 0 usecs > > [ 0.700737][ T1] calling __initstub__kmod_rockchip_iommu__348_1415_rk_iommu_init4+0x0/0x8 @ 1 > > [ 0.702238][ C5] SError Interrupt on CPU5, code 0xbf000002 -- SError > > [ 0.702248][ C5] CPU: 5 PID: 48 Comm: kworker/5:1 Not tainted 5.15.35-android13-5-00092-g525d77310a20 #1 > > [ 0.702261][ C5] Hardware name: Radxa ROCK Pi 4B (DT) > > [ 0.702266][ C5] Workqueue: pm genpd_power_off_work_fn.cfi_jt > > [ 0.702289][ C5] pstate: 804000c5 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > [ 0.702301][ C5] pc : regmap_mmio_read32le+0x14/0x2c > > [ 0.702318][ C5] lr : regmap_mmio_read+0x68/0xd0 > > [ 0.702331][ C5] sp : ffffffc00b6d3b40 > > [ 0.702335][ C5] x29: ffffffc00b6d3b40 x28: 0000000000000000 x27: 0000000000000000 > > [ 0.702351][ C5] x26: ffffff8000923680 x25: ffffffc009abc2a0 x24: ffffff8000930c00 > > [ 0.702364][ C5] x23: 0000000000000014 x22: ffffff8000930c00 x21: 0000000000000008 > > [ 0.702378][ C5] x20: ffffff8000922300 x19: ffffff8000923680 x18: ffffffc00b66d058 > > [ 0.702391][ C5] x17: 000000000000ba7e x16: ffffffc00a4dee04 x15: 000000000000b67e > > [ 0.702405][ C5] x14: 00000000028dd7a0 x13: 0000000000000040 x12: 0000000000000000 > > [ 0.702419][ C5] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000005 > > [ 0.702432][ C5] x8 : 0000000000000000 x7 : 00756d6d6f692e30 x6 : 3030383035366666 > > [ 0.702445][ C5] x5 : 0000000000000001 x4 : 028dea248fba33d6 x3 : 0000000000000000 > > [ 0.702457][ C5] x2 : ffffff8000923680 x1 : 0000000000000008 x0 : 0000000000000000 > > [ 0.702472][ C5] Kernel panic - not syncing: Asynchronous SError Interrupt > > [ 0.702477][ C5] CPU: 5 PID: 48 Comm: kworker/5:1 Not tainted 5.15.35-android13-5-00092-g525d77310a20 #1 > > [ 0.702487][ C5] Hardware name: Radxa ROCK Pi 4B (DT) > > [ 0.702492][ C5] Workqueue: pm genpd_power_off_work_fn.cfi_jt > > [ 0.702506][ C5] Call trace: > > [ 0.702508][ C5] dump_backtrace.cfi_jt+0x0/0x8 > > [ 0.702525][ C5] dump_stack_lvl+0x80/0xb8 > > [ 0.702536][ C5] panic+0x180/0x444 > > [ 0.702547][ C5] arm64_serror_panic+0x1c0/0x210 > > [ 0.702561][ C5] do_serror+0x17c/0x218 > > [ 0.702572][ C5] el1h_64_error_handler+0x38/0x50 > > [ 0.702581][ C5] el1h_64_error+0x7c/0x80 > > [ 0.702589][ C5] regmap_mmio_read32le+0x14/0x2c > > [ 0.702603][ C5] _regmap_bus_reg_read+0x3c/0x90 > > [ 0.702614][ C5] _regmap_read+0xb0/0x24c > > [ 0.702623][ C5] rockchip_pd_power+0x6c4/0xbc0 > > [ 0.702638][ C5] rockchip_pd_power_off+0x18/0x28 > > [ 0.702652][ C5] _genpd_power_off+0x178/0x388 > > [ 0.702663][ C5] genpd_power_off+0x188/0x2e4 > > [ 0.702673][ C5] genpd_power_off_work_fn+0x54/0xe4 > > [ 0.702683][ C5] process_one_work+0x254/0x5a0 > > [ 0.702696][ C5] worker_thread+0x3ec/0x920 > > [ 0.702707][ C5] kthread+0x168/0x1dc > > [ 0.702716][ C5] ret_from_fork+0x10/0x20 > > [ 0.702726][ C5] SMP: stopping secondary CPUs > > > > > > _______________________________________________ > > Linux-rockchip mailing list > > Linux-rockchip@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-rockchip -- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog