From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F02BC433F5 for ; Wed, 18 May 2022 15:31:11 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 4345281E79; Wed, 18 May 2022 17:31:09 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="sT81YHpD"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id C469980F7B; Wed, 18 May 2022 13:55:31 +0200 (CEST) Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 7837083612 for ; Wed, 18 May 2022 13:55:27 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=lee.jones@linaro.org Received: by mail-wm1-x32c.google.com with SMTP id r6-20020a1c2b06000000b00396fee5ebc9so920521wmr.1 for ; Wed, 18 May 2022 04:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=kCCZttYe88nnTP/ci0agOImauJrzNziOVL7F9EsssVg=; b=sT81YHpDXhU45Qa9jKOrgFTm+661R7Y1nhg6pDh+t2Wr8aiR4aqMmCBduGV3L3fPm0 88jraQoLAs0E29e0eRBFkQxguvW29+jUhGJL/koQYDrTz5Ugyq9M7sH3ZlCNXtboAi8a qo71Gvc9NNp/sogT/vUJKGXZUHuVdIDZf0UQGhE81q2yb9MityIGnGLrhyB6c9dB3PFN GIpnMk+nZD3AHd/qOzumMpt2dZuOMGbRlPARzRBxjzlOWRV7WUdfO4ZBfi7J86h45TXV iEcSDYXMTfOl7ZLWyviT45Gvt6Tuxqf4dRMt03S5mtVAJxmsOLOa/MTbmSUft/pOWjMJ d+lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=kCCZttYe88nnTP/ci0agOImauJrzNziOVL7F9EsssVg=; b=Lkqd/8dhbDaHiwG0O4dz8yDfKIyL6KBOpfKANBfogfEQMDX5Ph3qtZyR/K8rkWq9x1 2AwtHF2RB+SOAj3J983yyUBslL34Tt9YIpxco2TX8slza1AGsVdPwGyJnM5BKWRtuf+i uOZE1mAUHzsueB4e35tr4LVOxWDLCF6TwotvxoJTHuMN3THXxaw7pK4+W0EhpYhhKyrf 0HS44NDAFnIMfoWGwcKXnfN2p6s5fsuATJsdPLhryQu+3okTGQsR2ITGidSfzJhgxYFr p7HoAZOKI2mHVXoVXNLKGsBN51HiZyCC7ZRRHs0XlYIcAvK5XrYARgM5DA0BX87ptF1R 0xwA== X-Gm-Message-State: AOAM5306sxazAgLaSo6CS+u/YDzY/K6A47TsVPLrHZCFamOnz0DJQbr9 L+JxzoYXVQvUZ9XX+EcuxLM5Lw== X-Google-Smtp-Source: ABdhPJy9D5teJUnV427nOduKPpgIBBeSHfMiOqPMa95j/tODu6xyxM4Ii2mxk1m/uDR/coamLeL2ww== X-Received: by 2002:a7b:c156:0:b0:395:b669:5c83 with SMTP id z22-20020a7bc156000000b00395b6695c83mr34907779wmi.141.1652874926783; Wed, 18 May 2022 04:55:26 -0700 (PDT) Received: from google.com (cpc155339-bagu17-2-0-cust87.1-3.cable.virginm.net. [86.27.177.88]) by smtp.gmail.com with ESMTPSA id ay34-20020a05600c1e2200b003942a244f54sm5101071wmb.45.2022.05.18.04.55.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 May 2022 04:55:26 -0700 (PDT) Date: Wed, 18 May 2022 12:55:24 +0100 From: Lee Jones To: Hugh Cole-Baker Cc: u-boot@lists.denx.de, Caesar Wang , Vasily Khoruzhick , Kever Yang , Hugh Cole-Baker , Heiko Stuebner , Elaine Zhang , Brian Norris , linux-rockchip@lists.infradead.org, linux-arm-kernel@lists.infradead.org Subject: Re: [U-Boot] [PATCH 2/2] rockchip: rk3399: rockpro64: enable force power on reset workaround Message-ID: References: <20191128061433.1952869-1-anarsoul@gmail.com> <20191128061433.1952869-2-anarsoul@gmail.com> <1ab40490-a232-b5fd-6180-17867aa6a053@rock-chips.com> <6C83A835-37D3-4E48-BB08-024BB2703FB8@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6C83A835-37D3-4E48-BB08-024BB2703FB8@gmail.com> X-Mailman-Approved-At: Wed, 18 May 2022 17:31:07 +0200 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean Looping int a few relevant/active kernel people/lists for full coverage. On Sun, 01 Dec 2019, Hugh Cole-Baker wrote: > > On 29 Nov 2019, at 01:06, Vasily Khoruzhick wrote: > > On Thu, Nov 28, 2019 at 4:59 PM Kever Yang wrote: > >> > >> Hi Vasily, > >> > >> On 2019/11/28 下午11:51, Vasily Khoruzhick wrote: > >>> On Thu, Nov 28, 2019 at 1:23 AM Kever Yang wrote: > >>>> Hi Vasily, > >>>> > >>>> I think this should not be needed, see comments below. > >>> Hi Kever, > >>> > >>> I've spent 2 weeks of my evenings debugging this issue but > >> > >> I can understand you work pretty hard on make it work, it's not so easy > >> to identify the root cause > >> > >> some times, thanks very much for working on this. > >> > >>> unfortunately I don't have a proper fix. This is the only solution > >>> that makes my rockpro64 reboot reliably with mainline u-boot and ATF. > >>> See my comments below. > > I also had a problem where Linux would hang or panic after rebooting, with > mainline u-boot and ATF on a rockpro64. This patch does fix the issue for me, > I have tested it by performing 10 reboots from Linux in a row and I've seen > no hangs or panics. > > I noticed the Armbian project have recently included a patch to ATF [1] which > switches all power domains on before ATF performs a soft reset. I have also > tested using u-boot mainline, without any patches to u-boot, but including ATF > patched with your reset fix [2] and the Armbian power domains patch [1]. This > also fixes the same hanging on reboot issue for me without modifications to > u-boot, I've also tested 10 reboots in a row with this ATF and seen no hangs. > > So this u-boot patch may not be needed if ATF is patched instead to switch > power domains on before soft reset. > > FWIW, when I was able to see panic messages from Linux when it panicked on > boot, the call trace always seemed to include rockchip_pd_power_off() [3]. > > [1] https://github.com/armbian/build/blob/master/patch/atf/atf-rk3399/switch-power-domains-on-before-reset.patch > [2] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/2512 > [3] https://gist.github.com/sigmaris/c0e155c8cb0a325d84f549185f9a568c This last paste looks remarkably similar to an issue currently seen on the Radxa ROCK Pi 4B (RK3399) during power-up after a soft reboot (`sudo reboot`) is issued. We're presently running v5.15.35 [0]. It's not clear how this issue (present 3 years ago) was finally resolved. From the thread, it looks as if the fix might have made its way into ATF, but I'm 87.6% sure ATF is not running on this platform (yet). Note that the u-boot we're using is also quite old: U-Boot 2019.10-09248-g8511c75bb4 (Jan 08 2020 - 17:13:03 -0800) ... so this could easily be the root cause. The current plan is to try to update this ASAP. However early attempts are yet to result in a successful boot. I see that Brian recently added a few patches related to PD/DVFS, but again, these appear to be ATF related. Would anyone be able to shed some light onto this for me please? As always, any help would be gratefully received. Kind regards, Lee [0] Full reboot log can be seen at: https://pastebin.ubuntu.com/p/MjZP2V6kQ3/ [ 0.699736][ T1] initcall __initstub__kmod_iommu__362_155_iommu_subsys_init4+0x0/0x8 returned 0 after 0 usecs [ 0.700737][ T1] calling __initstub__kmod_rockchip_iommu__348_1415_rk_iommu_init4+0x0/0x8 @ 1 [ 0.702238][ C5] SError Interrupt on CPU5, code 0xbf000002 -- SError [ 0.702248][ C5] CPU: 5 PID: 48 Comm: kworker/5:1 Not tainted 5.15.35-android13-5-00092-g525d77310a20 #1 [ 0.702261][ C5] Hardware name: Radxa ROCK Pi 4B (DT) [ 0.702266][ C5] Workqueue: pm genpd_power_off_work_fn.cfi_jt [ 0.702289][ C5] pstate: 804000c5 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.702301][ C5] pc : regmap_mmio_read32le+0x14/0x2c [ 0.702318][ C5] lr : regmap_mmio_read+0x68/0xd0 [ 0.702331][ C5] sp : ffffffc00b6d3b40 [ 0.702335][ C5] x29: ffffffc00b6d3b40 x28: 0000000000000000 x27: 0000000000000000 [ 0.702351][ C5] x26: ffffff8000923680 x25: ffffffc009abc2a0 x24: ffffff8000930c00 [ 0.702364][ C5] x23: 0000000000000014 x22: ffffff8000930c00 x21: 0000000000000008 [ 0.702378][ C5] x20: ffffff8000922300 x19: ffffff8000923680 x18: ffffffc00b66d058 [ 0.702391][ C5] x17: 000000000000ba7e x16: ffffffc00a4dee04 x15: 000000000000b67e [ 0.702405][ C5] x14: 00000000028dd7a0 x13: 0000000000000040 x12: 0000000000000000 [ 0.702419][ C5] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000005 [ 0.702432][ C5] x8 : 0000000000000000 x7 : 00756d6d6f692e30 x6 : 3030383035366666 [ 0.702445][ C5] x5 : 0000000000000001 x4 : 028dea248fba33d6 x3 : 0000000000000000 [ 0.702457][ C5] x2 : ffffff8000923680 x1 : 0000000000000008 x0 : 0000000000000000 [ 0.702472][ C5] Kernel panic - not syncing: Asynchronous SError Interrupt [ 0.702477][ C5] CPU: 5 PID: 48 Comm: kworker/5:1 Not tainted 5.15.35-android13-5-00092-g525d77310a20 #1 [ 0.702487][ C5] Hardware name: Radxa ROCK Pi 4B (DT) [ 0.702492][ C5] Workqueue: pm genpd_power_off_work_fn.cfi_jt [ 0.702506][ C5] Call trace: [ 0.702508][ C5] dump_backtrace.cfi_jt+0x0/0x8 [ 0.702525][ C5] dump_stack_lvl+0x80/0xb8 [ 0.702536][ C5] panic+0x180/0x444 [ 0.702547][ C5] arm64_serror_panic+0x1c0/0x210 [ 0.702561][ C5] do_serror+0x17c/0x218 [ 0.702572][ C5] el1h_64_error_handler+0x38/0x50 [ 0.702581][ C5] el1h_64_error+0x7c/0x80 [ 0.702589][ C5] regmap_mmio_read32le+0x14/0x2c [ 0.702603][ C5] _regmap_bus_reg_read+0x3c/0x90 [ 0.702614][ C5] _regmap_read+0xb0/0x24c [ 0.702623][ C5] rockchip_pd_power+0x6c4/0xbc0 [ 0.702638][ C5] rockchip_pd_power_off+0x18/0x28 [ 0.702652][ C5] _genpd_power_off+0x178/0x388 [ 0.702663][ C5] genpd_power_off+0x188/0x2e4 [ 0.702673][ C5] genpd_power_off_work_fn+0x54/0xe4 [ 0.702683][ C5] process_one_work+0x254/0x5a0 [ 0.702696][ C5] worker_thread+0x3ec/0x920 [ 0.702707][ C5] kthread+0x168/0x1dc [ 0.702716][ C5] ret_from_fork+0x10/0x20 [ 0.702726][ C5] SMP: stopping secondary CPUs -- Lee Jones [李琼斯] Principal Technical Lead - Developer Services Linaro.org │ Open source software for Arm SoCs Follow Linaro: Facebook | Twitter | Blog