Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Francesco Dolcini]

Hello,

On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> Hello,
> it seems that an old regression is back on v6.12, reproduced on -rc2
> (not sure about rc1).
> 
> The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> 
> Is it now back?

I was able to reproduce this issue once more, this time with 6.11.7.
As I wrote in another email the issue is not systematic as it used to
be.

Any idea?

[    4.287586] SError Interrupt on CPU1, code 0x00000000bf000002 -- SError
[    4.287594] CPU: 1 UID: 0 PID: 195 Comm: (udev-worker) Not tainted 6.11.7-7.1.0-devel-00010-g5374bda92db5 #1
[    4.287601] Hardware name: Toradex Verdin iMX8M Plus WB on Dahlia Board (DT)
[    4.287604] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    4.287609] pc : clk_imx8mp_audiomix_runtime_resume+0x18/0x40
[    4.287623] lr : pm_generic_runtime_resume+0x2c/0x44
[    4.287631] sp : ffff80008216b740
[    4.287633] x29: ffff80008216b740 x28: ffff800080e85010 x27: ffff8000794758d8
[    4.287642] x26: ffff0000c0a664f4 x25: 0000000000000000 x24: 0000000000000000
[    4.287649] x23: 0000000000000000 x22: ffff0000c0d01120 x21: 0000000000000000
[    4.287655] x20: ffff0000c1308800 x19: ffff0000c0a68c10 x18: 0000000000047820
[    4.287662] x17: 0000000000000000 x16: ffff0000faf87820 x15: 0000000000000002
[    4.287668] x14: 0000000000001400 x13: 00000000000003a0 x12: 0000000054a32c00
[    4.287675] x11: 0000000000000000 x10: ffff8000817b9d36 x9 : 0000000000000008
[    4.287682] x8 : 0000000000000008 x7 : 0000000000000000 x6 : 0000000000000000
[    4.287688] x5 : ffff800081ec0000 x4 : ffff0000c1a0b488 x3 : ffff800080bd23f0
[    4.287694] x2 : 0000000000000000 x1 : 0000000000000004 x0 : ffff800081ec0300
[    4.287703] Kernel panic - not syncing: Asynchronous SError Interrupt
[    4.287705] CPU: 1 UID: 0 PID: 195 Comm: (udev-worker) Not tainted 6.11.7-7.1.0-devel-00010-g5374bda92db5 #1
[    4.287711] Hardware name: Toradex Verdin iMX8M Plus WB on Dahlia Board (DT)
[    4.287714] Call trace:
[    4.287716]  dump_backtrace+0x94/0x114
[    4.287725]  show_stack+0x18/0x24
[    4.287732]  dump_stack_lvl+0x34/0x8c
[    4.287740]  dump_stack+0x18/0x24
[    4.287747]  panic+0x390/0x3a4
[    4.287751]  nmi_panic+0x40/0x8c
[    4.287755]  arm64_serror_panic+0x64/0x70
[    4.287759]  do_serror+0x3c/0x70
[    4.287763]  el1h_64_error_handler+0x30/0x48
[    4.287771]  el1h_64_error+0x64/0x68
[    4.287776]  clk_imx8mp_audiomix_runtime_resume+0x18/0x40
[    4.287784]  __genpd_runtime_resume+0x30/0x80
[    4.287791]  genpd_runtime_resume+0x114/0x29c
[    4.287799]  __rpm_callback+0x48/0x1d8
[    4.287806]  rpm_callback+0x68/0x74
[    4.287813]  rpm_resume+0x46c/0x6bc
[    4.287819]  __pm_runtime_resume+0x50/0x94
[    4.287825]  pm_runtime_get_suppliers+0x60/0x8c
[    4.287831]  __driver_probe_device+0x4c/0x15c
[    4.287838]  driver_probe_device+0x3c/0x110
[    4.287844]  __driver_attach+0xf0/0x1f8
[    4.287850]  bus_for_each_dev+0x7c/0xdc
[    4.287856]  driver_attach+0x24/0x30
[    4.287862]  bus_add_driver+0x110/0x234
[    4.287868]  driver_register+0x5c/0x124
[    4.287874]  __platform_driver_register+0x28/0x34
[    4.287882]  sdma_driver_init+0x20/0x1000 [imx_sdma]
[    4.287895]  do_one_initcall+0x80/0x1c8
[    4.287900]  do_init_module+0x60/0x218
[    4.287907]  load_module+0x1fcc/0x2040
[    4.287912]  init_module_from_file+0x88/0xcc
[    4.287918]  __arm64_sys_finit_module+0x254/0x34c
[    4.287924]  invoke_syscall.constprop.0+0x50/0xe4
[    4.287932]  do_el0_svc+0xa8/0xc8
[    4.287938]  el0_svc+0x3c/0x13c
[    4.287945]  el0t_64_sync_handler+0x120/0x12c
[    4.287952]  el0t_64_sync+0x190/0x194
[    4.287958] SMP: stopping secondary CPUs
[    4.287965] Kernel Offset: disabled
[    4.287966] CPU features: 0x00,00000000,00100000,4200420b
[    4.287970] Memory Limit: none
[    4.590829] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

Francesco

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Francesco Dolcini]

On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > it seems that an old regression is back on v6.12, reproduced on -rc2
> > (not sure about rc1).
> > 
> > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > 
> > Is it now back?
> 
> I was able to reproduce this issue once more, this time with 6.11.7.
> As I wrote in another email the issue is not systematic as it used to
> be.
> 
> Any idea?

Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
too short and therefore we have such non-systematic failures?

Francesco

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Shengjiu Wang]

On Tue, Nov 12, 2024 at 5:20 PM Francesco Dolcini <francesco@dolcini.it> wrote:
>
> On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> > On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > > it seems that an old regression is back on v6.12, reproduced on -rc2
> > > (not sure about rc1).
> > >
> > > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > >
> > > Is it now back?
> >
> > I was able to reproduce this issue once more, this time with 6.11.7.
> > As I wrote in another email the issue is not systematic as it used to
> > be.
> >
> > Any idea?
>
> Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
> too short and therefore we have such non-systematic failures?
>

Francesco,  it seems hard to reproduce it on my i.MX8MP-EVK board.

If it is easy to reproduce on your side, you can try to enlarge the delay
time to see if there is any improvement.

Thanks.

Best regards
Shengjiu Wang

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Francesco Dolcini]

On Fri, Nov 15, 2024 at 11:13:58AM +0800, Shengjiu Wang wrote:
> On Tue, Nov 12, 2024 at 5:20 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> >
> > On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> > > On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > > > it seems that an old regression is back on v6.12, reproduced on -rc2
> > > > (not sure about rc1).
> > > >
> > > > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > > > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > > >
> > > > Is it now back?
> > >
> > > I was able to reproduce this issue once more, this time with 6.11.7.
> > > As I wrote in another email the issue is not systematic as it used to
> > > be.
> > >
> > > Any idea?
> >
> > Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
> > too short and therefore we have such non-systematic failures?
> >
> 
> Francesco,  it seems hard to reproduce it on my i.MX8MP-EVK board.
> 
> If it is easy to reproduce on your side, you can try to enlarge the delay
> time to see if there is any improvement.

It's hard also for me to reproduce, we just have a relatively extensive
test farm and 2 times it happened while doing unrelated tests. I was hoping we
could have some idea on what's going on, I'll see if I can put together some
kind of stress test, being able to reproduce it more systematically would
certainly help.

Francesco

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Shengjiu Wang]

On Fri, Nov 15, 2024 at 10:47 PM Francesco Dolcini <francesco@dolcini.it> wrote:
>
> On Fri, Nov 15, 2024 at 11:13:58AM +0800, Shengjiu Wang wrote:
> > On Tue, Nov 12, 2024 at 5:20 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> > >
> > > On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> > > > On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > > > > it seems that an old regression is back on v6.12, reproduced on -rc2
> > > > > (not sure about rc1).
> > > > >
> > > > > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > > > > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > > > >
> > > > > Is it now back?
> > > >
> > > > I was able to reproduce this issue once more, this time with 6.11.7.
> > > > As I wrote in another email the issue is not systematic as it used to
> > > > be.
> > > >
> > > > Any idea?
> > >
> > > Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
> > > too short and therefore we have such non-systematic failures?
> > >
> >
> > Francesco,  it seems hard to reproduce it on my i.MX8MP-EVK board.
> >
> > If it is easy to reproduce on your side, you can try to enlarge the delay
> > time to see if there is any improvement.
>
> It's hard also for me to reproduce, we just have a relatively extensive
> test farm and 2 times it happened while doing unrelated tests. I was hoping we
> could have some idea on what's going on, I'll see if I can put together some
> kind of stress test, being able to reproduce it more systematically would
> certainly help.
>

With my test,  the issue reproduced with delay 5us/6us. but hard reproduced
with 7us.
I think we may need to use a delay of 10us for safety.

Best regards
Shengjiu Wang

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Francesco Dolcini]

Hello Shengjiu,

On Tue, Nov 19, 2024 at 05:35:36PM +0800, Shengjiu Wang wrote:
> On Fri, Nov 15, 2024 at 10:47 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> >
> > On Fri, Nov 15, 2024 at 11:13:58AM +0800, Shengjiu Wang wrote:
> > > On Tue, Nov 12, 2024 at 5:20 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> > > >
> > > > On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> > > > > On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > > > > > it seems that an old regression is back on v6.12, reproduced on -rc2
> > > > > > (not sure about rc1).
> > > > > >
> > > > > > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > > > > > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > > > > >
> > > > > > Is it now back?
> > > > >
> > > > > I was able to reproduce this issue once more, this time with 6.11.7.
> > > > > As I wrote in another email the issue is not systematic as it used to
> > > > > be.
> > > > >
> > > > > Any idea?
> > > >
> > > > Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
> > > > too short and therefore we have such non-systematic failures?
> > > >
> > >
> > > Francesco,  it seems hard to reproduce it on my i.MX8MP-EVK board.
> > >
> > > If it is easy to reproduce on your side, you can try to enlarge the delay
> > > time to see if there is any improvement.
> >
> > It's hard also for me to reproduce, we just have a relatively extensive
> > test farm and 2 times it happened while doing unrelated tests. I was hoping we
> > could have some idea on what's going on, I'll see if I can put together some
> > kind of stress test, being able to reproduce it more systematically would
> > certainly help.
> >
> 
> With my test,  the issue reproduced with delay 5us/6us. but hard reproduced
> with 7us.
> I think we may need to use a delay of 10us for safety.

Great that you were able to narrow this down and confirm the issue.
I wonder if you would have any information on what is the actual delay
that the HW would need, instead of guessing numbers. If not, well, let's
go with 15usec, or 10usec, your call in the end.

Will you send a patch?

Francesco

Re: clk_imx8mp_audiomix_runtime_resume Kernel panic regression on v6.12

[Adam Ford]

On Tue, Nov 19, 2024 at 5:22 AM Francesco Dolcini <francesco@dolcini.it> wrote:
>
> Hello Shengjiu,
>
> On Tue, Nov 19, 2024 at 05:35:36PM +0800, Shengjiu Wang wrote:
> > On Fri, Nov 15, 2024 at 10:47 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> > >
> > > On Fri, Nov 15, 2024 at 11:13:58AM +0800, Shengjiu Wang wrote:
> > > > On Tue, Nov 12, 2024 at 5:20 PM Francesco Dolcini <francesco@dolcini.it> wrote:
> > > > >
> > > > > On Tue, Nov 12, 2024 at 08:59:58AM +0100, Francesco Dolcini wrote:
> > > > > > On Mon, Oct 07, 2024 at 03:25:55PM +0200, Francesco Dolcini wrote:
> > > > > > > it seems that an old regression is back on v6.12, reproduced on -rc2
> > > > > > > (not sure about rc1).
> > > > > > >
> > > > > > > The original report is from https://lore.kernel.org/all/20240424164725.GA18760@francesco-nb/
> > > > > > > and it was fixed with https://lore.kernel.org/all/1715396125-3724-1-git-send-email-shengjiu.wang@nxp.com/.
> > > > > > >
> > > > > > > Is it now back?
> > > > > >
> > > > > > I was able to reproduce this issue once more, this time with 6.11.7.
> > > > > > As I wrote in another email the issue is not systematic as it used to
> > > > > > be.
> > > > > >
> > > > > > Any idea?
> > > > >
> > > > > Frank, Shengjiu, could it be that the udelay(5) in imx_pgc_power_up() is
> > > > > too short and therefore we have such non-systematic failures?
> > > > >
> > > >
> > > > Francesco,  it seems hard to reproduce it on my i.MX8MP-EVK board.
> > > >
> > > > If it is easy to reproduce on your side, you can try to enlarge the delay
> > > > time to see if there is any improvement.
> > >
> > > It's hard also for me to reproduce, we just have a relatively extensive
> > > test farm and 2 times it happened while doing unrelated tests. I was hoping we
> > > could have some idea on what's going on, I'll see if I can put together some
> > > kind of stress test, being able to reproduce it more systematically would
> > > certainly help.
> > >
> >
> > With my test,  the issue reproduced with delay 5us/6us. but hard reproduced
> > with 7us.
> > I think we may need to use a delay of 10us for safety.
>
> Great that you were able to narrow this down and confirm the issue.
> I wonder if you would have any information on what is the actual delay
> that the HW would need, instead of guessing numbers. If not, well, let's
> go with 15usec, or 10usec, your call in the end.
>
> Will you send a patch?

Peng (or anyone from NXP),

Do any of the clock speeds have an impact on the propagation time?
If the clocks are running at overdrive speeds vs nominal speeds vs
underclocked, could this time be adjusted accordingly somehow?

adam

>
> Francesco
>