Oops on mcp251xfd open on 6.6.52

Oops on mcp251xfd open on 6.6.52

[Francesco Dolcini]

Hello,
I got the following Oops on doing a simple ip link set canX up, running
on an i.MX8MM SoC.

This was reproduced with some automated testing, I have not tried to
reproduce it so far. Any idea or hint? It seems systematic, introduced
around 12th september on the v6.6.y branch.

[   75.694960] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
[   75.926065] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   75.933032] pc : timecounter_read+0x18/0x70
[   75.937230] lr : mcp251xfd_ring_init+0x1dc/0x944 [mcp251xfd]
[   75.942903] sp : ffffffc08496b460
[   75.946218] x29: ffffffc08496b460 x28: 0000000000000000 x27: ffffff800a7608c0
[   75.953364] x26: 0000000000000001 x25: 0000000000000000 x24: ffffff800c50a000
[   75.960504] x23: ffffff800a7628c0 x22: 0000000000000000 x21: 0000000000000460
[   75.967648] x20: ffffff800a760e00 x19: ffffff800a763830 x18: 0000000000000000
[   75.974796] x17: 000000040044ffff x16: 00500072b5593519 x15: 0000000000000000
[   75.981942] x14: 000000000003ecee x13: 0000000000000299 x12: 0000000000000002
[   75.989083] x11: 0000000000000000 x10: 0000000000000950 x9 : ffffffc08496aef0
[   75.996224] x8 : ffffff800a760de0 x7 : 0000000000000000 x6 : 00000000256ab2b4
[   76.003365] x5 : 0000000000000001 x4 : 0000000000002f70 x3 : 0000000000000004
[   76.010508] x2 : 0000000000000010 x1 : ffffff800a761680 x0 : 0000000000000000
[   76.017652] Call trace:
[   76.020099]  timecounter_read+0x18/0x70
[   76.023942]  mcp251xfd_ring_init+0x1dc/0x944 [mcp251xfd]
[   76.029263]  mcp251xfd_chip_start+0x228/0x28c [mcp251xfd]
[   76.034669]  mcp251xfd_open+0x8c/0x298 [mcp251xfd]
[   76.039468]  __dev_open+0x100/0x1d8
[   76.042967]  __dev_change_flags+0x194/0x20c
[   76.047158]  dev_change_flags+0x24/0x6c
[   76.051002]  do_setlink+0x27c/0xeb0
[   76.054495]  __rtnl_newlink+0x520/0x88c
[   76.058333]  rtnl_newlink+0x50/0x7c
[   76.061824]  rtnetlink_rcv_msg+0x128/0x378
[   76.065922]  netlink_rcv_skb+0x60/0x130
[   76.069764]  rtnetlink_rcv+0x18/0x24
[   76.073342]  netlink_unicast+0x300/0x36c
[   76.077268]  netlink_sendmsg+0x194/0x400
[   76.081194]  ____sys_sendmsg+0x214/0x248
[   76.085120]  ___sys_sendmsg+0xb0/0xfc
[   76.088788]  __sys_sendmsg+0x78/0xd0
[   76.092370]  __arm64_sys_sendmsg+0x24/0x30
[   76.096470]  invoke_syscall.constprop.0+0x50/0xec
[   76.101179]  do_el0_svc+0x40/0xc8
[   76.104497]  el0_svc+0x48/0x148
[   76.107643]  el0t_64_sync_handler+0x120/0x12c
[   76.112002]  el0t_64_sync+0x190/0x194
[   76.115677] Code: 910003fd f9000bf3 aa0003f3 f9400000 (f9400001)
[   76.121781] ---[ end trace 0000000000000000 ]---

Francesco

Re: Oops on mcp251xfd open on 6.6.52

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

On 23.09.2024 13:53:10, Francesco Dolcini wrote:
> I got the following Oops on doing a simple ip link set canX up, running
> on an i.MX8MM SoC.
> 
> This was reproduced with some automated testing, I have not tried to
> reproduce it so far. Any idea or hint? It seems systematic, introduced
> around 12th september on the v6.6.y branch.

Which Kernel version are you using?

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Oops on mcp251xfd open on 6.6.52

[Francesco Dolcini]

Hello Marc,

On Mon, Sep 23, 2024 at 01:57:07PM +0200, Marc Kleine-Budde wrote:
> On 23.09.2024 13:53:10, Francesco Dolcini wrote:
> > I got the following Oops on doing a simple ip link set canX up, running
> > on an i.MX8MM SoC.
> > 
> > This was reproduced with some automated testing, I have not tried to
> > reproduce it so far. Any idea or hint? It seems systematic, introduced
> > around 12th september on the v6.6.y branch.
> 
> Which Kernel version are you using?

v6.6.52 is reproducing the issue, I guess it started on some previous
v6.6.x kernel (with x < 52), but I was not able to dig out the logs
running on older kernel from our automated testing (yet).

I was not able to test mainline (yet).

Francesco

Re: Oops on mcp251xfd open on 6.6.52

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On 23.09.2024 14:00:27, Francesco Dolcini wrote:
> Hello Marc,
> 
> On Mon, Sep 23, 2024 at 01:57:07PM +0200, Marc Kleine-Budde wrote:
> > On 23.09.2024 13:53:10, Francesco Dolcini wrote:
> > > I got the following Oops on doing a simple ip link set canX up, running
> > > on an i.MX8MM SoC.
> > > 
> > > This was reproduced with some automated testing, I have not tried to
> > > reproduce it so far. Any idea or hint? It seems systematic, introduced
> > > around 12th september on the v6.6.y branch.
> > 
> > Which Kernel version are you using?
> 
> v6.6.52 is reproducing the issue, I guess it started on some previous
> v6.6.x kernel (with x < 52), but I was not able to dig out the logs
> running on older kernel from our automated testing (yet).
> 
> I was not able to test mainline (yet).

Can you cherry-pick these and see if it fixes your problem?

51b2a7216122 ("can: mcp251xfd: properly indent labels")
a7801540f325 ("can: mcp251xfd: move mcp251xfd_timestamp_start()/stop() into mcp251xfd_chip_start/stop()")

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Oops on mcp251xfd open on 6.6.52

[Francesco Dolcini]

Hello Marc,

On Mon, Sep 23, 2024 at 04:36:16PM +0200, Marc Kleine-Budde wrote:
> On 23.09.2024 14:00:27, Francesco Dolcini wrote:
> > On Mon, Sep 23, 2024 at 01:57:07PM +0200, Marc Kleine-Budde wrote:
> > > On 23.09.2024 13:53:10, Francesco Dolcini wrote:
> > > > I got the following Oops on doing a simple ip link set canX up, running
> > > > on an i.MX8MM SoC.
> > > > 
> > > > This was reproduced with some automated testing, I have not tried to
> > > > reproduce it so far. Any idea or hint? It seems systematic, introduced
> > > > around 12th september on the v6.6.y branch.
> > > 
> > > Which Kernel version are you using?
> > 
> > v6.6.52 is reproducing the issue, I guess it started on some previous
> > v6.6.x kernel (with x < 52), but I was not able to dig out the logs
> > running on older kernel from our automated testing (yet).
> > 
> > I was not able to test mainline (yet).
> 
> Can you cherry-pick these and see if it fixes your problem?
> 
> 51b2a7216122 ("can: mcp251xfd: properly indent labels")
> a7801540f325 ("can: mcp251xfd: move mcp251xfd_timestamp_start()/stop() into mcp251xfd_chip_start/stop()")

Yes, these 2 commits fix the issue.
In addition I noticed that also 6.1.111 is affected by the exact same Oops.

Thanks!

BTW, to reproduce, just

  ip link set can0 type can bitrate 1000000
  ip link set can0 up 

How to we move forward now? Would you ask greg to backport to stable kernel also
these 2 patches (I assume that something was backported that broke the
functionality at some point)? 

Francesco

Re: Oops on mcp251xfd open on 6.6.52

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

On 24.09.2024 00:41:04, Francesco Dolcini wrote:
> Hello Marc,
> 
> On Mon, Sep 23, 2024 at 04:36:16PM +0200, Marc Kleine-Budde wrote:
> > On 23.09.2024 14:00:27, Francesco Dolcini wrote:
> > > On Mon, Sep 23, 2024 at 01:57:07PM +0200, Marc Kleine-Budde wrote:
> > > > On 23.09.2024 13:53:10, Francesco Dolcini wrote:
> > > > > I got the following Oops on doing a simple ip link set canX up, running
> > > > > on an i.MX8MM SoC.
> > > > > 
> > > > > This was reproduced with some automated testing, I have not tried to
> > > > > reproduce it so far. Any idea or hint? It seems systematic, introduced
> > > > > around 12th september on the v6.6.y branch.
> > > > 
> > > > Which Kernel version are you using?
> > > 
> > > v6.6.52 is reproducing the issue, I guess it started on some previous
> > > v6.6.x kernel (with x < 52), but I was not able to dig out the logs
> > > running on older kernel from our automated testing (yet).
> > > 
> > > I was not able to test mainline (yet).
> > 
> > Can you cherry-pick these and see if it fixes your problem?
> > 
> > 51b2a7216122 ("can: mcp251xfd: properly indent labels")
> > a7801540f325 ("can: mcp251xfd: move mcp251xfd_timestamp_start()/stop() into mcp251xfd_chip_start/stop()")
> 
> Yes, these 2 commits fix the issue.
> In addition I noticed that also 6.1.111 is affected by the exact same Oops.
> 
> Thanks!

Thanks for testing!

> BTW, to reproduce, just
> 
>   ip link set can0 type can bitrate 1000000
>   ip link set can0 up 
> 
> How to we move forward now? Would you ask greg to backport to stable kernel also
> these 2 patches

Yeah, that's pretty much what I'm going to do now :).

> (I assume that something was backported that broke the
> functionality at some point)? 

Yes, it's:

upstream:
| 24436be590c6 ("can: mcp251xfd: rx: add workaround for erratum DS80000789E 6 of mcp2518fd")

v6.6:
| 5ea24ddc26a7 ("can: mcp251xfd: rx: add workaround for erratum DS80000789E 6 of mcp2518fd")

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Oops on mcp251xfd open on 6.6.52

[Marc Kleine-Budde]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On 24.09.2024 08:21:06, Marc Kleine-Budde wrote:
> > How to we move forward now? Would you ask greg to backport to stable kernel also
> > these 2 patches
> 
> Yeah, that's pretty much what I'm going to do now :).

done:
https://lore.kernel.org/all/20240924-truthful-authentic-basilisk-aaab90-mkl@pengutronix.de

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]