* Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash.
@ 2013-11-20 2:07 Glen, Andrew
2013-11-24 22:51 ` Marc Kleine-Budde
0 siblings, 1 reply; 3+ messages in thread
From: Glen, Andrew @ 2013-11-20 2:07 UTC (permalink / raw)
To: linux-can@vger.kernel.org
I've been getting occasional crashes running the 3.12 mainline kernel with a Debian rootfs on an AM335x (Beaglebone Black).
The crash would only ever occur the first time the can0 device is brought up, i.e.: Reboot, call to 'ip link set can0 type can bitrate 250000 triple-sampling on', then 'ip link set can0 up' and about 40% of the time I'd get a crash.
If the 'ip link set can0 up' succeeded I could then bring the can0 device up/down repeatedly with no issues.
Strangely, this problem would not occur running the same kernel with the Ubuntu rootfs - but I think this is unrelated to the issue.
As the crash dump suggests, the problem is caused by a call to '__pm_runtime_resume' being made from within an interrupt routine - occuring when the CAN device is brought up immediately into a bus error (assumedly caused by starting up the can receiver midway through a transmission by another device).
Power management should not occur from within an interrupt routine, so by making a special interrupt version of the routine problematically calling '__pm_runtime_resume' (i.e. 'c_can_get_berr_counter'), but with this call removed, I was initially able to get around the problem.
Further researching into the issue, I found the following in '/Documentation/power/runtime_pm.txt':
It is safe to execute the following helper functions from interrupt context:
pm_request_idle()
pm_request_autosuspend()
pm_schedule_suspend()
pm_request_resume()
pm_runtime_get_noresume()
pm_runtime_get()
pm_runtime_put_noidle()
pm_runtime_put()
pm_runtime_put_autosuspend()
pm_runtime_enable()
pm_suspend_ignore_children()
pm_runtime_set_active()
pm_runtime_set_suspended()
pm_runtime_suspended()
pm_runtime_mark_last_busy()
pm_runtime_autosuspend_expiration()
If pm_runtime_irq_safe() has been called for a device then the following helper
functions may also be used in interrupt context:
pm_runtime_idle()
pm_runtime_suspend()
pm_runtime_autosuspend()
pm_runtime_resume()
pm_runtime_get_sync()
pm_runtime_put_sync()
pm_runtime_put_sync_suspend()
pm_runtime_put_sync_autosuspend()
In 'c_can.c' as we find it in 3.12 mainline, calls to 'pm_runtime_resume()' are being made from the interrupt context (i.e. omap3_intc_handle_irq ->handle_IRQ->irq_exit->do_softirq->__do_softirq->net_rx_action->c_can_poll->c_can_state_change->c_can_get_berr_counter->__pm_runtime_resume) without 'pm_runtime_irq_safe' having first been called.
So it makes sense that these problems were occurring, and that the problem goes away after inserting this call.
I propose something like the following to fix this problem:
diff --git a/drivers/net/can/c_can/c_can.c b/drivers/net/can/c_can/c_can.c index a668cd4..638fd22 100644
--- a/drivers/net/can/c_can/c_can.c
+++ b/drivers/net/can/c_can/c_can.c
<at> <at> -1103,6 +1103,8 <at> <at> static int c_can_open(struct net_device *dev)
int err;
struct c_can_priv *priv = netdev_priv(dev);
+ pm_runtime_irq_safe(priv->device);
+
c_can_pm_runtime_get_sync(priv);
c_can_reset_ram(priv, true);
Regards,
Andrew Glen.
________________________________
Actuant Corporation Email Notice
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.
Thank you.
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash.
2013-11-20 2:07 Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash Glen, Andrew
@ 2013-11-24 22:51 ` Marc Kleine-Budde
2013-11-25 0:32 ` Glen, Andrew
0 siblings, 1 reply; 3+ messages in thread
From: Marc Kleine-Budde @ 2013-11-24 22:51 UTC (permalink / raw)
To: Glen, Andrew; +Cc: linux-can@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 4353 bytes --]
On 11/20/2013 03:07 AM, Glen, Andrew wrote:
> I've been getting occasional crashes running the 3.12 mainline kernel
> with a Debian rootfs on an AM335x (Beaglebone Black).
> The crash would only ever occur the first time the can0 device is
> brought up, i.e.: Reboot, call to 'ip link set can0 type can bitrate
> 250000 triple-sampling on', then 'ip link set can0 up' and about 40%
> of the time I'd get a crash.
> If the 'ip link set can0 up' succeeded I could then bring the can0
> device up/down repeatedly with no issues.
> Strangely, this problem would not occur running the same kernel with
> the Ubuntu rootfs - but I think this is unrelated to the issue.
> As the crash dump suggests, the problem is caused by a call to
> '__pm_runtime_resume' being made from within an interrupt routine -
> occuring when the CAN device is brought up immediately into a bus
> error (assumedly caused by starting up the can receiver midway
> through a transmission by another device).
Can you provide the crash dump?
> Power management should not occur from within an interrupt routine,
> so by making a special interrupt version of the routine
> problematically calling '__pm_runtime_resume' (i.e.
> 'c_can_get_berr_counter'), but with this call removed, I was
> initially able to get around the problem.
> Further researching into the issue, I found the following in
> '/Documentation/power/runtime_pm.txt':
> It is safe to execute the following helper functions from interrupt context:
>
> pm_request_idle()
> pm_request_autosuspend()
> pm_schedule_suspend()
> pm_request_resume()
> pm_runtime_get_noresume()
> pm_runtime_get()
> pm_runtime_put_noidle()
> pm_runtime_put()
> pm_runtime_put_autosuspend()
> pm_runtime_enable()
> pm_suspend_ignore_children()
> pm_runtime_set_active()
> pm_runtime_set_suspended()
> pm_runtime_suspended()
> pm_runtime_mark_last_busy()
> pm_runtime_autosuspend_expiration()
>
> If pm_runtime_irq_safe() has been called for a device then the
> following helper functions may also be used in interrupt context:
>
> pm_runtime_idle()
> pm_runtime_suspend()
> pm_runtime_autosuspend()
> pm_runtime_resume()
> pm_runtime_get_sync()
> pm_runtime_put_sync()
> pm_runtime_put_sync_suspend()
> pm_runtime_put_sync_autosuspend()
>
>
> In 'c_can.c' as we find it in 3.12 mainline, calls to
> 'pm_runtime_resume()' are being made from the interrupt context (i.e.
> omap3_intc_handle_irq
> ->handle_IRQ->irq_exit->do_softirq->__do_softirq->net_rx_action->c_can_poll->c_can_state_change->c_can_get_berr_counter->__pm_runtime_resume)
> without 'pm_runtime_irq_safe' having first been called.
> So it makes sense that these problems were occurring, and that the
> problem goes away after inserting this call.
>
> I propose something like the following to fix this problem:
>
> diff --git a/drivers/net/can/c_can/c_can.c b/drivers/net/can/c_can/c_can.c index a668cd4..638fd22 100644
> --- a/drivers/net/can/c_can/c_can.c
> +++ b/drivers/net/can/c_can/c_can.c
> <at> <at> -1103,6 +1103,8 <at> <at> static int c_can_open(struct net_device *dev)
> int err;
> struct c_can_priv *priv = netdev_priv(dev);
>
> + pm_runtime_irq_safe(priv->device);
> +
> c_can_pm_runtime_get_sync(priv);
> c_can_reset_ram(priv, true);
Thanks for the detailed analysis. The problem with the
c_can_get_berr_counter() function is, that is is called from IRQ handler
and from the userspace via the priv->can.do_get_berr_counter callback.
If the c_can_get_berr_counter() is called from the IRQ handler, there is
no need to call c_can_pm_runtime_get_sync(), as the device is up the
open() function has been called. As the callback can be called if the
device is down, we need the c_can_pm_runtime_get_sync() call.
From my point of view (I'm not familiar with runtime pm)
pm_runtime_irq_safe() has some drawbacks. I think we have a better
option, by introducing __c_can_get_berr_counter() which will not call
into the runtime PM.
Marc
--
Pengutronix e.K. | Marc Kleine-Budde |
Industrial Linux Solutions | Phone: +49-231-2826-924 |
Vertretung West/Dortmund | Fax: +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686 | http://www.pengutronix.de |
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* RE: Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash.
2013-11-24 22:51 ` Marc Kleine-Budde
@ 2013-11-25 0:32 ` Glen, Andrew
0 siblings, 0 replies; 3+ messages in thread
From: Glen, Andrew @ 2013-11-25 0:32 UTC (permalink / raw)
To: Marc Kleine-Budde; +Cc: linux-can@vger.kernel.org
Abbreviated crashdump below:
Debian GNU/Linux 7 arm ttyO0
arm login: [ 34.689047] c_can_platform 481cc000.d_can can0: setting BTR=1c05 BRPE=0000
[ 34.696387] BUG: scheduling while atomic: ip/2215/0x40000100
[ 34.702351] Modules linked in: usb_f_acm u_serial g_multi libcomposite u_rndis u_ether arc4 snd_soc_omap snd_pcm_dmaengine snd_soc_core regmap_spi snd_compress ath9k_htc snd_pcm snd_timer snd mac80211 soundcore snd_page_alloc ath9k_common ath9k_hw ath cfg80211 c_can_platform c_can can_dev
[ 34.729468] CPU: 0 PID: 2215 Comm: ip Not tainted 3.12.0-bone8 #2
[ 34.735931] [<c00136fc>] (unwind_backtrace+0x0/0xdc) from [<c0010a30>] (show_stack+0x10/0x14)
[ 34.744941] [<c0010a30>] (show_stack+0x10/0x14) from [<c05bdc90>] (dump_stack+0x70/0x8c)
[ 34.753488] [<c05bdc90>] (dump_stack+0x70/0x8c) from [<c05b9b20>] (__schedule_bug+0x48/0x64)
[ 34.762384] [<c05b9b20>] (__schedule_bug+0x48/0x64) from [<c05c1694>] (__schedule+0x68/0x6c0)
[ 34.771368] [<c05c1694>] (__schedule+0x68/0x6c0) from [<c005a6c0>] (__cond_resched+0x24/0x34)
[ 34.780349] [<c005a6c0>] (__cond_resched+0x24/0x34) from [<c05c2070>] (_cond_resched+0x3c/0x44)
[ 34.789514] [<c05c2070>] (_cond_resched+0x3c/0x44) from [<c038e12c>] (__pm_runtime_resume+0x24/0x78)
[ 34.799155] [<c038e12c>] (__pm_runtime_resume+0x24/0x78) from [<bf00e0fc>] (c_can_get_berr_counter+0x24/0x60 [c_can])
[ 34.810342] [<bf00e0fc>] (c_can_get_berr_counter+0x24/0x60 [c_can]) from [<bf00e164>] (c_can_handle_state_change+0x2c/0x160 [c_can])
[ 34.822894] [<bf00e164>] (c_can_handle_state_change+0x2c/0x160 [c_can]) from [<bf00eb80>] (c_can_poll+0xb4/0x7f0 [c_can])
[ 34.834445] [<bf00eb80>] (c_can_poll+0xb4/0x7f0 [c_can]) from [<c04da208>] (net_rx_action+0xe4/0x1b8)
[ 34.844161] [<c04da208>] (net_rx_action+0xe4/0x1b8) from [<c003bffc>] (__do_softirq+0xec/0x1d4)
[ 34.853323] [<c003bffc>] (__do_softirq+0xec/0x1d4) from [<c003c17c>] (do_softirq+0x48/0x54)
[ 34.862119] [<c003c17c>] (do_softirq+0x48/0x54) from [<c003c3d4>] (irq_exit+0x84/0xdc)
[ 34.870461] [<c003c3d4>] (irq_exit+0x84/0xdc) from [<c000e2c8>] (handle_IRQ+0x6c/0x90)
[ 34.878802] [<c000e2c8>] (handle_IRQ+0x6c/0x90) from [<c00085e4>] (omap3_intc_handle_irq+0x54/0x68)
[ 34.888328] [<c00085e4>] (omap3_intc_handle_irq+0x54/0x68) from [<c0011480>] (__irq_svc+0x40/0x50)
[ 34.897756] Exception stack(0xde6b7980 to 0xde6b79c8)
[ 34.903080] 7980: 00000000 c0d824b0 60000013 60000013 0000003e c08a47b8 00000000 00000001
[ 34.911689] 79a0: 00000000 00000006 de6b6008 c08e0b3c c08a6bb8 de6b79c8 00000000 c006ee24
[ 34.920293] 79c0: 60000013 ffffffff
[ 34.923970] [<c0011480>] (__irq_svc+0x40/0x50) from [<c006ee24>] (vprintk_emit+0x434/0x4ac)
[ 34.932777] [<c006ee24>] (vprintk_emit+0x434/0x4ac) from [<c0385550>] (dev_vprintk_emit+0x48/0x50)
[ 34.942215] [<c0385550>] (dev_vprintk_emit+0x48/0x50) from [<c0385570>] (dev_printk_emit+0x18/0x28)
[ 34.951769] [<c0385570>] (dev_printk_emit+0x18/0x28) from [<c04d9e40>] (__netdev_printk+0x64/0xbc)
[ 34.961209] [<c04d9e40>] (__netdev_printk+0x64/0xbc) from [<c04da098>] (netdev_info+0x30/0x44)
[ 34.970286] [<c04da098>] (netdev_info+0x30/0x44) from [<bf00e7c4>] (c_can_start+0x174/0x200 [c_can])
[ 34.979913] [<bf00e7c4>] (c_can_start+0x174/0x200 [c_can]) from [<bf00ea70>] (c_can_open+0xc0/0x11c [c_can])
[ 34.990267] [<bf00ea70>] (c_can_open+0xc0/0x11c [c_can]) from [<c04dc280>] (__dev_open+0xa8/0x104)
[ 34.999703] [<c04dc280>] (__dev_open+0xa8/0x104) from [<c04dc494>] (__dev_change_flags+0x8c/0x118)
[ 35.009141] [<c04dc494>] (__dev_change_flags+0x8c/0x118) from [<c04dc598>] (dev_change_flags+0x10/0x44)
[ 35.019038] [<c04dc598>] (dev_change_flags+0x10/0x44) from [<c04e7eb8>] (do_setlink+0x2b8/0x784)
[ 35.028292] [<c04e7eb8>] (do_setlink+0x2b8/0x784) from [<c04e8eec>] (rtnl_newlink+0x250/0x460)
[ 35.037363] [<c04e8eec>] (rtnl_newlink+0x250/0x460) from [<c04e9298>] (rtnetlink_rcv_msg+0x19c/0x1b0)
[ 35.047079] [<c04e9298>] (rtnetlink_rcv_msg+0x19c/0x1b0) from [<c04fd664>] (netlink_rcv_skb+0x58/0xac)
[ 35.056878] [<c04fd664>] (netlink_rcv_skb+0x58/0xac) from [<c04e7ac0>] (rtnetlink_rcv+0x18/0x24)
[ 35.066129] [<c04e7ac0>] (rtnetlink_rcv+0x18/0x24) from [<c04fcd60>] (netlink_unicast+0xac/0x18c)
[ 35.075472] [<c04fcd60>] (netlink_unicast+0xac/0x18c) from [<c04fd494>] (netlink_sendmsg+0x5bc/0x638)
[ 35.085187] [<c04fd494>] (netlink_sendmsg+0x5bc/0x638) from [<c04c5698>] (sock_sendmsg+0x70/0x8c)
[ 35.094534] [<c04c5698>] (sock_sendmsg+0x70/0x8c) from [<c04c67e4>] (___sys_sendmsg.part.27+0x17c/0x200)
[ 35.104516] [<c04c67e4>] (___sys_sendmsg.part.27+0x17c/0x200) from [<c04c75a4>] (__sys_sendmsg+0x4c/0x70)
[ 35.114589] [<c04c75a4>] (__sys_sendmsg+0x4c/0x70) from [<c000d9e0>] (ret_fast_syscall+0x0/0x30)
-----Original Message-----
From: Marc Kleine-Budde [mailto:mkl@pengutronix.de]
Sent: Monday, 25 November 2013 11:52 a.m.
To: Glen, Andrew
Cc: linux-can@vger.kernel.org
Subject: Re: Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash.
On 11/20/2013 03:07 AM, Glen, Andrew wrote:
> I've been getting occasional crashes running the 3.12 mainline kernel
> with a Debian rootfs on an AM335x (Beaglebone Black).
> The crash would only ever occur the first time the can0 device is
> brought up, i.e.: Reboot, call to 'ip link set can0 type can bitrate
> 250000 triple-sampling on', then 'ip link set can0 up' and about 40%
> of the time I'd get a crash.
> If the 'ip link set can0 up' succeeded I could then bring the can0
> device up/down repeatedly with no issues.
> Strangely, this problem would not occur running the same kernel with
> the Ubuntu rootfs - but I think this is unrelated to the issue.
> As the crash dump suggests, the problem is caused by a call to
> '__pm_runtime_resume' being made from within an interrupt routine -
> occuring when the CAN device is brought up immediately into a bus
> error (assumedly caused by starting up the can receiver midway through
> a transmission by another device).
Can you provide the crash dump?
> Power management should not occur from within an interrupt routine, so
> by making a special interrupt version of the routine problematically
> calling '__pm_runtime_resume' (i.e.
> 'c_can_get_berr_counter'), but with this call removed, I was initially
> able to get around the problem.
> Further researching into the issue, I found the following in
> '/Documentation/power/runtime_pm.txt':
> It is safe to execute the following helper functions from interrupt context:
>
> pm_request_idle()
> pm_request_autosuspend()
> pm_schedule_suspend()
> pm_request_resume()
> pm_runtime_get_noresume()
> pm_runtime_get()
> pm_runtime_put_noidle()
> pm_runtime_put()
> pm_runtime_put_autosuspend()
> pm_runtime_enable()
> pm_suspend_ignore_children()
> pm_runtime_set_active()
> pm_runtime_set_suspended()
> pm_runtime_suspended()
> pm_runtime_mark_last_busy()
> pm_runtime_autosuspend_expiration()
>
> If pm_runtime_irq_safe() has been called for a device then the
> following helper functions may also be used in interrupt context:
>
> pm_runtime_idle()
> pm_runtime_suspend()
> pm_runtime_autosuspend()
> pm_runtime_resume()
> pm_runtime_get_sync()
> pm_runtime_put_sync()
> pm_runtime_put_sync_suspend()
> pm_runtime_put_sync_autosuspend()
>
>
> In 'c_can.c' as we find it in 3.12 mainline, calls to
> 'pm_runtime_resume()' are being made from the interrupt context (i.e.
> omap3_intc_handle_irq
> ->handle_IRQ->irq_exit->do_softirq->__do_softirq->net_rx_action->c_can
> ->_poll->c_can_state_change->c_can_get_berr_counter->__pm_runtime_resu
> ->me)
> without 'pm_runtime_irq_safe' having first been called.
> So it makes sense that these problems were occurring, and that the
> problem goes away after inserting this call.
>
> I propose something like the following to fix this problem:
>
> diff --git a/drivers/net/can/c_can/c_can.c
> b/drivers/net/can/c_can/c_can.c index a668cd4..638fd22 100644
> --- a/drivers/net/can/c_can/c_can.c
> +++ b/drivers/net/can/c_can/c_can.c
> <at> <at> -1103,6 +1103,8 <at> <at> static int c_can_open(struct net_device *dev)
> int err;
> struct c_can_priv *priv = netdev_priv(dev);
>
> + pm_runtime_irq_safe(priv->device);
> +
> c_can_pm_runtime_get_sync(priv);
> c_can_reset_ram(priv, true);
Thanks for the detailed analysis. The problem with the
c_can_get_berr_counter() function is, that is is called from IRQ handler and from the userspace via the priv->can.do_get_berr_counter callback.
If the c_can_get_berr_counter() is called from the IRQ handler, there is no need to call c_can_pm_runtime_get_sync(), as the device is up the
open() function has been called. As the callback can be called if the device is down, we need the c_can_pm_runtime_get_sync() call.
From my point of view (I'm not familiar with runtime pm)
pm_runtime_irq_safe() has some drawbacks. I think we have a better option, by introducing __c_can_get_berr_counter() which will not call into the runtime PM.
Marc
--
Pengutronix e.K. | Marc Kleine-Budde |
Industrial Linux Solutions | Phone: +49-231-2826-924 |
Vertretung West/Dortmund | Fax: +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686 | http://www.pengutronix.de |
________________________________
Actuant Corporation Email Notice
This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.
Thank you.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2013-11-25 0:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-11-20 2:07 Calls to 'pm_runtime_resume()' made from the interrupt context without 'pm_runtime_irq_safe' in 'c_can.c' causing occasional kernel crash Glen, Andrew
2013-11-24 22:51 ` Marc Kleine-Budde
2013-11-25 0:32 ` Glen, Andrew
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox