SDIO runtime crash on resume

SDIO runtime crash on resume

[Daniel Drake]

Hi,

Testing linux-next, I get a crash during resume. Reproduced on an
XO-1.5. mmc1 is the wifi card, for which the driver is not loaded.

Reverting 9b966aae64^..ed2a978594 (9 runtime sdio patches) causes the
issue to go away.


mmc1: error -110 during resume (card was removed?)
PM: resume of devices complete after 338.994 msecs
Restarting tasks ...
hub 1-0:1.0: over-current change on port 1
done.
usb 1-2: USB disconnect, address 2
mmc1: card 0001 removed
BUG: unable to handle kernel NULL pointer dereference at 00000104
IP: [<b05e9d95>] __pm_runtime_idle+0x15/0x53
*pde = 39a62067 *pte = 00000000
Oops: 0002 [#1] PREEMPT
last sysfs file: /sys/devices/pci0000:00/0000:00:10.4/usb1/idVendor
Modules linked in:

Pid: 2011, comm: kworker/u:12 Not tainted 2.6.36-next-20101027+ #10 XO/XO
EIP: 0060:[<b05e9d95>] EFLAGS: 00010202 CPU: 0
EIP is at __pm_runtime_idle+0x15/0x53
EAX: 00000004 EBX: 00000004 ECX: 00000001 EDX: 00000005
ESI: e9b58c10 EDI: e9f5bf40 EBP: e9f5bf18 ESP: e9f5bf04
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process kworker/u:12 (pid: 2011, ti=e9f5a000 task=e9dd45d0 task.ti=e9f5a000)
Stack:
 e9b58a48 e9b58c10 e9b58a48 e9b58c10 e9f5bf40 e9f5bf28 b06474d0 e9b58a48
 e9b58c10 e9f5bf50 b0642fab 00061a80 000493e0 00030d40 000186a0 b0740857
 e9b58cb0 e9e4ed38 b09372c0 e9f5bf7c b0433f50 e9f5bf64 b041a8de b0642f29
Call Trace:
 [<b06474d0>] ? mmc_sdio_detect+0x85/0x89
 [<b0642fab>] ? mmc_rescan+0x82/0x279
 [<b0740857>] ? sub_preempt_count+0x81/0x8e
 [<b0433f50>] ? process_one_work+0x1d0/0x31d
 [<b041a8de>] ? get_parent_ip+0xb/0x31
 [<b0642f29>] ? mmc_rescan+0x0/0x279
 [<b043420c>] ? worker_thread+0x16f/0x2a2
 [<b043409d>] ? worker_thread+0x0/0x2a2
 [<b0436ba7>] ? kthread+0x66/0x6b
 [<b0436b41>] ? kthread+0x0/0x6b
 [<b0402db6>] ? kernel_thread_helper+0x6/0x10
Code: 08 00 00 00 89 d8 e8 6f fe ff ff 89 f0 e8 9d 44 15 00 5b 5e 5d
c3 55 89 e5 57 56 53 83 ec 08 0f 1f 44 00 00 f6 c2 04 89 c3 74 0f <ff>
88 00 01
EIP: [<b05e9d95>] __pm_runtime_idle+0x15/0x53 SS:ESP 0068:e9f5bf04
CR2: 0000000000000104
---[ end trace b018ceefced52bd9 ]---

Daniel

Re: SDIO runtime crash on resume

[Ohad Ben-Cohen]

Hi Daniel,

On Wed, Oct 27, 2010 at 8:46 PM, Daniel Drake <dsd@laptop.org> wrote:
> Hi,
>
> Testing linux-next, I get a crash during resume. Reproduced on an
> XO-1.5. mmc1 is the wifi card, for which the driver is not loaded.

For some reason I can't get linux-next to boot on my setup (strangely
VFS refuses to mount my NFS rootfs).

Until I get this sorted, can you please tell me if you can reproduce
this with mmc-next ?

Can you please also send full logs (from boot to crash preferably) + .config ?

Thanks,
Ohad.

>
> Reverting 9b966aae64^..ed2a978594 (9 runtime sdio patches) causes the
> issue to go away.
>
>
> mmc1: error -110 during resume (card was removed?)
> PM: resume of devices complete after 338.994 msecs
> Restarting tasks ...
> hub 1-0:1.0: over-current change on port 1
> done.
> usb 1-2: USB disconnect, address 2
> mmc1: card 0001 removed
> BUG: unable to handle kernel NULL pointer dereference at 00000104
> IP: [<b05e9d95>] __pm_runtime_idle+0x15/0x53
> *pde = 39a62067 *pte = 00000000
> Oops: 0002 [#1] PREEMPT
> last sysfs file: /sys/devices/pci0000:00/0000:00:10.4/usb1/idVendor
> Modules linked in:
>
> Pid: 2011, comm: kworker/u:12 Not tainted 2.6.36-next-20101027+ #10 XO/XO
> EIP: 0060:[<b05e9d95>] EFLAGS: 00010202 CPU: 0
> EIP is at __pm_runtime_idle+0x15/0x53
> EAX: 00000004 EBX: 00000004 ECX: 00000001 EDX: 00000005
> ESI: e9b58c10 EDI: e9f5bf40 EBP: e9f5bf18 ESP: e9f5bf04
>  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
> Process kworker/u:12 (pid: 2011, ti=e9f5a000 task=e9dd45d0 task.ti=e9f5a000)
> Stack:
>  e9b58a48 e9b58c10 e9b58a48 e9b58c10 e9f5bf40 e9f5bf28 b06474d0 e9b58a48
>  e9b58c10 e9f5bf50 b0642fab 00061a80 000493e0 00030d40 000186a0 b0740857
>  e9b58cb0 e9e4ed38 b09372c0 e9f5bf7c b0433f50 e9f5bf64 b041a8de b0642f29
> Call Trace:
>  [<b06474d0>] ? mmc_sdio_detect+0x85/0x89
>  [<b0642fab>] ? mmc_rescan+0x82/0x279
>  [<b0740857>] ? sub_preempt_count+0x81/0x8e
>  [<b0433f50>] ? process_one_work+0x1d0/0x31d
>  [<b041a8de>] ? get_parent_ip+0xb/0x31
>  [<b0642f29>] ? mmc_rescan+0x0/0x279
>  [<b043420c>] ? worker_thread+0x16f/0x2a2
>  [<b043409d>] ? worker_thread+0x0/0x2a2
>  [<b0436ba7>] ? kthread+0x66/0x6b
>  [<b0436b41>] ? kthread+0x0/0x6b
>  [<b0402db6>] ? kernel_thread_helper+0x6/0x10
> Code: 08 00 00 00 89 d8 e8 6f fe ff ff 89 f0 e8 9d 44 15 00 5b 5e 5d
> c3 55 89 e5 57 56 53 83 ec 08 0f 1f 44 00 00 f6 c2 04 89 c3 74 0f <ff>
> 88 00 01
> EIP: [<b05e9d95>] __pm_runtime_idle+0x15/0x53 SS:ESP 0068:e9f5bf04
> CR2: 0000000000000104
> ---[ end trace b018ceefced52bd9 ]---
>
> Daniel
>

Re: SDIO runtime crash on resume

[Ohad Ben-Cohen]

Hi Daniel,

On Wed, Oct 27, 2010 at 10:37 PM, Ohad Ben-Cohen <ohad@wizery.com> wrote:
...
> can you please tell me if you can reproduce this with mmc-next ?
>
> Can you please also send full logs (from boot to crash preferably) + .config ?

Please 'echo 8 > /proc/sys/kernel/printk' before generating the logs.

Some additional questions (some of them will be answered by the
logs+config you will send, but in case you're not in front of the
setup you can just answer those directly):

- What is your mmc host ? sdhci-pci presumably ? I might be able to
get such hardware.. But not before next week (LPC).. Hey any chance
you or Chris plan to attend LPC with this setup ? I'm going to be
there and if I had your setup with a reproducible bug that would make
this very easy to fix...

- During boot, was you card identified and added (i assume yes) ?

- After boot, was your card powered down (i assume yes) ?
  check this out by:
  mount -t debugfs none /sys/kernel/debug
  cat /sys/kernel/debug/mmc1/ios

- Just for general knowledge - as soon as you load your driver
(libertas, right ?) - was your card powered up and correctly
configured (sdio clock/voltage/bus width. again check with
/sys/kernel/debug/mmc2/ios) ?

> mmc1: error -110 during resume (card was removed?)

I assume there are no failures before it ? the system was correctly suspended ?

Anyway this timeout (-110) shouldn't happen. The device might still be
powered off for some reason. I guess the timeout is coming from
mmc_sdio_init_card(). I just wonder whether it's coming already from
mmc_send_io_op_cond(), or whether it's a side effect of 9b966aae64 and
thus it's coming from a later stage of mmc_sdio_init_card. If you can
please put some pr_info("%s %d: ret %d\n", __func__, __LINE__, ret)
lines throughout that function that could be great.

>> mmc1: card 0001 removed

care to put a dump_stack() in the beginning of mmc_remove_card() ?

>> IP: [<b05e9d95>] __pm_runtime_idle+0x15/0x53

That can be all sort of badness.

I'll wait for the logs+more info (+chance to have this setup in LPC ;)
and we'll continue from there.

Thanks,
Ohad.

Re: SDIO runtime crash on resume

[Chris Ball]

Hi Ohad,

On Thu, Oct 28, 2010 at 04:39:34PM +0200, Ohad Ben-Cohen wrote:
> - What is your mmc host ? sdhci-pci presumably ? I might be able to
> get such hardware.. But not before next week (LPC).. Hey any chance
> you or Chris plan to attend LPC with this setup ? I'm going to be
> there and if I had your setup with a reproducible bug that would make
> this very easy to fix...

I'll just give you an XO-1.5 at LPC -- that way you can work on it
either during the conference or afterwards.  We should still get a
.config from Daniel, though, and steps to reproduce.

Thanks!  Look forward to meeting you there,

- Chris.
-- 
Chris Ball   <cjb@laptop.org>   <http://printf.net/>
One Laptop Per Child

Re: SDIO runtime crash on resume

[Ohad Ben-Cohen]

Hi Chris,

On Thu, Oct 28, 2010 at 4:55 PM, Chris Ball <cjb@laptop.org> wrote:
> I'll just give you an XO-1.5 at LPC -- that way you can work on it
> either during the conference or afterwards.  We should still get a
> .config from Daniel, though, and steps to reproduce.

That would be really good, thanks a lot !

> Thanks!  Look forward to meeting you there,

I look forward to meeting you too, thanks again !
Ohad.

Re: SDIO runtime crash on resume

[Daniel Drake]

On 28 October 2010 15:39, Ohad Ben-Cohen <ohad@wizery.com> wrote:
> - What is your mmc host ? sdhci-pci presumably ? I might be able to
> get such hardware.. But not before next week (LPC).. Hey any chance
> you or Chris plan to attend LPC with this setup ? I'm going to be
> there and if I had your setup with a reproducible bug that would make
> this very easy to fix...

Yes, it's sdhci-pci.

> - Just for general knowledge - as soon as you load your driver
> (libertas, right ?) - was your card powered up and correctly
> configured (sdio clock/voltage/bus width. again check with
> /sys/kernel/debug/mmc2/ios) ?

No driver for this hardware was loaded.

> I assume there are no failures before it ? the system was correctly suspended ?

It looked that way.

Config is here:
http://dev.laptop.org/~dsd/20101029/xo1p5-config
Note that anything built as a module is not available on the setup in
question where the bug was found. So =m may as well say "is not set"

If you still need it, I'll try and get the other info to you soon.

Daniel

Re: SDIO runtime crash on resume

[Ohad Ben-Cohen]

On Fri, Oct 29, 2010 at 4:48 PM, Daniel Drake <dsd@laptop.org> wrote:
>> - Just for general knowledge - as soon as you load your driver
>> (libertas, right ?) - was your card powered up and correctly
>> configured (sdio clock/voltage/bus width. again check with
>> /sys/kernel/debug/mmc2/ios) ?
>
> No driver for this hardware was loaded.

Yeah, still that question is relevant because both scenarios have very
similar code paths.

> Config is here:
> http://dev.laptop.org/~dsd/20101029/xo1p5-config
> Note that anything built as a module is not available on the setup in
> question where the bug was found. So =m may as well say "is not set"

OK, thanks !

> If you still need it, I'll try and get the other info to you soon.

I'm anxious to get it :), but Chris will have an XO-1.5 at LPC, which
is just around the corner, so I don't mind waiting a bit. Having a
setup with a reproducible bug in front of me would make life very
easy.

Thanks again,
Ohad.