[Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Viresh Kumar]

Hi,

I am working with sdhci-spear.c driver on SPEAr SoC.
When i suspend the system, with card inserted, the system hangs, and after
2 minutes i get following log:


root@10.199.15.17:~# rtcwake -s 5 -m standby
rtcwake: wakeup from "standby" using /dev/rtc0 at Sat Nov 12 12:14:13 2011

PM: Syncing filesystems ... done.
mmc0: card 0007 removed

[Hangs here... This is not in log, but added by me]

INFO: task mmcqd/0:678 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mmcqd/0       D 803bbfa8     0   678      2 0x00000000
Backtrace: 
[<803bba3c>] (schedule+0x0/0x664) from [<802f1048>] (__mmc_claim_host+0xe4/0x180)
[<802f0f64>] (__mmc_claim_host+0x0/0x180) from [<802f95fc>] (mmc_blk_issue_rw_rq+0x34/0x4ec)
[<802f95c8>] (mmc_blk_issue_rw_rq+0x0/0x4ec) from [<802f9bfc>] (mmc_blk_issue_rq+0x148/0x150)
[<802f9ab4>] (mmc_blk_issue_rq+0x0/0x150) from [<802f9cf8>] (mmc_queue_thread+0xf4/0xf8)
[<802f9c04>] (mmc_queue_thread+0x0/0xf8) from [<80075468>] (kthread+0x90/0x98)
 r8:00000000 r7:00000013 r6:802f9c04 r5:be8e8a14 r4:be84fd28
r3:00000000
[<800753d8>] (kthread+0x0/0x98) from [<8005fd00>] (do_exit+0x0/0x5f8)
 r6:8005fd00 r5:800753d8 r4:be84fd28
INFO: task rtcwake:1554 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rtcwake       D 803bbfa8     0  1554   1550 0x00000000
Backtrace: 
[<803bba3c>] (schedule+0x0/0x664) from [<803bc2f4>] (schedule_timeout+0x20/0x1d0)
[<803bc2d4>] (schedule_timeout+0x0/0x1d0) from [<803bb8f8>] (wait_for_common+0xf4/0x18c)
 r7:00000002 r6:bea49fc0 r5:bea49fbc r4:7fffffff
[<803bb804>] (wait_for_common+0x0/0x18c) from [<803bba38>] (wait_for_completion+0x18/0x1c)
 r8:00000000 r7:00000001 r6:bea49fbc r5:be95cc08 r4:be95cc00
[<803bba20>] (wait_for_completion+0x0/0x1c) from [<800754bc>] (kthread_stop+0x4c/0x84)
[<80075470>] (kthread_stop+0x0/0x84) from [<802fa20c>] (mmc_cleanup_queue+0x24/0x88)
 r6:be9ced60 r5:00000000 r4:be8e8a14 r3:00000000
[<802fa1e8>] (mmc_cleanup_queue+0x0/0x88) from [<802f9004>] (mmc_blk_remove+0x30/0x48)
 r7:00000001 r6:be9c5d40 r5:be96ec08 r4:be8e8a00
[<802f8fd4>] (mmc_blk_remove+0x0/0x48) from [<802f240c>] (mmc_bus_remove+0x1c/0x24)
 r5:80534254 r4:be96ec08
[<802f23f0>] (mmc_bus_remove+0x0/0x24) from [<8020e13c>] (__device_release_driver+0x6c/0xac)
[<8020e0d0>] (__device_release_driver+0x0/0xac) from [<8020e1a0>] (device_release_driver+0x24/0x30)
 r5:be96ec08 r4:be96ec3c
[<8020e17c>] (device_release_driver+0x0/0x30) from [<8020dd30>] (bus_remove_device+0x80/0x90)
 r5:be9c5c08 r4:be96ec08
[<8020dcb0>] (bus_remove_device+0x0/0x90) from [<8020b5d4>] (device_del+0x118/0x170)
 r4:be96ec08 r3:00000002
[<8020b4bc>] (device_del+0x0/0x170) from [<802f2864>] (mmc_remove_card+0x64/0x78)
 r6:be9c5d40 r5:be9c5d18 r4:be96ec00 r3:60000013
[<802f2800>] (mmc_remove_card+0x0/0x78) from [<802f4630>] (mmc_sd_remove+0x38/0x48)
 r4:be9c5c00 r3:802f45f8
[<802f45f8>] (mmc_sd_remove+0x0/0x48) from [<802f236c>] (mmc_pm_notify+0x90/0xe0)
 r4:be9c5c00 r3:802f45f8
[<802f22dc>] (mmc_pm_notify+0x0/0xe0) from [<8007a05c>] (notifier_call_chain+0x40/0x70)
 r7:00000000 r6:ffffffff r5:00000000 r4:00000003
[<8007a01c>] (notifier_call_chain+0x0/0x70) from [<8007a344>] (__blocking_notifier_call_chain+0x50/0x68)
 r8:803c1fc4 r7:00000003 r6:00000000 r5:ffffffff r4:80509798
r3:ffffffff
[<8007a2f4>] (__blocking_notifier_call_chain+0x0/0x68) from [<8007a37c>] (blocking_notifier_call_chain+0x20/0x28)
 r7:00000008 r6:00000001 r5:00000001 r4:be3dd000
[<8007a35c>] (blocking_notifier_call_chain+0x0/0x28) from [<80089c4c>] (pm_notifier_call_chain+0x1c/0x34)
[<80089c30>] (pm_notifier_call_chain+0x0/0x34) from [<8008a47c>] (enter_state+0x70/0xd8)
[<8008a40c>] (enter_state+0x0/0xd8) from [<80089b58>] (state_store+0xb4/0xc8)
 r5:00000007 r4:be3dd000
[<80089aa4>] (state_store+0x0/0xc8) from [<8019b530>] (kobj_attr_store+0x1c/0x28)
[<8019b514>] (kobj_attr_store+0x0/0x28) from [<8010f62c>] (sysfs_write_file+0x110/0x144)
[<8010f51c>] (sysfs_write_file+0x0/0x144) from [<800c63a0>] (vfs_write+0xbc/0x148)
[<800c62e4>] (vfs_write+0x0/0x148) from [<800c6624>] (sys_write+0x44/0x70)
 r8:00000008 r7:00000000 r6:00000000 r5:2ac02000 r4:be2b0780
[<800c65e0>] (sys_write+0x0/0x70) from [<8003ab60>] (ret_fast_syscall+0x0/0x30)
 r8:8003ad08 r7:00000004 r6:000130e0 r5:2ac02000 r4:00000008


Controller specific suspend/resume are still not called, so i think it might
be related to core code. I am using 2.6.37 kernel version.

Any help on this?

-- 
viresh

Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Chris Ball]

Hi Viresh,

On Fri, Nov 11 2011, Viresh Kumar wrote:
> Controller specific suspend/resume are still not called, so i think it might
> be related to core code. I am using 2.6.37 kernel version.
>
> Any help on this?

Please try to reproduce on 3.1 and let us know if it's present there.

Thanks,

- Chris.
-- 
Chris Ball   <cjb@laptop.org>   <http://printf.net/>
One Laptop Per Child

Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Viresh Kumar]

On 11/12/2011 8:54 AM, Chris Ball wrote:
> On Fri, Nov 11 2011, Viresh Kumar wrote:
>> > Controller specific suspend/resume are still not called, so i think it might
>> > be related to core code. I am using 2.6.37 kernel version.
>> >
>> > Any help on this?
> Please try to reproduce on 3.1 and let us know if it's present there.

Hi Chris,

Support for SPEAr13xx is not present on 3.1 and it will take some time to
move support there. I have a bit more info which might able you to guide us:

file: drivers/mmc/card/queue.c

static int mmc_queue_thread(void *d)
{
	...
	do {
		req = blk_fetch_request(q);
		...
		if (!req) {
			if (kthread_should_stop()) {
				set_current_state(TASK_RUNNING);
				break;
			}
			up(&mq->thread_sem);
			schedule();
			down(&mq->thread_sem);
			continue;
		}
		set_current_state(TASK_RUNNING);
		mq->issue_fn(mq, req);
	} while (1);
	...
}

- In normal cases, i.e. when we are able to stop this thread,
  req is returned as NULL from blk_fetch_request()
- Suspend hang only occurs when the card is mounted.
- When suspend hangs, req is not returned as NULL and the card is
  already removed by kernel and so we never check kthread_should_stop().
  So it hangs.

-- 
viresh

RE: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Dong, Chuanxiao]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

Hi Viresh,
I also meet the hang issue when suspending a system with a mounted SD card. Actually it caused by a dead lock of mmc_claim_host. Maybe we faced the same issue.
So may be the two mails in the attachment can help you.

Thanks
Chuanxiao

> -----Original Message-----
> From: linux-mmc-owner@vger.kernel.org
> [mailto:linux-mmc-owner@vger.kernel.org] On Behalf Of Viresh Kumar
> Sent: Tuesday, November 15, 2011 1:47 PM
> To: Chris Ball
> Cc: linux-mmc@vger.kernel.org; Shiraz HASHIM; Armando VISCONTI
> Subject: Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.
> 
> On 11/12/2011 8:54 AM, Chris Ball wrote:
> > On Fri, Nov 11 2011, Viresh Kumar wrote:
> >> > Controller specific suspend/resume are still not called, so i think it might
> >> > be related to core code. I am using 2.6.37 kernel version.
> >> >
> >> > Any help on this?
> > Please try to reproduce on 3.1 and let us know if it's present there.
> 
> Hi Chris,
> 
> Support for SPEAr13xx is not present on 3.1 and it will take some time to
> move support there. I have a bit more info which might able you to guide us:
> 
> file: drivers/mmc/card/queue.c
> 
> static int mmc_queue_thread(void *d)
> {
> 	...
> 	do {
> 		req = blk_fetch_request(q);
> 		...
> 		if (!req) {
> 			if (kthread_should_stop()) {
> 				set_current_state(TASK_RUNNING);
> 				break;
> 			}
> 			up(&mq->thread_sem);
> 			schedule();
> 			down(&mq->thread_sem);
> 			continue;
> 		}
> 		set_current_state(TASK_RUNNING);
> 		mq->issue_fn(mq, req);
> 	} while (1);
> 	...
> }
> 
> - In normal cases, i.e. when we are able to stop this thread,
>   req is returned as NULL from blk_fetch_request()
> - Suspend hang only occurs when the card is mounted.
> - When suspend hangs, req is not returned as NULL and the card is
>   already removed by kernel and so we never check kthread_should_stop().
>   So it hangs.
> 
> --
> viresh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Type: message/rfc822, Size: 4953 bytes --]

From: "Dong, Chuanxiao" <chuanxiao.dong@intel.com>
To: "linux-mmc@vger.kernel.org" <linux-mmc@vger.kernel.org>
Cc: "He, Bo" <bo.he@intel.com>
Subject: [RFC]mmc: fix dead lock issue when system entering S3
Date: Tue, 17 May 2011 15:53:52 +0800
Message-ID: <20110517075352.GA3992@intel.com>

Hi all,
   I encountered a dead lock issue. Two threads try to claim host, unfortunately
   one thread needs to sync with the other. I encountered this issue with SD
   card when testing system entering S3. Does anybody encounter the same issue?

Environment:
1. without CONFIG_MMC_UNSAFE_RESUME operation
2. the SD card mounted

During system tried to enter S3, mmc_pm_notifier will be called first to remove
SD card. So calling sequence in SD remove thread is like this:
        mmc_claim_host
        bus_ops->remove
        ....
        ....
        mmc_cleanup_queue
        kthread_stop(mq->thread)
        ...
        ...

Mean while, mmc_cleanup_queue wakes up mq->thread, the calling sequence in mq->thread is
like this:
        mmc_queue_thread
        mq->issue_fn (mmc_blk_issue_rq)
        mmc_claim_host (dead lock)
        ....
        ....

Since mmc_claim_host is called in mq->thread (not SD remove thread) again,
unfortunately right now host is already claimed by SD remove thread which is
also waiting for mq->thread finished, so cause a dead lock here.

Move the mmc_claim_host(in mmc_pm_notifier) after bus_ops->remove can resolve
this dead lock. mmc_suspend_host() is using the same way to claim host.

Need to know your suggestions.

Signed-off-by: He Bo <bo.he@intel.com>
Signed-off-by: Chuanxiao Dong <chuanxiao.dong@intel.com>
---
 drivers/mmc/core/core.c |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 68091dd..1e27588 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -1850,11 +1850,10 @@ int mmc_pm_notify(struct notifier_block *notify_block,
                if (!host->bus_ops || host->bus_ops->suspend)
                        break;

-               mmc_claim_host(host);
-
                if (host->bus_ops->remove)
                        host->bus_ops->remove(host);

+               mmc_claim_host(host);
                mmc_detach_bus(host);
                mmc_release_host(host);
                host->pm_flags = 0;
--
1.7.3.4

--
To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #3: Type: message/rfc822, Size: 3614 bytes --]

From: Andrei Warkentin <andreiw@motorola.com>
To: "Dong, Chuanxiao" <chuanxiao.dong@intel.com>
Cc: "linux-mmc@vger.kernel.org" <linux-mmc@vger.kernel.org>, "He, Bo" <bo.he@intel.com>
Subject: Re: [RFC]mmc: fix dead lock issue when system entering S3
Date: Wed, 18 May 2011 15:58:00 +0800
Message-ID: <BANLkTi=H5NR4vzkD+X4fZXw-PPFYwO51ow@mail.gmail.com>

Hey,

On Tue, May 17, 2011 at 5:03 AM, Dong, Chuanxiao
<chuanxiao.dong@intel.com> wrote:
> Just found Andrei Warkentin also submitted a RFC patch, which included more fix for mounted root file system media. Please ignore this thread.

I submitted an RFC patch as an idea up for discussion, but certainly
the issue never went away :-).. Your patch will not quite work. If you
have a filesystem it will not unmount, so mmc0 (for example) will
never release, so when a new (or same) card re-enumerates, it will be
mmc1...

I believe my patch basically created "md orphans" which would be
reconnected on resume, which is bit of a really nasty hack... I'm not
sure if there is a better way. I could have sworn I saw a patch a week
ago that attempted to resolve the suspend/resume with mounted fs issue
by deferring removal until resume, but I don't see it now, and I'm not
sure if it dealt well with the whole attempting to unmount fs in mmc
block cleanup path...

I guess I can resubmit...

A

Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Viresh Kumar]

On 11/15/2011 12:27 PM, Dong, Chuanxiao wrote:
> Hi Viresh,
> I also meet the hang issue when suspending a system with a mounted SD card. Actually it caused by a dead lock of mmc_claim_host. Maybe we faced the same issue.
> So may be the two mails in the attachment can help you.

Hi Dong,

You wouldn't believe it, but just 5 mins earlier i found the mail
chain you sent to me. I got this fix of moving claim_host below remove
and it worked for me too. But i faced another issue with the patch when
i suspend/resume again and again in a loop. It might or might not be
related to this patch.

But i don't see this work getting pushed into Linux.
What is the current state at your SoC? Are you keeping this issue open as
"Don't suspend with MMC card mounted"

Or you are working with this fix?

Thanks for the mail.

-- 
viresh

RE: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Dong, Chuanxiao]

I am using a 2.6.35+, not a 3.0 kernel. So with the patch in my board, it works. Andrei Warkentin has mentioned that, if the mounted device contains root file system, that patch cannot fix this problem really since the device node will be changed after resuming back. That is also the reason not push to linux-mmc

Thanks
Chuanxiao

> -----Original Message-----
> From: Viresh Kumar [mailto:viresh.kumar@st.com]
> Sent: Tuesday, November 15, 2011 3:10 PM
> To: Dong, Chuanxiao
> Cc: Chris Ball; linux-mmc@vger.kernel.org; Shiraz HASHIM; Armando VISCONTI;
> Ajay KHANDELWAL
> Subject: Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.
> 
> On 11/15/2011 12:27 PM, Dong, Chuanxiao wrote:
> > Hi Viresh,
> > I also meet the hang issue when suspending a system with a mounted SD card.
> Actually it caused by a dead lock of mmc_claim_host. Maybe we faced the same
> issue.
> > So may be the two mails in the attachment can help you.
> 
> Hi Dong,
> 
> You wouldn't believe it, but just 5 mins earlier i found the mail
> chain you sent to me. I got this fix of moving claim_host below remove
> and it worked for me too. But i faced another issue with the patch when
> i suspend/resume again and again in a loop. It might or might not be
> related to this patch.
> 
> But i don't see this work getting pushed into Linux.
> What is the current state at your SoC? Are you keeping this issue open as
> "Don't suspend with MMC card mounted"
> 
> Or you are working with this fix?
> 
> Thanks for the mail.
> 
> --
> viresh

Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Sujit Reddy Thumma]

Hi Ulf,

I see similar issue with your patch in 3.2 "mmc: core: Prevent too long 
response times for suspend".

                 if (mmc_try_claim_host(host)) {
<snip>
/* if CONFIG_MMC_UNSAFE_RESUME is not set, remove() callback would get 
blocked until mmcqd gets mmc_claim_host() causing deadlock here */
                                 if (host->bus_ops->remove)
                                         host->bus_ops->remove(host);
<snip>
                 } else {
                         err = -EBUSY;
                 }
         }

Proabably, we can do something like this:

diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 5278ffb..0177d4a 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -2309,6 +2309,7 @@ int mmc_suspend_host(struct mmc_host *host)
                                  * We simply "remove" the card in this 
case.
                                  * It will be redetected on resume.
                                  */
+                               mmc_do_release_host(host);
                                 if (host->bus_ops->remove)
                                         host->bus_ops->remove(host);
                                 mmc_claim_host(host);
@@ -2317,8 +2318,9 @@ int mmc_suspend_host(struct mmc_host *host)
                                 mmc_release_host(host);
                                 host->pm_flags = 0;
                                 err = 0;
+                       } else {
+                               mmc_do_release_host(host);
                         }
-                       mmc_do_release_host(host);
                 } else {
                         err = -EBUSY;
                 }

Please let me know if you agree. I can post a patch for this.


Thanks
Sujit

On 11/15/2011 12:27 PM, Dong, Chuanxiao wrote:
> Hi Viresh,
> I also meet the hang issue when suspending a system with a mounted SD card. Actually it caused by a dead lock of mmc_claim_host. Maybe we faced the same issue.
> So may be the two mails in the attachment can help you.
>
> Thanks
> Chuanxiao
>
>> -----Original Message-----
>> From: linux-mmc-owner@vger.kernel.org
>> [mailto:linux-mmc-owner@vger.kernel.org] On Behalf Of Viresh Kumar
>> Sent: Tuesday, November 15, 2011 1:47 PM
>> To: Chris Ball
>> Cc: linux-mmc@vger.kernel.org; Shiraz HASHIM; Armando VISCONTI
>> Subject: Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.
>>
>> On 11/12/2011 8:54 AM, Chris Ball wrote:
>>> On Fri, Nov 11 2011, Viresh Kumar wrote:
>>>>> Controller specific suspend/resume are still not called, so i think it might
>>>>> be related to core code. I am using 2.6.37 kernel version.
>>>>>
>>>>> Any help on this?
>>> Please try to reproduce on 3.1 and let us know if it's present there.
>>
>> Hi Chris,
>>
>> Support for SPEAr13xx is not present on 3.1 and it will take some time to
>> move support there. I have a bit more info which might able you to guide us:
>>
>> file: drivers/mmc/card/queue.c
>>
>> static int mmc_queue_thread(void *d)
>> {
>> 	...
>> 	do {
>> 		req = blk_fetch_request(q);
>> 		...
>> 		if (!req) {
>> 			if (kthread_should_stop()) {
>> 				set_current_state(TASK_RUNNING);
>> 				break;
>> 			}
>> 			up(&mq->thread_sem);
>> 			schedule();
>> 			down(&mq->thread_sem);
>> 			continue;
>> 		}
>> 		set_current_state(TASK_RUNNING);
>> 		mq->issue_fn(mq, req);
>> 	} while (1);
>> 	...
>> }
>>
>> - In normal cases, i.e. when we are able to stop this thread,
>>    req is returned as NULL from blk_fetch_request()
>> - Suspend hang only occurs when the card is mounted.
>> - When suspend hangs, req is not returned as NULL and the card is
>>    already removed by kernel and so we never check kthread_should_stop().
>>    So it hangs.
>>
>> --
>> viresh
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thanks & Regards,
Sujit Reddy Thumma

Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [Query] mmc/sdhci: Suspend hangs if card is inserted at suspend.

[Ulf Hansson]

Sujit Reddy Thumma wrote:
> Hi Ulf,
> 
> I see similar issue with your patch in 3.2 "mmc: core: Prevent too long 
> response times for suspend".
> 
>                  if (mmc_try_claim_host(host)) {
> <snip>
> /* if CONFIG_MMC_UNSAFE_RESUME is not set, remove() callback would get 
> blocked until mmcqd gets mmc_claim_host() causing deadlock here */
>                                  if (host->bus_ops->remove)
>                                          host->bus_ops->remove(host);
> <snip>

You are definitely right. I did not test my patch properly, without 
CONFIG_MMC_UNSAFE_RESUME, I realize now.

>                  } else {
>                          err = -EBUSY;
>                  }
>          }
> 
> Proabably, we can do something like this:
> 
> diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
> index 5278ffb..0177d4a 100644
> --- a/drivers/mmc/core/core.c
> +++ b/drivers/mmc/core/core.c
> @@ -2309,6 +2309,7 @@ int mmc_suspend_host(struct mmc_host *host)
>                                   * We simply "remove" the card in this 
> case.
>                                   * It will be redetected on resume.
>                                   */
> +                               mmc_do_release_host(host);
>                                  if (host->bus_ops->remove)
>                                          host->bus_ops->remove(host);
>                                  mmc_claim_host(host);
> @@ -2317,8 +2318,9 @@ int mmc_suspend_host(struct mmc_host *host)
>                                  mmc_release_host(host);
>                                  host->pm_flags = 0;
>                                  err = 0;
> +                       } else {
> +                               mmc_do_release_host(host);
>                          }
> -                       mmc_do_release_host(host);

Your changes looks feasible! Thanks for fixing this issue!


>                  } else {
>                          err = -EBUSY;
>                  }
> 
> Please let me know if you agree. I can post a patch for this.
> 


BR
Ulf Hansson