OMAP HSMMC problems with off-mode

OMAP HSMMC problems with off-mode

[Paul Walmsley]

Madhusudhan,

On my OMAP35xx BeagleBoard here, HSMMC is non-functional after returning 
from dynamic off-mode.  The MMC LED turns on and stays on, and the process 
hangs, although the rest of the system still seems to run.  Looks like 
you're listed as the maintainer for this code.  Do you know why this is, 
and what can be done to fix it?

This is using linux-omap master at commit 
a04fd22204b13ce34a3f8a8157f83c44d64f8da9, using omap2plus_defconfig with 
DSS configured in.  My boot and root partitions are on MMC.  Following are 
the commands I used to reproduce this problem.


regards

- Paul

echo 1 > /debug/pm_debug/sleep_while_idle
echo 1 > /debug/pm_debug/enable_off_mode
echo 5 > /sys/devices/platform/omap/omap-hsuart.0/sleep_timeout
echo 5 > /sys/devices/platform/omap/omap-hsuart.1/sleep_timeout
echo 5 > /sys/devices/platform/omap/omap-hsuart.2/sleep_timeout

Re: OMAP HSMMC problems with off-mode

[Chikkature Rajashekar, Madhusudhan]

Paul,

Last I tested OFF mode was after Adrian's power saving series was merged. I
tested it on Kevin PM branch then and it was functional on my omap3 board.

I need to check on the latest commit.

Did you see this problem in suspend/resume path or the cpuidle path?

Regards,
Madhu
On Tue, Dec 7, 2010 at 12:21 PM, Paul Walmsley <paul@pwsan.com> wrote:
>
> Madhusudhan,
>
> On my OMAP35xx BeagleBoard here, HSMMC is non-functional after returning
> from dynamic off-mode.  The MMC LED turns on and stays on, and the process
> hangs, although the rest of the system still seems to run.  Looks like
> you're listed as the maintainer for this code.  Do you know why this is,
> and what can be done to fix it?
>
> This is using linux-omap master at commit
> a04fd22204b13ce34a3f8a8157f83c44d64f8da9, using omap2plus_defconfig with
> DSS configured in.  My boot and root partitions are on MMC.  Following are
> the commands I used to reproduce this problem.
>
>
> regards
>
> - Paul
>
> echo 1 > /debug/pm_debug/sleep_while_idle
> echo 1 > /debug/pm_debug/enable_off_mode
> echo 5 > /sys/devices/platform/omap/omap-hsuart.0/sleep_timeout
> echo 5 > /sys/devices/platform/omap/omap-hsuart.1/sleep_timeout
> echo 5 > /sys/devices/platform/omap/omap-hsuart.2/sleep_timeout
>

Re: OMAP HSMMC problems with off-mode

[Paul Walmsley]

Hello Madhusudhan,

On Tue, 7 Dec 2010, Chikkature Rajashekar, Madhusudhan wrote:

> Last I tested OFF mode was after Adrian's power saving series was merged. I
> tested it on Kevin PM branch then and it was functional on my omap3 board.
>
> I need to check on the latest commit.
> 
> Did you see this problem in suspend/resume path or the cpuidle path?

I noticed it on the CPUIdle path.


- Paul

Re: OMAP HSMMC problems with off-mode

[Adrian Hunter]

On 07/12/10 20:21, ext Paul Walmsley wrote:
>
> Madhusudhan,
>
> On my OMAP35xx BeagleBoard here, HSMMC is non-functional after returning
> from dynamic off-mode.  The MMC LED turns on and stays on, and the process
> hangs, although the rest of the system still seems to run.  Looks like
> you're listed as the maintainer for this code.  Do you know why this is,
> and what can be done to fix it?

It is at least because omap_pm_get_dev_context_loss_count() is not
implemented.  Tero Kristo was looking at that recently.

>
> This is using linux-omap master at commit
> a04fd22204b13ce34a3f8a8157f83c44d64f8da9, using omap2plus_defconfig with
> DSS configured in.  My boot and root partitions are on MMC.  Following are
> the commands I used to reproduce this problem.
>
>
> regards
>
> - Paul
>
> echo 1>  /debug/pm_debug/sleep_while_idle
> echo 1>  /debug/pm_debug/enable_off_mode
> echo 5>  /sys/devices/platform/omap/omap-hsuart.0/sleep_timeout
> echo 5>  /sys/devices/platform/omap/omap-hsuart.1/sleep_timeout
> echo 5>  /sys/devices/platform/omap/omap-hsuart.2/sleep_timeout
>
>

Re: OMAP HSMMC problems with off-mode

[Chikkature Rajashekar, Madhusudhan]

On Tue, Dec 7, 2010 at 1:51 PM, Adrian Hunter <adrian.hunter@nokia.com> wrote:
> On 07/12/10 20:21, ext Paul Walmsley wrote:
>>
>> Madhusudhan,
>>
>> On my OMAP35xx BeagleBoard here, HSMMC is non-functional after returning
>> from dynamic off-mode.  The MMC LED turns on and stays on, and the process
>> hangs, although the rest of the system still seems to run.  Looks like
>> you're listed as the maintainer for this code.  Do you know why this is,
>> and what can be done to fix it?
>
> It is at least because omap_pm_get_dev_context_loss_count() is not
> implemented.  Tero Kristo was looking at that recently.
>

Yes. I agree that is the problem. In the .32 kernel I had hooked it to
"get_last_off_on_transaction_id" which helped.
But that functionality does not exist anymore. So something equivalent
to tell the driver when the OFF was hit will make it work.

Regards,
Madhu
>>
>> This is using linux-omap master at commit
>> a04fd22204b13ce34a3f8a8157f83c44d64f8da9, using omap2plus_defconfig with
>> DSS configured in.  My boot and root partitions are on MMC.  Following are
>> the commands I used to reproduce this problem.
>>
>>
>> regards
>>
>> - Paul
>>
>> echo 1>  /debug/pm_debug/sleep_while_idle
>> echo 1>  /debug/pm_debug/enable_off_mode
>> echo 5>  /sys/devices/platform/omap/omap-hsuart.0/sleep_timeout
>> echo 5>  /sys/devices/platform/omap/omap-hsuart.1/sleep_timeout
>> echo 5>  /sys/devices/platform/omap/omap-hsuart.2/sleep_timeout
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OMAP HSMMC problems with off-mode

[Paul Walmsley]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5346 bytes --]

Hi,

On Tue, 7 Dec 2010, Chikkature Rajashekar, Madhusudhan wrote:

> On Tue, Dec 7, 2010 at 1:51 PM, Adrian Hunter <adrian.hunter@nokia.com> wrote:
> >
> > It is at least because omap_pm_get_dev_context_loss_count() is not
> > implemented.  Tero Kristo was looking at that recently.
> >
> 
> Yes. I agree that is the problem. In the .32 kernel I had hooked it to
> "get_last_off_on_transaction_id" which helped.
> But that functionality does not exist anymore. So something equivalent
> to tell the driver when the OFF was hit will make it work.

OK, let's see if we can get that fixed in at least some trivial 
way for 2.6.38.  While working on this, I applied this trivial patch:

diff --git a/arch/arm/plat-omap/omap-pm-noop.c b/arch/arm/plat-omap/omap-pm-noop.c
index e129ce8..781aa5f 100644
--- a/arch/arm/plat-omap/omap-pm-noop.c
+++ b/arch/arm/plat-omap/omap-pm-noop.c
@@ -30,6 +30,8 @@ struct omap_opp *dsp_opps;
 struct omap_opp *mpu_opps;
 struct omap_opp *l3_opps;
 
+static int dummy_context_loss_counter;
+
 /*
  * Device-driver-originated constraints (via board-*.c files)
  */
@@ -303,7 +305,7 @@ int omap_pm_get_dev_context_loss_count(struct device *dev)
 	 * off counter.
 	 */
 
-	return 0;
+	return dummy_context_loss_counter++;
 }
 
 

... which causes drivers to believe that device context has been lost 
after each call to omap_pm_get_dev_context_loss_count().  Brutal, but 
effective for chasing out context save/restore bugs.  This patch causes an 
abort in the MMC driver with the current linux-omap master:

[    1.960693] Unhandled fault: external abort on non-linefetch (0x1028) at 0xfa09c014
[    1.968872] Internal error: : 1028 [#1] SMP
[    1.973266] last sysfs file: 
[    1.976379] Modules linked in:
[    1.979583] CPU: 0    Not tainted  (2.6.37-rc4-09012-gfea0fb8 #65)
[    1.986083] PC is at omap_hsmmc_context_restore+0x58/0x2ec
[    1.991851] LR is at omap_hsmmc_context_restore+0x4c/0x2ec
[    1.997619] pc : [<c0345590>]    lr : [<c0345584>]    psr: 20000013
[    1.997619] sp : cf82deb0  ip : cf82c000  fp : cf890408
[    2.009643] r10: cf826160  r9 : 00000053  r8 : 00000001
[    2.015136] r7 : cfad2000  r6 : cfad23c0  r5 : 00000000  r4 : ffff6b1f
[    2.021972] r3 : 0000000e  r2 : fa09c000  r1 : fa09c014  r0 : ffff6b22
[    2.028808] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[    2.036499] Control: 10c5387d  Table: 80004019  DAC: 00000017
[    2.042510] Process swapper (pid: 1, stack limit = 0xcf82c2f8)
[    2.048645] Stack: (0xcf82deb0 to 0xcf82e000)
[    2.053192] dea0:                                     cfad23c0 00000000 cfad2000 cfad23c0
[    2.061798] dec0: 00000000 c0345844 cfad2000 00000000 cfad2000 c0338040 00000000 00000001
[    2.070373] dee0: cfad2000 c002c728 00000000 00000003 cf886c08 cfad0a60 c05fdbc0 cf890408
[    2.078948] df00: cf890408 c05fdbc0 cfaef1a0 c05f06b8 00000000 00000000 00000000 c02a0368
[    2.087524] df20: cf890408 c029f388 cf890408 cf89043c c05fdbc0 c029f4b0 00000000 c029f448
[    2.096130] df40: c05fdbc0 c029eb3c cf826eb8 cf8b5890 cf8001e0 00000080 c05fdbc0 c029e468
[    2.104705] df60: c04f3804 00000468 00000000 c05fdbac c061bd54 c05fdbc0 c061bd4c c002c400
[    2.113281] df80: 00000000 c029f7a8 c05fdbac c061bd54 c05aa190 c061bd4c c002c400 c02a0770
[    2.121856] dfa0: c00335dc c061bd54 c05aa190 c0050574 00000000 00000000 c0503d6c 00000192
[    2.130462] dfc0: c061bd54 c00335dc c061bd54 c05aa190 c061bd4c 00000000 00000000 c00084ec
[    2.139038] dfe0: 00000000 c00083a0 c005bcf0 00000013 00000000 c005bcf0 1177ed0a 0073ef00
[    2.147644] [<c0345590>] (omap_hsmmc_context_restore+0x58/0x2ec) from [<c0345844>] (omap_hsmmc_enable_fclk+0x20/0x28)
[    2.158782] [<c0345844>] (omap_hsmmc_enable_fclk+0x20/0x28) from [<c0338040>] (mmc_host_enable+0x6c/0x90)
[    2.168823] [<c0338040>] (mmc_host_enable+0x6c/0x90) from [<c002c728>] (omap_hsmmc_probe+0x314/0x96c)
[    2.178497] [<c002c728>] (omap_hsmmc_probe+0x314/0x96c) from [<c02a0368>] (platform_drv_probe+0x18/0x1c)
[    2.188476] [<c02a0368>] (platform_drv_probe+0x18/0x1c) from [<c029f388>] (driver_probe_device+0xc8/0x188)
[    2.198608] [<c029f388>] (driver_probe_device+0xc8/0x188) from [<c029f4b0>] (__driver_attach+0x68/0x8c)
[    2.208465] [<c029f4b0>] (__driver_attach+0x68/0x8c) from [<c029eb3c>] (bus_for_each_dev+0x44/0x74)
[    2.217956] [<c029eb3c>] (bus_for_each_dev+0x44/0x74) from [<c029e468>] (bus_add_driver+0x104/0x294)
[    2.227569] [<c029e468>] (bus_add_driver+0x104/0x294) from [<c029f7a8>] (driver_register+0xa8/0x130)
[    2.237152] [<c029f7a8>] (driver_register+0xa8/0x130) from [<c02a0770>] (platform_driver_probe+0x18/0x8c)
[    2.247192] [<c02a0770>] (platform_driver_probe+0x18/0x8c) from [<c0050574>] (do_one_initcall+0xc8/0x1a0)
[    2.257263] [<c0050574>] (do_one_initcall+0xc8/0x1a0) from [<c00084ec>] (kernel_init+0x14c/0x214)
[    2.266571] [<c00084ec>] (kernel_init+0x14c/0x214) from [<c005bcf0>] (kernel_thread_exit+0x0/0x8)
[    2.275878] Code: ebf53fa3 e5962048 e2821014 e0800004 (e5913000) 
[    2.282409] ---[ end trace a0fb0e5d1754cad1 ]---
[    2.287353] Kernel panic - not syncing: Attempted to kill init!

Looks like 0xfa09c014 is the MMC SYSSTATUS register... maybe there's 
something wrong with the clock control.


- Paul

Re: OMAP HSMMC problems with off-mode

[Paul Walmsley]

Hi,

> Looks like 0xfa09c014 is the MMC SYSSTATUS register... maybe there's 
> something wrong with the clock control.

Just sent a patch for this one, it's relatively minor.

After applying it and the 'brutal context save/restore test' patch, HSMMC 
off-mode indeed seems to work much better.

Are there are any remaining fixes that anyone is aware of that need to go 
in for this driver to work reliably in off-mode?  Or is it considered to 
be in good shape for off-mode in mainline once the OMAP PM no-op context 
loss tracking code is fixed?

regards


- Paul

Re: OMAP HSMMC problems with off-mode

[Kevin Hilman]

Paul Walmsley <paul@pwsan.com> writes:

> Hi,
>
> On Tue, 7 Dec 2010, Chikkature Rajashekar, Madhusudhan wrote:
>
>> On Tue, Dec 7, 2010 at 1:51 PM, Adrian Hunter <adrian.hunter@nokia.com> wrote:
>> >
>> > It is at least because omap_pm_get_dev_context_loss_count() is not
>> > implemented.  Tero Kristo was looking at that recently.
>> >
>> 
>> Yes. I agree that is the problem. In the .32 kernel I had hooked it to
>> "get_last_off_on_transaction_id" which helped.
>> But that functionality does not exist anymore. So something equivalent
>> to tell the driver when the OFF was hit will make it work.
>
> OK, let's see if we can get that fixed in at least some trivial 
> way for 2.6.38.  While working on this, I applied this trivial patch:
>
> diff --git a/arch/arm/plat-omap/omap-pm-noop.c b/arch/arm/plat-omap/omap-pm-noop.c
> index e129ce8..781aa5f 100644
> --- a/arch/arm/plat-omap/omap-pm-noop.c
> +++ b/arch/arm/plat-omap/omap-pm-noop.c
> @@ -30,6 +30,8 @@ struct omap_opp *dsp_opps;
>  struct omap_opp *mpu_opps;
>  struct omap_opp *l3_opps;
>  
> +static int dummy_context_loss_counter;
> +
>  /*
>   * Device-driver-originated constraints (via board-*.c files)
>   */
> @@ -303,7 +305,7 @@ int omap_pm_get_dev_context_loss_count(struct device *dev)
>  	 * off counter.
>  	 */
>  
> -	return 0;
> +	return dummy_context_loss_counter++;
>  }
>  
>  
>
> ... which causes drivers to believe that device context has been lost 
> after each call to omap_pm_get_dev_context_loss_count().  Brutal, but 
> effective for chasing out context save/restore bugs.  

Tested-by: Kevin Hilman <khilman@deeprootsystems.com>

I verified that this, in combination with your other patch[1] results in
working off-suspend with MMC on 34xx/n900, 35xx/beagle and 36xx/zoom3.

Paul, do you want to submit a formal patch for this for 2.6.38? if not,
I can add a changelog and queue this with other PM core changes for 2.6.38.

Kevin

[1] MMC: omap_hsmmc: enable interface clock before calling mmc_host_enable()
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OMAP HSMMC problems with off-mode

[Paul Walmsley]

Hi Kevin

On Thu, 9 Dec 2010, Kevin Hilman wrote:

> Paul, do you want to submit a formal patch for this for 2.6.38? if not,
> I can add a changelog and queue this with other PM core changes for 2.6.38.

we should probably come up with a patch for .38 that only increments this 
when off-mode is entered... otherwise there will probably be a performance 
hit when off-mode-supporting devices are reinitialized after every 
clk_enable() :-)  The HSMMC driver also resets the IP block when that 
happens.



- Paul

Re: OMAP HSMMC problems with off-mode

[Paul Walmsley]

On Thu, 9 Dec 2010, Paul Walmsley wrote:

> On Thu, 9 Dec 2010, Kevin Hilman wrote:
> 
> > Paul, do you want to submit a formal patch for this for 2.6.38? if not,
> > I can add a changelog and queue this with other PM core changes for 2.6.38.
> 
> we should probably come up with a patch for .38 that only increments this 
> when off-mode is entered... otherwise there will probably be a performance 
> hit when off-mode-supporting devices are reinitialized after every 
> clk_enable() :-)  The HSMMC driver also resets the IP block when that 
> happens.

Although, just thinking about it, it might be worth adding a debugfs 
option or Kconfig option for the OMAP PM no-op layer that enables 'brutal 
mode' for testing?


- Paul

Re: OMAP HSMMC problems with off-mode

[Kevin Hilman]

Paul Walmsley <paul@pwsan.com> writes:

> Hi Kevin
>
> On Thu, 9 Dec 2010, Kevin Hilman wrote:
>
>> Paul, do you want to submit a formal patch for this for 2.6.38? if not,
>> I can add a changelog and queue this with other PM core changes for 2.6.38.
>
> we should probably come up with a patch for .38 that only increments this 
> when off-mode is entered... otherwise there will probably be a performance 
> hit when off-mode-supporting devices are reinitialized after every 
> clk_enable() :-)  The HSMMC driver also resets the IP block when that 
> happens.

I have a series for this (will post shortly), but it only works for
omap_devices, since otherwise it's rather hacky to find the powerdomain
for a given device.

Unfortunately, MMC is still not converted to omap_device/omap_hwmod, so
my series doesn't help for MMC. :(

Kevin