Re: Tracking down suspend/resume ext3/mmc issues on imx233

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Shawn Guo]

Copy a few more lists to get wider audience ...

Regards,
Shawn

On Thu, Sep 06, 2012 at 10:03:35PM -0700, Mike Thompson wrote:
> Hi,
> 
> I'm working on adding power management support for the imx233 on
> 3.6-rc2.  In general I'm working on porting the pm.c file from the
> Freescale 2.6.35 kernel for both "standby" and "mem" suspend/resume.
> I'm making pretty good progress on porting the code, but I'm running
> into an issue outside the immediate code I'm working on.
> 
> Right now I'm just testing out imx233 suspend/resume functionality by
> implementing a dummy "standby" function that just returns rather than
> putting the SoC into a low power state.  I believe this should have
> the kernel go through suspend operations, hit my dummy function which
> does nothing and then go through resume operations.  Unfortunately,
> upon resume I'm getting ext3 file system errors in that nothing can be
> read from the root file system on the SD card.  A trace of the
> suspend/resume cycle is show below:
> 
> root@olinuxino:~# echo standby > /sys/power/state
> [   90.080000] PM: Syncing filesystems ... done.
> [   90.110000] mmc0: card aaaa removed
> [   90.330000] Freezing user space processes ... (elapsed 0.01 seconds) done.
> [   90.360000] Freezing remaining freezable tasks ... (elapsed 0.01
> seconds) done.
> [   90.400000] mxs_mmc_suspend()
> [   90.400000] PM: suspend of devices complete after 19.531 msecs
> [   90.410000] PM: late suspend of devices complete after 2.843 msecs
> [   90.420000] PM: noirq suspend of devices complete after 4.531 msecs
> [   90.430000] PM: noirq resume of devices complete after 1.906 msecs
> [   90.440000] PM: early resume of devices complete after 1.843 msecs
> [   90.450000] mxs_mmc_resume()
> [   90.500000] PM: resume of devices complete after 51.062 msecs
> [   90.510000] Restarting tasks ... done.
> [   90.640000] mmc0: host does not support reading read-only switch.
> assuming write-enable.
> [   90.650000] mmc0: new high speed SD card at address aaaa
> [   90.670000] mmcblk0: mmc0:aaaa SU02G 1.84 GiB
> [   90.680000]  mmcblk0: p1 p2
> [   90.700000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #15970 offset 0
> [   90.710000] Aborting journal on device mmcblk0p2.
> [   90.720000] JBD: Error -5 detected when updating journal superblock
> for mmcblk0p2.
> [   90.730000] EXT3-fs (mmcblk0p2): error: remounting filesystem read-only
> [   90.730000] EXT3-fs (mmcblk0p2): I/O error while writing superblock
> [   90.740000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #2 offset 0
> udevd[1309]: failed to execute '/sbin/modprobe' '/sbin/modprobe -b
> mmc:block': No such file or directory
> [   90.790000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #15970 offset 0
> udevd[1310]: failed to execute '/sbin/blkid' '/sbin/blkid -o udev -p
> /dev/.tmp-block-179:8': No such file or directory
> udevd[1311]: failed to execute '/sbin/blkid' '/sbin/blkid -o udev -p
> /dev/.tmp-block-179:10': No such file or directory
> udevd[1312]: failed to execute '/sbin/blkid' '/sbin/blkid -o udev -p
> /dev/.tmp-block-179:9': No such file or directory
> root@olinuxino:~#
> root@olinuxino:~# ls
> [  432.780000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #55890 offset 0
> [  432.790000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #55891 offset 0
> [  432.800000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #55890 offset 0
> [  432.810000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #15970 offset 0
> [  432.820000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #7985 offset 0
> [  432.840000] EXT3-fs error (device mmcblk0p2): ext3_find_entry:
> reading directory #55890 offset 0
> -bash: ls: command not found
> root@olinuxino:~#
> 
> The first reported error is in the ext3 filesystem buffer code where
> the file system buffers aren't being filled by the underlying block
> device.  At least that's how I'm interpreting the portion of the ext3
> file system code that is failing.  However, the mmc device is
> correctly reporting finding p1 and p2 partitions on the device which
> it would indicate the partition data is being read from the SD card.
> 
> I'm hoping others might have suggestions on how I should go about
> tracking down why the ext3 file system can no longer read from the mmc
> device upon resume.  For instance, useful places to put some tracing
> code to understand what might be failing, or how to determine what
> differences there might be before suspend and after resume that might
> point to the failure.
> 
> Thanks,
> 
> Mike Thompson
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Mike Thompson]

I figured the source of my problem.  Normally, the Linux kernel will
treat mmc cards as removed during suspend.  This is in case the card
is swapped by the user during the suspend.  However, on the OLinuXino
the mmc device is the root file system  If the Linux kernel treats the
mmc card as removed the frozen tasks with open file handles to the
root filesystem device (such as the journal tasks) will fail when they
are thawed during resume.  This is the problem I was seeing.

Fortunately, there is already a specific kernel config setting
CONFIG_MMC_UNSAFE_RESUME that needs to be set for systems such as the
OLinuXino that uses the mmc device for the root filesystem.  I
overlooked this config setting and spent half a day trying to figure
out the source of the file system errors on resume.  Next time I'll do
a better job of looking through related config settings before trying
to debug an issue that is probably already dealt with.

Mike

On Thu, Sep 6, 2012 at 11:05 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
> Copy a few more lists to get wider audience ...
>
> Regards,
> Shawn
>
> On Thu, Sep 06, 2012 at 10:03:35PM -0700, Mike Thompson wrote:
>> I'm working on adding power management support for the imx233 on
>> 3.6-rc2.  In general I'm working on porting the pm.c file from the
>> Freescale 2.6.35 kernel for both "standby" and "mem" suspend/resume.
>> I'm making pretty good progress on porting the code, but I'm running
>> into an issue outside the immediate code I'm working on.
>>
>> ...
>>
>> The first reported error is in the ext3 filesystem buffer code where
>> the file system buffers aren't being filled by the underlying block
>> device.  At least that's how I'm interpreting the portion of the ext3
>> file system code that is failing.  However, the mmc device is
>> correctly reporting finding p1 and p2 partitions on the device which
>> it would indicate the partition data is being read from the SD card.
>>
>> I'm hoping others might have suggestions on how I should go about
>> tracking down why the ext3 file system can no longer read from the mmc
>> device upon resume.  For instance, useful places to put some tracing
>> code to understand what might be failing, or how to determine what
>> differences there might be before suspend and after resume that might
>> point to the failure.
>>
>> Thanks,
>>
>> Mike Thompson
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Matt Sealey]

Wouldn't it be better if the root filesystem was marked as
non-removable in the device tree - or in the case of a truly removable
card, just marked in the MMC subsystem - and the MMC subsystem skipped
the "it could be removed" for suspend/resume operations?

Whether you can or you cannot remove an SD memory device or eMMC
controller, having an option that marked "DANGEROUS" that says "MMC
layer will assume that all cards stayed in their respective slots
during the suspend" implies that on a system such as the
numbering-more-than-I-can-count embedded devices here with eMMC and
any number greater than zero SD card slots (most of them having at
least two), ALL of them are treated as unsafe resume, when in fact
only one of them might be.

My use cases here are usually boot from eMMC or boot from external SD
card (Snowball, Efika MX but one SD card slot is usually not needed to
go through the unsafe-resume process, all FSL i.MX boards since MX51
Babbage through the new MX6 boards, Pandaboard, Beagleboard). Just
because it's possible to boot from external, removable SD card
shouldn't mean we all have to configure our kernels explicitly to go
through a strange and unusual path for suspend and resume which leaves
ALL cards 'mounted' unecessarily over suspend/resume, especially in
the common case where eMMC is also not the ONLY boot medium (PATA,
SATA, non-eMMC flash) and eMMC or SD *MAY* only be the root filesystem
(as opposed to *WILL*)

??

-- 
Matt Sealey <matt@genesi-usa.com>
Product Development Analyst, Genesi USA, Inc.


On Sat, Sep 8, 2012 at 11:17 AM, Mike Thompson <mpthompson@gmail.com> wrote:
> I figured the source of my problem.  Normally, the Linux kernel will
> treat mmc cards as removed during suspend.  This is in case the card
> is swapped by the user during the suspend.  However, on the OLinuXino
> the mmc device is the root file system  If the Linux kernel treats the
> mmc card as removed the frozen tasks with open file handles to the
> root filesystem device (such as the journal tasks) will fail when they
> are thawed during resume.  This is the problem I was seeing.
>
> Fortunately, there is already a specific kernel config setting
> CONFIG_MMC_UNSAFE_RESUME that needs to be set for systems such as the
> OLinuXino that uses the mmc device for the root filesystem.  I
> overlooked this config setting and spent half a day trying to figure
> out the source of the file system errors on resume.  Next time I'll do
> a better job of looking through related config settings before trying
> to debug an issue that is probably already dealt with.
>
> Mike
>
> On Thu, Sep 6, 2012 at 11:05 PM, Shawn Guo <shawn.guo@linaro.org> wrote:
>> Copy a few more lists to get wider audience ...
>>
>> Regards,
>> Shawn
>>
>> On Thu, Sep 06, 2012 at 10:03:35PM -0700, Mike Thompson wrote:
>>> I'm working on adding power management support for the imx233 on
>>> 3.6-rc2.  In general I'm working on porting the pm.c file from the
>>> Freescale 2.6.35 kernel for both "standby" and "mem" suspend/resume.
>>> I'm making pretty good progress on porting the code, but I'm running
>>> into an issue outside the immediate code I'm working on.
>>>
>>> ...
>>>
>>> The first reported error is in the ext3 filesystem buffer code where
>>> the file system buffers aren't being filled by the underlying block
>>> device.  At least that's how I'm interpreting the portion of the ext3
>>> file system code that is failing.  However, the mmc device is
>>> correctly reporting finding p1 and p2 partitions on the device which
>>> it would indicate the partition data is being read from the SD card.
>>>
>>> I'm hoping others might have suggestions on how I should go about
>>> tracking down why the ext3 file system can no longer read from the mmc
>>> device upon resume.  For instance, useful places to put some tracing
>>> code to understand what might be failing, or how to determine what
>>> differences there might be before suspend and after resume that might
>>> point to the failure.
>>>
>>> Thanks,
>>>
>>> Mike Thompson
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Theodore Ts'o]

On Mon, Sep 10, 2012 at 10:11:48AM -0500, Matt Sealey wrote:
> Wouldn't it be better if the root filesystem was marked as
> non-removable in the device tree - or in the case of a truly removable
> card, just marked in the MMC subsystem - and the MMC subsystem skipped
> the "it could be removed" for suspend/resume operations?

I agree, this makes a lot of sense.  If the root file system
disappears, you're toasted either way, so it's fair to assume that the
device on which the root file system is located should is
non-removable.

						- Ted

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Pavel Machek]

On Mon 2012-09-10 12:33:45, Theodore Ts'o wrote:
> On Mon, Sep 10, 2012 at 10:11:48AM -0500, Matt Sealey wrote:
> > Wouldn't it be better if the root filesystem was marked as
> > non-removable in the device tree - or in the case of a truly removable
> > card, just marked in the MMC subsystem - and the MMC subsystem skipped
> > the "it could be removed" for suspend/resume operations?
> 
> I agree, this makes a lot of sense.  If the root file system
> disappears, you're toasted either way, so it's fair to assume that the
> device on which the root file system is located should is
> non-removable.

I'm not sure I agree.

If you treat root fs as removable, you'll get "crash". You'll need to
replay the journal, but data is safe.

If you treat it as non-removable, and someone manages to remove it,
mount, and reinsert, you'll get silent data corruption.

If the card is not removable, mark it as such...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Theodore Ts'o]

On Thu, Sep 20, 2012 at 01:23:49AM +0200, Pavel Machek wrote:
> 
> I'm not sure I agree.
> 
> If you treat root fs as removable, you'll get "crash". You'll need to
> replay the journal, but data is safe.
> 
> If you treat it as non-removable, and someone manages to remove it,
> mount, and reinsert, you'll get silent data corruption.

We could detect this case; if the file system gets mounted, the last
mount time will change.  So one of the things we could do is have the
file system code freeze the file system at suspend time, so the file
system is consistent (which will reduce the probability of data loss
if the system never comes back up after the suspend), and save the
last mount time and last write time in memory.  When the system comes
back from resume, have the file system code check the last mount and
last write time, and if they have changed, it can refuse the resume
and abort the system to avoid data corruption.  It would require
making ext3/ext4 suspend-aware, but it would be doable, if we really
wanted to support this.

						- Ted

Re: Tracking down suspend/resume ext3/mmc issues on imx233

[Pavel Machek]

Hi!

On Wed 2012-09-19 20:01:10, Theodore Ts'o wrote:
> On Thu, Sep 20, 2012 at 01:23:49AM +0200, Pavel Machek wrote:
> > 
> > I'm not sure I agree.
> > 
> > If you treat root fs as removable, you'll get "crash". You'll need to
> > replay the journal, but data is safe.
> > 
> > If you treat it as non-removable, and someone manages to remove it,
> > mount, and reinsert, you'll get silent data corruption.
> 
> We could detect this case; if the file system gets mounted, the last
> mount time will change.  So one of the things we could do is have the
> file system code freeze the file system at suspend time, so the file
> system is consistent (which will reduce the probability of data loss
> if the system never comes back up after the suspend), and save the
> last mount time and last write time in memory.  When the system comes
> back from resume, have the file system code check the last mount and
> last write time, and if they have changed, it can refuse the resume
> and abort the system to avoid data corruption.  It would require
> making ext3/ext4 suspend-aware, but it would be doable, if we really
> wanted to support this.

Yes please; I'd love to see this done.

We used to do sync() to get filesystem in consistent state, and I
believe that these days we are doing filesystem freeze; but code
validating the last mount time is certainly missing.

Looking forward,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html