UBIFS recovery taking too long

UBIFS recovery taking too long

[David Mosberger-Tang]

We recently encountered a strange issue where UBIFS recovery is
suddenly taking a lot longer than usual (v3.7-based kernel).  In our
case, recovery took about 30 seconds.  This caused a serious problem
because our watchdog timer was set to 15 seconds, effectively
rendering the devices unbootable.

Does anybody know what triggers this slow recovery mode?  Also, how
can we calculate a worst-case recovery time for a given flash
chip-size.

Thanks,

  --david

Fwd: UBIFS recovery taking too long

[David Mosberger-Tang]

I've had no luck getting any response from the linux-mtd mailing list
regarding the issue reported below.
I think it is a very serious issue since it can easily render an
embedded system unusable.

  --david

---------- Forwarded message ----------
From: David Mosberger-Tang <dmosberger@gmail.com>
Date: Thu, Nov 7, 2013 at 11:44 AM
Subject: UBIFS recovery taking too long
To: "linux-mtd@lists.infradead.org" <linux-mtd@lists.infradead.org>


We recently encountered a strange issue where UBIFS recovery is
suddenly taking a lot longer than usual (v3.7-based kernel).  In our
case, recovery took about 30 seconds.  This caused a serious problem
because our watchdog timer was set to 15 seconds, effectively
rendering the devices unbootable.

Does anybody know what triggers this slow recovery mode?  Also, how
can we calculate a worst-case recovery time for a given flash
chip-size.

Thanks,

  --david

Re: UBIFS recovery taking too long

[Shuah Khan]

Adding ubifs maintainers.

-- Shuah

On Mon, Dec 9, 2013 at 1:53 PM, David Mosberger-Tang
<dmosberger@gmail.com> wrote:
> I've had no luck getting any response from the linux-mtd mailing list
> regarding the issue reported below.
> I think it is a very serious issue since it can easily render an
> embedded system unusable.
>
>   --david
>
> ---------- Forwarded message ----------
> From: David Mosberger-Tang <dmosberger@gmail.com>
> Date: Thu, Nov 7, 2013 at 11:44 AM
> Subject: UBIFS recovery taking too long
> To: "linux-mtd@lists.infradead.org" <linux-mtd@lists.infradead.org>
>
>
> We recently encountered a strange issue where UBIFS recovery is
> suddenly taking a lot longer than usual (v3.7-based kernel).  In our
> case, recovery took about 30 seconds.  This caused a serious problem
> because our watchdog timer was set to 15 seconds, effectively
> rendering the devices unbootable.
>
> Does anybody know what triggers this slow recovery mode?  Also, how
> can we calculate a worst-case recovery time for a given flash
> chip-size.
>
> Thanks,
>
>   --david
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: UBIFS recovery taking too long

[Adrian Hunter]

On 09/12/13 23:30, Shuah Khan wrote:
> Adding ubifs maintainers.
> 
> -- Shuah
> 
> On Mon, Dec 9, 2013 at 1:53 PM, David Mosberger-Tang
> <dmosberger@gmail.com> wrote:
>> I've had no luck getting any response from the linux-mtd mailing list
>> regarding the issue reported below.
>> I think it is a very serious issue since it can easily render an
>> embedded system unusable.
>>
>>   --david
>>
>> ---------- Forwarded message ----------
>> From: David Mosberger-Tang <dmosberger@gmail.com>
>> Date: Thu, Nov 7, 2013 at 11:44 AM
>> Subject: UBIFS recovery taking too long
>> To: "linux-mtd@lists.infradead.org" <linux-mtd@lists.infradead.org>
>>
>>
>> We recently encountered a strange issue where UBIFS recovery is
>> suddenly taking a lot longer than usual (v3.7-based kernel).  In our
>> case, recovery took about 30 seconds.  This caused a serious problem
>> because our watchdog timer was set to 15 seconds, effectively
>> rendering the devices unbootable.
>>
>> Does anybody know what triggers this slow recovery mode?  Also, how
>> can we calculate a worst-case recovery time for a given flash
>> chip-size.

Slowness is probably caused by trying to make free space.  Was the file
system very full?  A smaller journal might help.

Probably the only way to determine worst-case recovery time is to test it.
Worst case conditions are likely to be a full journal and a nearly full file
system.

Ideally you want to capture a file system image that exhibits the slow
behaviour, then you can enable debug messages and see what it is doing.

>>
>> Thanks,
>>
>>   --david
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 

Re: UBIFS recovery taking too long

[David Mosberger-Tang]

Adrian,

On Tue, Dec 10, 2013 at 12:25 AM, Adrian Hunter <adrian.hunter@intel.com> wrote:

> Slowness is probably caused by trying to make free space.  Was the file
> system very full?  A smaller journal might help.

That's possible.  We have not been able to reproduce the issue in
house but a customer had it happen on several devices in a row (3 or 4
of them, I believe).  I don't think we are very sensitive to how long
it takes we just need to know an upper bound on the recovery time.

> Probably the only way to determine worst-case recovery time is to test it.
> Worst case conditions are likely to be a full journal and a nearly full file
> system.

I have not found a lot of UBIFS stresstest tools or even tools to
display what state UBIFS is in.  Can you point me to something?  I
assume lots of small writes would fill up the journal?  We'd be very
happy to test this more but as things stands, it's very hard to tell
how full the filesystem even is (due to compression) and I'm not sure
how to check on the state of the journal.

> Ideally you want to capture a file system image that exhibits the slow
> behaviour, then you can enable debug messages and see what it is doing.

OK, the two devices we've seen this initially have been restored so we
can't find out from those.
However, the customer affected by this issue may have another device
or two they haven't sent
back yet.  We might be able to snapshot the root filesystem on one of those.

  --david

>>>
>>> Thanks,
>>>
>>>   --david
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>