UBIFS: power cut test on 2.6.35 + NOR

UBIFS: power cut test on 2.6.35 + NOR

[enrico benetti]

Hi All,
I'd need a hint on the best way to go on with my development.

I'm working on a custom target based on 2.6.35.3 kernel with NOR device.

UBIFS R/W partitions seem not robust against power cut tests: I'm using
integck 1.4.9 and I can have unrecoverable errors on tested volume when
performing real power cuts.

First of all, I've done several MTD stress tests to validate the low
level driver.
I've checked git repos, and
http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/  
<http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/shortlog>
is updated to 08/2012 (that is approx 3.7.1 kernel revision) and no more
supported.

  From several discussions on the list I think the best way to solve (or
at least to have a better behaviour with ubi/ubifs) could be to get the
patches from this repo and merge them with my sources.

Do you agree or is it better to do more investigation on current issues?

Thanks in advance for your support,
Enrico

Re: UBIFS: power cut test on 2.6.35 + NOR

[Norbert van Bolhuis]

On 06/27/13 08:44, enrico benetti wrote:
> Hi All,
> I'd need a hint on the best way to go on with my development.
>
> I'm working on a custom target based on 2.6.35.3 kernel with NOR device.
>
> UBIFS R/W partitions seem not robust against power cut tests: I'm using
> integck 1.4.9 and I can have unrecoverable errors on tested volume when
> performing real power cuts.
>
> First of all, I've done several MTD stress tests to validate the low
> level driver.
> I've checked git repos, and
> http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/ <http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/shortlog>
> is updated to 08/2012 (that is approx 3.7.1 kernel revision) and no more
> supported.
>
>  From several discussions on the list I think the best way to solve (or
> at least to have a better behaviour with ubi/ubifs) could be to get the
> patches from this repo and merge them with my sources.
>
> Do you agree or is it better to do more investigation on current issues?
>
> Thanks in advance for your support,
> Enrico
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
>

yes, use http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git to update.

it has several "NOR power cut" fixes, for example:

http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/commit/152780c61d65b0cc5cd5cc974e926a9c47534d5e
http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/commit/9258edaca1d7b36abd0e23661579c35fe9c4f00f
http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/commit/e5d7669e2fa0797c0c110fa9bf3e51b013ba399b

which I doubt are in 2.6.35.3

Btw. we used the (no longer available) http://git.infradead.org/users/dedekind/ubifs-v2.6.29.git
(that has been kept up-to-date with mainline till June 2011) to update our sources, and we see
no NOR power cut issues.

---
N. van Bolhuis.

RE: UBIFS: power cut test on 2.6.35 + NOR

[Mats Kärrman]

Hi Enrico,

> From: linux-mtd [linux-mtd-bounces@lists.infradead.org] on behalf of enrico benetti [enrico.benetti@bluewind.it]
> Sent: Thursday, June 27, 2013 8:44 AM
> To: linux-mtd@lists.infradead.org
> Subject: UBIFS: power cut test on 2.6.35 + NOR

> I've checked git repos, and
> http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/
> <http://git.infradead.org/users/dedekind/ubifs-v2.6.35.git/shortlog>
> is updated to 08/2012 (that is approx 3.7.1 kernel revision) and no more
> supported.

I'm using 2.6.35.14 (last 2.6.35) with all UBIFS patches from the
repo you mentioned above. I have also added/ported all patches from
mainline 3.9 stable (excluding those only dealing with debug prints etc).

Still I have some minor (I hope) issues after running integck power-cut
tests for several days. They manifest in errors on xattr nodes.
There are some related patches in discussion, see:
http://thread.gmane.org/gmane.linux.kernel.lsm/18746
http://thread.gmane.org/gmane.linux.drivers.mtd/47021

BR // Mats

Re: UBIFS: power cut test on 2.6.35 + NOR

[enrico benetti]

> I'm using 2.6.35.14 (last 2.6.35) with all UBIFS patches from the
> repo you mentioned above. I have also added/ported all patches from
> mainline 3.9 stable (excluding those only dealing with debug prints etc).
>
> Still I have some minor (I hope) issues after running integck power-cut
> tests for several days. They manifest in errors on xattr nodes.
> There are some related patches in discussion, see:
> http://thread.gmane.org/gmane.linux.kernel.lsm/18746
> http://thread.gmane.org/gmane.linux.drivers.mtd/47021
>
>
Hi Mats, Norbert,
thanks for your kind feedbacks.

Mats, are you working on a NOR-based system or are you referring to NAND?

I'm going to merge from 2.6.35 back-port repo and I'll let you know.

Regards,
E

RE: UBIFS: power cut test on 2.6.35 + NOR

[Mats Kärrman]

>From: linux-mtd [linux-mtd-bounces@lists.infradead.org] on behalf of enrico benetti [enrico.benetti@bluewind.it]
>Sent: Thursday, June 27, 2013 2:56 PM
>To: linux-mtd@lists.infradead.org
>Subject: Re: UBIFS: power cut test on 2.6.35 + NOR
>
>> I'm using 2.6.35.14 (last 2.6.35) with all UBIFS patches from the
>> repo you mentioned above. I have also added/ported all patches from
>> mainline 3.9 stable (excluding those only dealing with debug prints etc).
>>
>> Still I have some minor (I hope) issues after running integck power-cut
>> tests for several days. They manifest in errors on xattr nodes.
>> There are some related patches in discussion, see:
>> http://thread.gmane.org/gmane.linux.kernel.lsm/18746
>> http://thread.gmane.org/gmane.linux.drivers.mtd/47021
>>
>>
>Hi Mats, Norbert,
>thanks for your kind feedbacks.
>
>Mats, are you working on a NOR-based system or are you referring to NAND?

NOR

>
>I'm going to merge from 2.6.35 back-port repo and I'll let you know.
>
>Regards,
>E

Re: UBIFS: power cut test on 2.6.35 + NOR

[enrico benetti]

>> I'm going to merge from 2.6.35 back-port repo and I'll let you know.
>>
Hi All,
I've merged down all 2.6.35 git repo patches and started again power cut
tests with integck.

After a whirl, I had an unrecoverable error during volume mount:

[   18.060413] UBI: attaching mtd7 to ubi0
[   18.064335] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[   18.070608] UBI: logical eraseblock size:    130944 bytes
[   18.076025] UBI: smallest flash I/O unit:    1
[   18.080472] UBI: VID header offset:          64 (aligned 64)
[   18.086145] UBI: data offset:                128
[   18.121404] UBI: attached mtd7 to ubi0
[   18.125167] UBI: MTD device name:            "wfs"
[   18.129962] UBI: MTD device size:            10 MiB
[   18.137534] UBI: number of good PEBs:        80
[   18.142129] UBI: number of bad PEBs:         0
[   18.146576] UBI: max. allowed volumes:       128
[   18.151196] UBI: wear-leveling threshold:    4096
[   18.155916] UBI: number of internal volumes: 1
[   18.160362] UBI: number of user volumes:     1
[   18.164818] UBI: available PEBs:             0
[   18.169264] UBI: total number of reserved PEBs: 80
[   18.174071] UBI: number of PEBs reserved for bad PEB handling: 0
[   18.180083] UBI: max/mean erase counter: 527/487
[   18.184741] UBI: image sequence number: 0
[   18.188766] UBI: background thread "ubi_bgt0d" started, PID 1219
UBI device number 0, total 80 LEBs (10475520 bytes, 10.0 MiB), available
0 LEBs (0 bytes), LEB size 130944 bytes (127.9 KiB)
[   20.391279] MTD do_write_buffer(): software timeout
[   20.452081] UBI error: nor_erase_prepare: cannot invalidate PEB 25,
write returned -5 read returned 2

Digging the web, I've found several discussions on write-timeout with
NOR devices (cfr.http://lists.infradead.org/pipermail/linux-mtd/2013-June/047177.html),
but this seems not to be the root cause.

Disabling background thread I was still not able to mount the volume,
but this patch (http://git.infradead.org/ubifs-2.6.git/commit/6fb4374f6b1b3932f3acfe9d353568d3d8599cad)
solved and at least mounting is ok.

As soon as write operations are performed on the filesystem again
there's a failure: it seems sector erasure is not properly
handled/requested.

Flash dump shows PEB 25 this way (= partially erased), so it turns out write operation can't work:

------------------------------------------------------------
00320040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
|................|
*
003201e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
|................|
*
00320220  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
|................|
*
003203e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
|................|
*
00320820  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
|................|
*
------------------------------------------------------------

Please note that I'm using a 'slow' device (M29EW), with a maximum
sector erasure time of 4 secs, and this seems to stress asynchronous
operations with ubifs and integck test.

I'll go on with commit log analysis to check if this kind of issue is
already solved/discussed.

Full partition erasure has solved mount/fs issue, as expected.

Regards,
E

Re: UBIFS: power cut test on 2.6.35 + NOR

[enrico benetti]

> Please note that I'm using a 'slow' device (M29EW), with a maximum
> sector erasure time of 4 secs, and this seems to stress asynchronous
> operations with ubifs and integck test.
>
> I'll go on with commit log analysis to check if this kind of issue is
> already solved/discussed.
>
Hi,
at least two commits for M29EW NOR chips within MTD repo...trying to 
merge and then check again.

E

Re: UBIFS: power cut test on 2.6.35 + NOR

[Brian Norris]

(Adding Huang, who has also debugged (different) problems with this NOR driver)

Hi Enrico,

On Fri, Jun 28, 2013 at 6:36 AM, enrico benetti
<enrico.benetti@bluewind.it> wrote:
> [   20.391279] MTD do_write_buffer(): software timeout
> [   20.452081] UBI error: nor_erase_prepare: cannot invalidate PEB 25,
> write returned -5 read returned 2
>
> Digging the web, I've found several discussions on write-timeout with
> NOR devices
> (cfr.http://lists.infradead.org/pipermail/linux-mtd/2013-June/047177.html),
> but this seems not to be the root cause.

I'm curious: what makes you think this has a different root cause?

I'm still working on debugging the do_write_buffer timeout on my
systems. I'd recommend trying to increase the timeout in
do_write_buffer(), just to see if your kernel isn't waiting long
enough. (In my case, I can pretty well show that while the code *says*
it's waiting for a whole jiffy -- at least 1ms -- it is in fact
waiting less than that.)

I believe Huang's resolution for his problem was actually a problem
with the flash part itself.

Brian

Re: UBIFS: power cut test on 2.6.35 + NOR

[Huang Shijie]

于 2013年06月29日 10:08, Brian Norris 写道:
> (Adding Huang, who has also debugged (different) problems with this NOR driver)
>
> Hi Enrico,
>
> On Fri, Jun 28, 2013 at 6:36 AM, enrico benetti
> <enrico.benetti@bluewind.it>  wrote:
>> [   20.391279] MTD do_write_buffer(): software timeout
>> [   20.452081] UBI error: nor_erase_prepare: cannot invalidate PEB 25,
>> write returned -5 read returned 2
>>
>> Digging the web, I've found several discussions on write-timeout with
>> NOR devices
>> (cfr.http://lists.infradead.org/pipermail/linux-mtd/2013-June/047177.html),
>> but this seems not to be the root cause.
> I'm curious: what makes you think this has a different root cause?
>
> I'm still working on debugging the do_write_buffer timeout on my
> systems. I'd recommend trying to increase the timeout in
> do_write_buffer(), just to see if your kernel isn't waiting long
> enough. (In my case, I can pretty well show that while the code *says*
> it's waiting for a whole jiffy -- at least 1ms -- it is in fact
> waiting less than that.)
>
> I believe Huang's resolution for his problem was actually a problem
> with the flash part itself.
yes.
For my timeout, Micron confirmed that there is a silicon bug in this 
type of NOR.


thanks
Huang Shijie