UBIFS trouble

UBIFS trouble

[Daniel Mack]

(re-posting with the correct linux-mtd ML address. Sorry)

Hey,

I'm running the latest git snapshot of the Linux kernel on a PXA3xx
based board. Connected to the PXA3xx is a 128MiB Nymonix NAND chip
which I've set to enourmously relaxed timings for now (factor 2 on all
values).

'nandtest -p 100 /dev/mtd3' runs for hours without any trouble and
succeeds every test. However, when I create an UBIFS on that mtd and
start bonnie to stress-test the filesystem layer, I keep on seeing
messages like this:

[  382.374993] UBIFS error (pid 1133): ubifs_read_node: bad node at LEB 58:106008
[  382.382178] UBIFS error (pid 1133): do_readpage: cannot read page 38693 of inode 66, error -22
[  382.390968] timed out writing command
[  382.394830] UBI error: ubi_io_read: error -74 while reading 76 bytes from PEB 245:105736, read 76 bytes
[  382.404186] UBIFS error (pid 1135): try_read_node: cannot read node type 1 from LEB 68:101640, error -74
[  382.413698] UBIFS error (pid 1135): ubifs_read_node: bad node type (255 but expected 1)
[  382.421718] UBIFS error (pid 1135): ubifs_read_node: bad node at LEB 68:101640
[  382.428941] UBIFS error (pid 1135): do_readpage: cannot read page 53753 of inode 66, error -22
[  648.164435] UBIFS error (pid 1138): ubifs_read_node: bad node type (70 but expected 9)
[  648.172317] UBIFS error (pid 1138): ubifs_read_node: bad node at LEB 110:112896
[  648.179677] UBIFS error (pid 1138): do_readpage: cannot read page 19432 of inode 32838, error -22
[  648.188650] timed out writing command
[  648.192379] UBI error: ubi_io_read: error -74 while reading 81 bytes from PEB 738:80616, read 81 bytes
[  648.201732] UBIFS error (pid 1136): try_read_node: cannot read node type 1 from LEB 72:76520, error -74
[  648.211137] UBIFS error (pid 1136): ubifs_read_node: bad node type (255 but expected 1)
[  648.219115] UBIFS error (pid 1136): ubifs_read_node: bad node at LEB 72:76520
[  648.226231] UBIFS error (pid 1136): do_readpage: cannot read page 26131 of inode 32838, error -22

Once in awhile only and not frequently, but it still scares me. Is there
anything I should consider? Any more information I can provide for
further debugging?

(btw - it happens with the timings found in the datasheet as well, I
just extended them to exclude timing issues as culprit.)

I'd like to use that filesystem in a device, but I need to get it to a
more stable point for that.

Many thanks,
Daniel

Re: UBIFS trouble

[Adrian Hunter]

Daniel Mack wrote:
> (re-posting with the correct linux-mtd ML address. Sorry)
> I'm running the latest git snapshot of the Linux kernel on a PXA3xx
> based board. Connected to the PXA3xx is a 128MiB Nymonix NAND chip
> which I've set to enourmously relaxed timings for now (factor 2 on all
> values).
> 
> 'nandtest -p 100 /dev/mtd3' runs for hours without any trouble and
> succeeds every test. However, when I create an UBIFS on that mtd and
> start bonnie to stress-test the filesystem layer, I keep on seeing
> messages like this:
> 
> [  382.374993] UBIFS error (pid 1133): ubifs_read_node: bad node at LEB 58:106008
> [  382.382178] UBIFS error (pid 1133): do_readpage: cannot read page 38693 of inode 66, error -22
> [  382.390968] timed out writing command
> [  382.394830] UBI error: ubi_io_read: error -74 while reading 76 bytes from PEB 245:105736, read 76 bytes
> [  382.404186] UBIFS error (pid 1135): try_read_node: cannot read node type 1 from LEB 68:101640, error -74
> [  382.413698] UBIFS error (pid 1135): ubifs_read_node: bad node type (255 but expected 1)
> [  382.421718] UBIFS error (pid 1135): ubifs_read_node: bad node at LEB 68:101640
> [  382.428941] UBIFS error (pid 1135): do_readpage: cannot read page 53753 of inode 66, error -22
> [  648.164435] UBIFS error (pid 1138): ubifs_read_node: bad node type (70 but expected 9)
> [  648.172317] UBIFS error (pid 1138): ubifs_read_node: bad node at LEB 110:112896
> [  648.179677] UBIFS error (pid 1138): do_readpage: cannot read page 19432 of inode 32838, error -22
> [  648.188650] timed out writing command
> [  648.192379] UBI error: ubi_io_read: error -74 while reading 81 bytes from PEB 738:80616, read 81 bytes
> [  648.201732] UBIFS error (pid 1136): try_read_node: cannot read node type 1 from LEB 72:76520, error -74
> [  648.211137] UBIFS error (pid 1136): ubifs_read_node: bad node type (255 but expected 1)
> [  648.219115] UBIFS error (pid 1136): ubifs_read_node: bad node at LEB 72:76520
> [  648.226231] UBIFS error (pid 1136): do_readpage: cannot read page 26131 of inode 32838, error -22
> 
> Once in awhile only and not frequently, but it still scares me. Is there
> anything I should consider? Any more information I can provide for
> further debugging?
> 
> (btw - it happens with the timings found in the datasheet as well, I
> just extended them to exclude timing issues as culprit.)
> 
> I'd like to use that filesystem in a device, but I need to get it to a
> more stable point for that.


Try the NAND tests in drivers/mtd/tests.

Re: UBIFS trouble

[Daniel Mack]

On Thu, Jul 09, 2009 at 09:00:42PM +0300, Adrian Hunter wrote:
>> [  648.188650] timed out writing command
>> [  648.192379] UBI error: ubi_io_read: error -74 while reading 81 bytes from PEB 738:80616, read 81 bytes
>> [  648.201732] UBIFS error (pid 1136): try_read_node: cannot read node type 1 from LEB 72:76520, error -74
>> [  648.211137] UBIFS error (pid 1136): ubifs_read_node: bad node type (255 but expected 1)
>> [  648.219115] UBIFS error (pid 1136): ubifs_read_node: bad node at LEB 72:76520
>> [  648.226231] UBIFS error (pid 1136): do_readpage: cannot read page 26131 of inode 32838, error -22
>>

[...]

> Try the NAND tests in drivers/mtd/tests.

Hmm. The torture test indees shows a problem now:

sh-3.2# modprobe mtd_torturetest dev=3
[  386.367730] 
[  386.369296] =================================================
[  386.375258] mtd_torturetest: Warning: this program is trying to wear out your flash, stop it if this is not wanted.
[  386.385811] mtd_torturetest: MTD device: 3
[  386.390021] mtd_torturetest: torture 32 eraseblocks (8-39) of mtd3
[  386.396238] mtd_torturetest: write verify enabled
[  388.675993] double bit error @ page 00000b79
[  388.680406] timed out writing command
[  388.687925] mtd_torturetest: error -74 while reading EB 36, read 131072
[  388.694536] mtd_torturetest: verify failed for 0xFF... pattern
[  388.700364] mtd_torturetest: finished after 0 erase cycles
[  388.705885] mtd_torturetest: error -74 occurred during torturing
[  388.711908] =================================================
modprobe: Failed to load module mtd_torturetest: Bad message.

Any hint what to debug?

Thanks,
Daniel

Re: UBIFS trouble

[Daniel Mack]

On Thu, Jul 09, 2009 at 08:11:14PM +0200, Daniel Mack wrote:
> > Try the NAND tests in drivers/mtd/tests.
> 
> Hmm. The torture test indees shows a problem now:
> 
> sh-3.2# modprobe mtd_torturetest dev=3
> [  386.367730] 
> [  386.369296] =================================================
> [  386.375258] mtd_torturetest: Warning: this program is trying to wear out your flash, stop it if this is not wanted.
> [  386.385811] mtd_torturetest: MTD device: 3
> [  386.390021] mtd_torturetest: torture 32 eraseblocks (8-39) of mtd3
> [  386.396238] mtd_torturetest: write verify enabled
> [  388.675993] double bit error @ page 00000b79
> [  388.680406] timed out writing command
> [  388.687925] mtd_torturetest: error -74 while reading EB 36, read 131072
> [  388.694536] mtd_torturetest: verify failed for 0xFF... pattern
> [  388.700364] mtd_torturetest: finished after 0 erase cycles
> [  388.705885] mtd_torturetest: error -74 occurred during torturing
> [  388.711908] =================================================
> modprobe: Failed to load module mtd_torturetest: Bad message.

Interesting. I think I solved that issue - it seems to be related to the
selected CPU govenor. Switching from 'conservative' to 'performance'
makes the torture test pass.

I'll discuss that on the ARM mailing list.

Thanks,
Daniel

UBIFS trouble

[Juergen Borleis]

Hi,

with an older barebox (2015.08) I can see in my NOR:

barebox@board: ls -l /mnt/nor0.file-system.ubi.rootfs/boot/
-rw-r--r--    1686716 System.map-4.1.0-none-board
-rw-r--r--      67615 config-4.1.0-none-board
lrwxrwxrwx         30 initrd.img -> initrd.img-4.1.0-none-board
-rw-r--r--    2180972 initrd.img-4.1.0-none-board
lrwxrwxrwx         67 oftree -> ../usr/lib/linux-image-4.1.0-none-board/omap3-mfr-board.dtb
-rw-r--r--    3375248 vmlinuz-4.1.0-none-board
-rw-r--r--    3426056 vmlinuz-ptx-board
lrwxrwxrwx         15 zImage -> vmlinuz-ptx-board

A 4.1 Linux kernel sees this filesystem in the same manner:

root@board:~# ll /boot/
total 10496
-rw-r--r-- 1 root root 1686716 Jul 11  2016 System.map-4.1.0-none-board
-rw-r--r-- 1 root root   67615 Jul 11  2016 config-4.1.0-none-board
lrwxrwxrwx 1 root root      30 Oct 24  2016 initrd.img -> initrd.img-4.1.0-none-board
-rw-r--r-- 1 root root 2180972 Oct 24  2016 initrd.img-4.1.0-none-board
lrwxrwxrwx 1 root root      67 Oct 24  2016 oftree -> ../usr/lib/linux-image-4.1.0-none-board/omap3-mfr-board.dtb
-rw-r--r-- 1 root root 3375248 Jul 11  2016 vmlinuz-4.1.0-none-board
-rw-r--r-- 1 root root 3426056 Oct  1  2016 vmlinuz-ptx-board
lrwxrwxrwx 1 root root      15 Oct  1  2016 zImage -> vmlinuz-ptx-board

A fresh barebox (2016.11.0) sees this filesystem differently:

barebox@board:/ ls -l /mnt/nor0.file-system.ubi.rootfs/boot/
-rw-r--r--        1686716 System.map-4.1.0-none-board
-rw-r--r--          67615 config-4.1.0-none-board
-rw-r--r--        2180972 initrd.img
-rw-r--r--        2180972 initrd.img-4.1.0-none-board
/mnt/nor0.file-system.ubi.rootfs/boot//oftree: No such file or directory
-rw-r--r--        3375248 vmlinuz-4.1.0-none-board
-rw-r--r--        3426056 vmlinuz-ptx-board
-rw-r--r--        3426056 zImage

Any idea?

jb

-- 
Pengutronix e.K.                              | Juergen Borleis             |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: UBIFS trouble

[Juergen Borleis]

FYI: soft links are broken since 2016.05.0.

jb

-- 
Pengutronix e.K.                              | Juergen Borleis             |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |
_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: UBIFS trouble

[Sascha Hauer]

On Tue, Nov 29, 2016 at 03:04:41PM +0100, Juergen Borleis wrote:
> FYI: soft links are broken since 2016.05.0.

Most likely this goes down to:

a63059d ubifs: update implementation from u-boot v2016.03

U-Boot resolves links in each filesystem implementation whereas barebox
handles links natively. Have a look at a63059d, identify the link
resolving code and revert this to the previously working barebox code.

Sascha


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox