ubifs panic with 2.6.39 stable - followup

ubifs panic with 2.6.39 stable - followup

[Brad Parker]

[sorry for the duplicate post; I tried to post a followup with just
  the url to the pastebin log but for some reason  it got stuck
  waiting for the moderator]

Running and older 2.6.31 kernel with UBIFS I see a panic which appears
to be during recovery of a bad block. The root fs (which is UBIFS)
won't mount and the kernel panics.

I upgraded the kernel 2.6.39-stable, hoping that would fix the problem,
as I noticed a lot of recovery fixes had gone in.  It still panics;
it appears the recovery fails.

I rebooted with "ignore_loglevel" and the output is here:

     http://pastebin.com/ETJjP4uw

The board is essentially an Olimex SAM9-L9260, with Samsung NAND.

I'm curious if this looks familiar and if it might be fixed post 2.5.39

thanks for any insight.

-brad

UBIFS: recovery needed
UBI error: ubi_io_read: error -74 (ECC error) while reading 126976 bytes 
from PEB 2970:4096, read 126976 bytes
UBIFS error (pid 1): ubifs_check_node: bad CRC: calculated 0xf510fb95, 
read 0x4f0a3196
UBIFS error (pid 1): ubifs_check_node: bad node at LEB 3589:45128
         magic          0x6101831
         crc            0x4f0a3196
         node_type      0 (inode node)
         group_type     2 (last of node group)
         sqnum          69874250
         len            160
         key            (23417, inode)
         creat_sqnum    230076
         size           608
         nlink          2
         atime          1306744746.0
         mtime          1316662493.0
         ctime          1316662493.0
         uid            0
         gid            0
         mode           16832
         flags          0x1
         xattr_cnt      2147483648
         xattr_size     0
         xattr_names    0
         compr_type     0x0
         data len       0
...
UBIFS error (pid 1): ubifs_recover_leb: bad node
UBIFS error (pid 1): ubifs_scanned_corruption: corruption at LEB 3589:45128
UBIFS error (pid 1): ubifs_scanned_corruption: first 8192 bytes from LEB 
3589:45128
...
UBIFS error (pid 1): ubifs_recover_leb: LEB 3589 scanning failed

Re: ubifs panic with 2.6.39 stable - followup

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

On Wed, 2012-01-04 at 06:39 -0500, Brad Parker wrote:
> [sorry for the duplicate post; I tried to post a followup with just
>   the url to the pastebin log but for some reason  it got stuck
>   waiting for the moderator]
> 
> Running and older 2.6.31 kernel with UBIFS I see a panic which appears
> to be during recovery of a bad block. The root fs (which is UBIFS)
> won't mount and the kernel panics.
> 
> I upgraded the kernel 2.6.39-stable, hoping that would fix the problem,
> as I noticed a lot of recovery fixes had gone in.  It still panics;
> it appears the recovery fails.
> 
> I rebooted with "ignore_loglevel" and the output is here:
> 
>      http://pastebin.com/ETJjP4uw
> 
> The board is essentially an Olimex SAM9-L9260, with Samsung NAND.
> 
> I'm curious if this looks familiar and if it might be fixed post 2.5.39
> 
> thanks for any insight.

Would you please also enable scanning, mount, and recovery debugging
messages and provide the log? 

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_how_debug

But in newer kernels this has been turned into dynamic debug, see
Documentation/dynamic-debug-howto.txt

(Gosh, I have to update the web site!)

In older kernels you can do this via the configuration menu, in newer
kernel this has been turned into Documentation/dynamic-debug-howto.txt

To use them at boot time, use ddebug_query=...

You need to use 3 format string to show "UBIFS DBG mnt" and the like.

I think, but did not verify, you need to add kernel boot 3 parameters:

ddebug_query="format UBIFS DBG rcvry" ddebug_query="format UBIFS DBG
mnt" ddebug_query="format UBIFS DBG scan"

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubifs panic with 2.6.39 stable - followup

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

On Wed, 2012-01-04 at 06:39 -0500, Brad Parker wrote:
> [sorry for the duplicate post; I tried to post a followup with just
>   the url to the pastebin log but for some reason  it got stuck
>   waiting for the moderator]
> 
> Running and older 2.6.31 kernel with UBIFS I see a panic which appears
> to be during recovery of a bad block. The root fs (which is UBIFS)
> won't mount and the kernel panics.
> 
> I upgraded the kernel 2.6.39-stable, hoping that would fix the problem,
> as I noticed a lot of recovery fixes had gone in.  It still panics;
> it appears the recovery fails.
> 
> I rebooted with "ignore_loglevel" and the output is here:
> 
>      http://pastebin.com/ETJjP4uw
> 
> The board is essentially an Olimex SAM9-L9260, with Samsung NAND.
> 
> I'm curious if this looks familiar and if it might be fixed post 2.5.39
> 
> thanks for any insight.
> 
> -brad
> 
> UBIFS: recovery needed
> UBI error: ubi_io_read: error -74 (ECC error) while reading 126976 bytes 
> from PEB 2970:4096, read 126976 bytes
> UBIFS error (pid 1): ubifs_check_node: bad CRC: calculated 0xf510fb95, 
> read 0x4f0a3196

So there is a corrupted inode node, and UBIFS believes it has been
corrupted not because of a power cut. I do not know why it is corrupted,
but if you use MLC then this may be related to the paired pages problem.

Anyway, there is another issue I see from the dump. Even if you somehow
make the node good again, UBIFS will still fail saying something like
"corrupt empty space". Look at line 292 in your pastebin:

ffffffff ffffffff ffffffff ffffffff ffffffff ffdfffff ffffffff ffffffff

See that little "d"? It means that the empty space has a bit-flip. The
question is why? Unstable bit I guess? Does your NAND driver / HW
provides ECC protection for empty pages?

Anyway, currently UBIFS cannot handle these situation. Someone needs to
do this - I can assist by reviewing and advising.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]