JFFS2: file contents in case of data CRC error

JFFS2: file contents in case of data CRC error

[llandre]

I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a 
Data CRC error but function nand_correct_data never returns -1, so I 
assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written 
to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" 
where all bytes are 0.
Anybody can help me about understanding if this is the expected 
behaviour of JFFS2?
Thanks in advance.


bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0
mtdblock_open
ok
bash-2.05# cp -v /mnt/nand0/f.img /tmp/
jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read 
0x80b8997f, calculated 0x57f0ea8e
`/mnt/nand0/f.img' -> `/tmp/f.img'

-- 
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

RE: JFFS2: file contents in case of data CRC error

[Korolev, Alexey]

Hi 
Could you please give information about how you can get this error? I
need to know use case. 

>I have a JFFS2 partition on 32MByte NAND device.
>When reading a specific file - see below for details - JFFS2 reports a 
>Data CRC error but function nand_correct_data never returns -1, so I 
>assume ECC algorithm is able to correct errors.
>However the file is not equal to the original one that has been written

>to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" 
>where all bytes are 0.

It is Ok. NAND error correction could return that data is correct even
if it is not correct. It finds the issues with guarantee only in case of
1 or 2 bits errors.  

>Anybody can help me about understanding if this is the expected 
>behaviour of JFFS2?
>Thanks in advance.


>bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0
>mtdblock_open
>ok
>bash-2.05# cp -v /mnt/nand0/f.img /tmp/
>jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read 
>0x80b8997f, calculated 0x57f0ea8e
>`/mnt/nand0/f.img' -> `/tmp/f.img'

This is a bad error! How obtained you this? What you did before getting
this?

Alexey
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: 125252, Moscow, Russian Federation, 
Chapayevsky Per, 14.

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Jffss2_write_super erasing all the blocks

[Nikhil Bansal (nikbansa)]

Hi,

I have a question about JFFS2 fs. In my setup, I am running JFFS2 on AMD
Spansion flash (NOR flash). What I noticed was that whenever I erase a
file from my flash, I see that jffs2_write_super function gets called
which in turn calls jffs2_erase_pending_blocks(0) which goes ahead and
deletes all the pending blocks. While this erase is going on super block
is locked and we can not perform any other operation with JFFS2. 

Can someone please explain the logic behind this or if there is
something I am missing here?

Thanks,
Nikhil 

-----Original Message-----
From: linux-mtd-bounces@lists.infradead.org
[mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of llandre
Sent: Thursday, January 31, 2008 8:54 AM
To: linux-mtd@lists.infradead.org
Subject: JFFS2: file contents in case of data CRC error

I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a
Data CRC error but function nand_correct_data never returns -1, so I
assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written
to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" 
where all bytes are 0.
Anybody can help me about understanding if this is the expected
behaviour of JFFS2?
Thanks in advance.


bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0 mtdblock_open
ok bash-2.05# cp -v /mnt/nand0/f.img /tmp/
jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read
0x80b8997f, calculated 0x57f0ea8e `/mnt/nand0/f.img' -> `/tmp/f.img'

--
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: JFFS2: file contents in case of data CRC error

[Matt Reimer]

On Jan 31, 2008 8:54 AM, llandre <r&d2@dave-tech.it> wrote:
> I have a JFFS2 partition on 32MByte NAND device.
> When reading a specific file - see below for details - JFFS2 reports a
> Data CRC error but function nand_correct_data never returns -1, so I
> assume ECC algorithm is able to correct errors.
> However the file is not equal to the original one that has been written
> to NAND. In fact, in the middle of the file, I see a 4-kByte "hole"
> where all bytes are 0.
> Anybody can help me about understanding if this is the expected
> behaviour of JFFS2?
> Thanks in advance.

I saw exactly this problem on an s3c2440 board back in October. It
turned out that the ECC correction code in the s3c2410 driver was
incorrect in a couple of ways, resulting in corrupt data being passed
up to JFFS2. I'm not sure, but I think what happened from there is
that JFFS2 failed to decompress a block and instead of returning an
error it left the page zeroed. You can use jffs2dump to correlate the
offset reported in the CRC error with the problematic inode. Check the
thread "bit flip" in October 2007.

What platform is this on?

Matt

Re: JFFS2: file contents in case of data CRC error

[Jamie Lokier]

Matt Reimer wrote:
> On Jan 31, 2008 8:54 AM, llandre <r&d2@dave-tech.it> wrote:
> > I have a JFFS2 partition on 32MByte NAND device.
> > When reading a specific file - see below for details - JFFS2 reports a
> > Data CRC error but function nand_correct_data never returns -1, so I
> > assume ECC algorithm is able to correct errors.
> > However the file is not equal to the original one that has been written
> > to NAND. In fact, in the middle of the file, I see a 4-kByte "hole"
> > where all bytes are 0.
> > Anybody can help me about understanding if this is the expected
> > behaviour of JFFS2?
> > Thanks in advance.
> 
> I saw exactly this problem on an s3c2440 board back in October. It
> turned out that the ECC correction code in the s3c2410 driver was
> incorrect in a couple of ways, resulting in corrupt data being passed
> up to JFFS2. I'm not sure, but I think what happened from there is
> that JFFS2 failed to decompress a block and instead of returning an
> error it left the page zeroed. You can use jffs2dump to correlate the
> offset reported in the CRC error with the problematic inode. Check the
> thread "bit flip" in October 2007.

I've seen similar things with cfi_cmdset_0002 on NOR flash, with Linux
2.4.26.  Occasional bit flips on reading flash, resulting in zero
blocks in the read file from JFFS2 - no errors reported to the
application.

I have reason to think our board timings were ever so slightly out of
spec, and haven't seen it in newer board revisions.

-- Jamie

Re: JFFS2: file contents in case of data CRC error

[llandre]

Alexey, Matt and Jamie,

thank you all for the precious information.

> Could you please give information about how you can get this error? I
> need to know use case. 
The issue has been reported by a customer working on PPC405EP-based 
system running linux 2.4.20.
The device under test has run a temperature cycle test. The temperature 
has been varied in the whole industrial range (-40/+85 deg C). After 
completing the test, the files stored on NAND have been read back and 
compared to the original ones - written on NAND before starting the 
temperature test - and the difference in the file f.img previously 
described arose.

>> to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" 
>> where all bytes are 0.
> 
> It is Ok. NAND error correction could return that data is correct even
> if it is not correct. It finds the issues with guarantee only in case of
> 1 or 2 bits errors.  
About this point I agree with catboat 
(http://article.gmane.org/gmane.linux.drivers.mtd/19463). In my opinion 
in this case JFFS2 should report an error code != 0 in order to warn the 
userland that something did not work perfectly.

>> bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0
>> mtdblock_open
>> ok
>> bash-2.05# cp -v /mnt/nand0/f.img /tmp/
>> jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read 
>> 0x80b8997f, calculated 0x57f0ea8e
>> `/mnt/nand0/f.img' -> `/tmp/f.img'
> 
> This is a bad error! How obtained you this? What you did before getting
> this?
See previous description.




-- 
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

Re: JFFS2: file contents in case of data CRC error

[llandre]

> up to JFFS2. I'm not sure, but I think what happened from there is
> that JFFS2 failed to decompress a block and instead of returning an
> error it left the page zeroed. You can use jffs2dump to correlate the
> offset reported in the CRC error with the problematic inode. Check the
> thread "bit flip" in October 2007.
Thanks a lot for this advice. The system is based on linux 2.4.20 and 
ELDK 2.0.2 so the jffs2dump is not available out-of-the-box. I'll try to 
build recent mtd-utils package.

> What platform is this on?
PowerPC 405EP.


Regards,
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

Re: JFFS2: file contents in case of data CRC error

[llandre]

> I've seen similar things with cfi_cmdset_0002 on NOR flash, with Linux
> 2.4.26.  Occasional bit flips on reading flash, resulting in zero
> blocks in the read file from JFFS2 - no errors reported to the
> application.
Have you found the JFFS2 portion of code that "zeroes" the block before 
passing it to the userland?

Regards,
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

RE: JFFS2: file contents in case of data CRC error

[Korolev, Alexey]

Hi


>> Could you please give information about how you can get this error? I
>> need to know use case. 
> The issue has been reported by a customer working on PPC405EP-based 
> system running linux 2.4.20.
> The device under test has run a temperature cycle test. The
temperature 
> has been varied in the whole industrial range (-40/+85 deg C). After 
> completing the test, the files stored on NAND have been read back and 
> compared to the original ones - written on NAND before starting the 
> temperature test - and the difference in the file f.img previously 
> described arose.

Ahh. The temperature tests. That explains a lot. 
Probably device worked improperly in the test but test did not turn off
device when NAND become unworkable. As result JFFS2 did not detect the
problem of broken CRC on mount but did it on attempt to read the broken
data and printed error message. 

So JFFS2 performed scan on read - found CRC error in node 
-> JFFS2 marked node to be obsolete
-> As result hole in the file has been formed. 
-> You read 0x0 in the middle of the file. 

I think nobody considered such tests before. I am not sure if any file
system may be reliable in such a case with temperature tests.

Unfortunately I have no idea how this issue could be solved :).


>>> to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" 
>>> where all bytes are 0.
>> 
>> It is Ok. NAND error correction could return that data is correct
even
>> if it is not correct. It finds the issues with guarantee only in case
of
>> 1 or 2 bits errors.  
> About this point I agree with catboat 
> (http://article.gmane.org/gmane.linux.drivers.mtd/19463). In my
opinion 
> in this case JFFS2 should report an error code != 0 in order to warn
the 
> userland that something did not work perfectly.
>
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: 125252, Moscow, Russian Federation, 
Chapayevsky Per, 14.

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: JFFS2: file contents in case of data CRC error

[Jamie Lokier]

Korolev, Alexey wrote:
> So JFFS2 performed scan on read - found CRC error in node 
> -> JFFS2 marked node to be obsolete
> -> As result hole in the file has been formed. 
> -> You read 0x0 in the middle of the file. 
> 
> I think nobody considered such tests before. I am not sure if any file
> system may be reliable in such a case with temperature tests.
> 
> Unfortunately I have no idea how this issue could be solved :).

If the corruption makes it impossible to detect the node corresponding
to a block, then indeed how can it be solved?  If the bit flips happen
in the node header, not the data, there really is no way to know that
some data is lost from the right block.

I'm thinking the only way to detect this with high reliability is to
store summaries of the existence of blocks in another part of storage,
with checksums and serial numbers - like some of the latest disk
filesystems begin to.  Then missing nodes are detectable and can
translate to I/O errors in userspace.

The other way, is for apps themselves to store checksums of their
files.  I had to do this, because we were getting occasional zeroed
blocks from JFFS2, and that happened in the middle of executables, so
we had apps which would run and occasionally crash or go wrong because
part of their code contained zeroes.

To avoid random wrong behaviour, I checksummed executable programs
before running them.  It does seem it would be better if the
filesystem offered better integrity or error guarantees.

-- Jamie

Re: JFFS2: file contents in case of data CRC error

[llandre]

> Ahh. The temperature tests. That explains a lot. 
> Probably device worked improperly in the test but test did not turn off
> device when NAND become unworkable. As result JFFS2 did not detect the
> problem of broken CRC on mount but did it on attempt to read the broken
> data and printed error message.
I was told by the customer that before starting the temperature test 
several files have been copied on NAND. During the temperature test the 
processor never accessed these files on NAND. Once the test has been 
completed, the files on NAND have been read back and compared to the 
original ones.

> So JFFS2 performed scan on read - found CRC error in node 
> -> JFFS2 marked node to be obsolete
Sure. This is the code snippet in function jffs2_get_inode_nodes:

if (crc != je32_to_cpu(node.i.data_crc)) {
	printk(KERN_NOTICE "jffs2_get_inode_nodes(): Data CRC failed on 		node
at 0x%08x: Read 0x%08x, calculated 0x%08x\n",
        	ref_offset(ref), je32_to_cpu(node.i.data_crc), crc);
	jffs2_mark_node_obsolete(c, ref);
	spin_lock(&c->erase_completion_lock);
	continue;
	...

> -> As result hole in the file has been formed. 
> -> You read 0x0 in the middle of the file.
I can't find the function that, in case of obsolete node, zeroes the 
chunk of data passed to user space. Can you?

> I think nobody considered such tests before. I am not sure if any file
> system may be reliable in such a case with temperature tests.
Hmm ... I don't think the environmental condition may influence JFFS2 
behaviour. The NAND used in the test is industrial grade so it is 
guaranteed by manufacturer that it works in -40/+85 degC range. JFFS2 
clearly does not anything about environment temperature. On the other 
hand I'm wondering if temperature variation caused multiple bit flipping 
in one NAND block that MTD was not able to correct and this led to the 
problem we are talking about.


-- 
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

Re: JFFS2: file contents in case of data CRC error

[llandre]

>> up to JFFS2. I'm not sure, but I think what happened from there is
>> that JFFS2 failed to decompress a block and instead of returning an
>> error it left the page zeroed. You can use jffs2dump to correlate the
>> offset reported in the CRC error with the problematic inode. Check the
>> thread "bit flip" in October 2007.
> Thanks a lot for this advice. The system is based on linux 2.4.20 and 
> ELDK 2.0.2 so the jffs2dump is not available out-of-the-box. I'll try to 
> build recent mtd-utils package.
Matt,

I managed to built jffs2dump. Here a snippet of the dump it generated:

...
          Inode      node at 0x01d0fc84, totlen 0x00000378, #ino     22, 
version     8, isize    26065, csize      820, dsize     1489, offset 
  24576
Wrong bitmask  at  0x01d0fffc, 0x0000
          Inode      node at 0x01d10000, totlen 0x00000307, #ino     20, 
version   356, isize  1318912, csize      707, dsize     3118, offset 
1315794
          Inode      node at 0x01d10308, totlen 0x000003b0, #ino     20, 
version   357, isize  1323008, csize      876, dsize     4096, offset 
1318912
...
          Inode      node at 0x01d9bdd0, totlen 0x0000006a, #ino     10, 
version     2, isize       38, csize       38, dsize       38, offset 
      0
          Inode      node at 0x01d9be3c, totlen 0x00000044, #ino     20, 
version     1, isize        0, csize        0, dsize        0, offset 
      0
          Dirent     node at 0x01d9be80, totlen 0x0000003f, #pino     4, 
version     4, #ino        20, nsize       23, name f.img
          Inode      node at 0x01d9bec0, totlen 0x0000013a, #ino     20, 
version     2, isize     1049, csize      246, dsize     1049, offset 
      0
Wrong bitmask  at  0x01d9bffc, 0x0000
          Inode      node at 0x01d9c000, totlen 0x00000550, #ino     18, 
version   230, isize   757760, csize     1292, dsize     1292, offset 
756468
...


IIUC the inode number of file f.img is 20 so I have to look for the 
lines including "#ino     20" to understand where the file physically is 
stored on NAND. One of these line is for example:

Inode      node at 0x01d10000, totlen 0x00000307, #ino     20, version 
  356, isize  1318912, csize      707, dsize     3118, offset  1315794

Then I looked at the binary dump of whole NAND device I created as follows:

nanddump /dev/mtd0 nand.bin 0 33554432

However around the offset 0x01d10000 in the file nand.bin all bytes are 
0xFF so it seems there are no data in that area. What offset in the file 
correspond to the physical address 0x01d10000 ?


-- 
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

Re: JFFS2: file contents in case of data CRC error

[llandre]

> I'm thinking the only way to detect this with high reliability is to
> store summaries of the existence of blocks in another part of storage,
> with checksums and serial numbers - like some of the latest disk
> filesystems begin to.
Can you name these file systems?


-- 
llandre

DAVE Electronics System House - R&D Department
web:   http://www.dave.eu
email: r&d2@dave-tech.it

SV: Jffss2_write_super erasing all the blocks

[Joakim Tjernlund]

This is a long standing issue that I tried to address fairly recently.
Please search the archive and you should find some posts/patch from
me about this.

 Jocke

> -----Ursprungligt meddelande-----
> Från: linux-mtd-bounces@lists.infradead.org [mailto:linux-mtd-bounces@lists.infradead.org] För Nikhil
> Bansal (nikbansa)
> Skickat: den 31 januari 2008 20:22
> Till: linux-mtd@lists.infradead.org
> Ämne: Jffss2_write_super erasing all the blocks
> 
> Hi,
> 
> I have a question about JFFS2 fs. In my setup, I am running JFFS2 on AMD
> Spansion flash (NOR flash). What I noticed was that whenever I erase a
> file from my flash, I see that jffs2_write_super function gets called
> which in turn calls jffs2_erase_pending_blocks(0) which goes ahead and
> deletes all the pending blocks. While this erase is going on super block
> is locked and we can not perform any other operation with JFFS2.
> 
> Can someone please explain the logic behind this or if there is
> something I am missing here?
> 
> Thanks,
> Nikhil
> 
> -----Original Message-----
> From: linux-mtd-bounces@lists.infradead.org
> [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of llandre
> Sent: Thursday, January 31, 2008 8:54 AM
> To: linux-mtd@lists.infradead.org
> Subject: JFFS2: file contents in case of data CRC error
> 
> I have a JFFS2 partition on 32MByte NAND device.
> When reading a specific file - see below for details - JFFS2 reports a
> Data CRC error but function nand_correct_data never returns -1, so I
> assume ECC algorithm is able to correct errors.
> However the file is not equal to the original one that has been written
> to NAND. In fact, in the middle of the file, I see a 4-kByte "hole"
> where all bytes are 0.
> Anybody can help me about understanding if this is the expected
> behaviour of JFFS2?
> Thanks in advance.
> 
> 
> bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0 mtdblock_open
> ok bash-2.05# cp -v /mnt/nand0/f.img /tmp/
> jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read
> 0x80b8997f, calculated 0x57f0ea8e `/mnt/nand0/f.img' -> `/tmp/f.img'
> 
> --
> llandre
> 
> DAVE Electronics System House - R&D Department
> web:   http://www.dave.eu
> email: r&d2@dave-tech.it
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

RE: JFFS2: file contents in case of data CRC error

[Korolev, Alexey]

Hi,

>> -> As result hole in the file has been formed. 
>> -> You read 0x0 in the middle of the file.
> I can't find the function that, in case of obsolete node, zeroes the 
> chunk of data passed to user space. Can you?
See jffs2_read_inode_range() function.

>> I think nobody considered such tests before. I am not sure if any
file
>> system may be reliable in such a case with temperature tests.
> Hmm ... I don't think the environmental condition may influence JFFS2 
> behaviour. The NAND used in the test is industrial grade so it is 
> guaranteed by manufacturer that it works in -40/+85 degC range. JFFS2 
> clearly does not anything about environment temperature. On the other 
> hand I'm wondering if temperature variation caused multiple bit
flipping 
> in one NAND block that MTD was not able to correct and this led to the

> problem we are talking about.

I guess the problem is related to multiple bit flipping caused by
temperature. 

Thanks,
Alexey

Re: JFFS2: file contents in case of data CRC error

[Jamie Lokier]

llandre wrote:
> >I'm thinking the only way to detect this with high reliability is to
> >store summaries of the existence of blocks in another part of storage,
> >with checksums and serial numbers - like some of the latest disk
> >filesystems begin to.
> Can you name these file systems?

ZFS, HammerFS and Btrfs, I think.

-- Jamie