ubifs error : read_znode: bad indexing node

ubifs error : read_znode: bad indexing node

[Matthieu CASTET]

Hi,

we have products where rw ubifs partition are corrupted.
They all failed with the "read_znode: bad indexing node at LEB 558:22056, error
6" or "read_znode: bad indexing node at LEB 446:62888, error 2".

The last ubifs stable commit we have is
http://git.infradead.org/users/dedekind/ubifs-v2.6.32.git/commit/7eb3b6c0999bd2cbc37adb2bd0fb8127f98240ea
.


Do you know if the bug is known ?
Google doesn't provide any info on this and looking at ubifs new commits don't
provide explicit bug fix.

Are there any document that explain how znode works ?


Matthieu

Re: ubifs error : read_znode: bad indexing node

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

Hi,

On Mon, 2012-06-11 at 15:59 +0200, Matthieu CASTET wrote:
> we have products where rw ubifs partition are corrupted.
> They all failed with the "read_znode: bad indexing node at LEB 558:22056, error
> 6" or "read_znode: bad indexing node at LEB 446:62888, error 2".

So we read the indexing node, and its CRC is correct, but then we
validate it by checking that various fields are withing possible limits,
and we see that there is some garbage.

This smells more like memory corruption somewhere - we write garbage or
corrupted buffers to the media.

I guess it would be a good improvement to UBIFS if we validated when we
write, not only read, then we could catch issues earlier.

> The last ubifs stable commit we have is
> http://git.infradead.org/users/dedekind/ubifs-v2.6.32.git/commit/7eb3b6c0999bd2cbc37adb2bd0fb8127f98240ea

Does not look like something which could cause those errors. Is that
problem reproducible? Only on these devices or on other too?


> Do you know if the bug is known ?
> Google doesn't provide any info on this and looking at ubifs new commits don't
> provide explicit bug fix.

No, this is the first time I see such a problem.

> Are there any document that explain how znode works ?

Well, Adrian's PDF file may give some insight. znode is just an in-RAM
representation of an indexing node on the flash.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubifs error : read_znode: bad indexing node

[Matthieu CASTET]

Hi Artem,

Artem Bityutskiy a écrit :
> Hi,
> 
> On Mon, 2012-06-11 at 15:59 +0200, Matthieu CASTET wrote:
>> we have products where rw ubifs partition are corrupted.
>> They all failed with the "read_znode: bad indexing node at LEB 558:22056, error
>> 6" or "read_znode: bad indexing node at LEB 446:62888, error 2".
> 
> So we read the indexing node, and its CRC is correct, but then we
> validate it by checking that various fields are withing possible limits,
> and we see that there is some garbage.
> 
> This smells more like memory corruption somewhere - we write garbage or
> corrupted buffers to the media.
Yes after some investigation corrupted bad look like kernel addresses :

branch to 0x144:0x10468 len 160 type 0x0 0xcffc5418 0
branch to 0x131:0x0 len 4144 type 1 0x4210 0x20000000
branch to 0x131:0x1030 len 108 type 1 0x4210 0x20000001
branch to 0x144:0x14800 len 160 type 0 0x4213 0x0
branch to 0x131:0xf800 len 4144 type 1 0x4213 0x20000000

branch to 0xd0:0x5278 len 4144 type 1 0x6fd1 0x20000003
branch to 0xd0:0x62a8 len 4144 type 1 0x6fd1 0x20000004
branch to 0xd0:0x72d8 len 4144 type 1 0x6fd1 0x20000005
branch to 0xd0:0xc0309638 len -1070091304 type 1 0x6fd1 0x20000006
branch to 0xd0:0x9338 len 4144 type 1 0x6fd1 0x20000007
branch to 0xd0:0xa368 len 4144 type 1 0x6fd1 0x20000008
branch to 0xd0:0xb398 len 3737 type 1 0x6fd1 0x20000009
branch to 0xc1:0xd9c0 len 160 type 0 0x6fd7 0x0

We may have a driver that corrupt the heap/memory.
> 
> I guess it would be a good improvement to UBIFS if we validated when we
> write, not only read, then we could catch issues earlier.
Yes, and there is already assert in debug mode that does such check.
But there is no guarantee that the corruption won't be done after the check ;)

> 
>> The last ubifs stable commit we have is
>> http://git.infradead.org/users/dedekind/ubifs-v2.6.32.git/commit/7eb3b6c0999bd2cbc37adb2bd0fb8127f98240ea
> 
> Does not look like something which could cause those errors. Is that
> problem reproducible? Only on these devices or on other too?
For a product we have many case of such corruption : bad index, bad  master node
, bad lpt, ...

But I don't believe it is a ubifs problem, but a bug on our side.

Thanks for your help,


Matthieu

Re: ubifs error : read_znode: bad indexing node

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Wed, 2012-06-20 at 11:51 +0200, Matthieu CASTET wrote:
> We may have a driver that corrupt the heap/memory.

If it is ARM and you do DMA in your driver - this may be the cause of
issues because UBIFS uses vmalloc()'ed memory and there are issues with
DMAing it. Also, various things related to cache coherency are often the
source of issues.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]