Trouble with UBIFS

Trouble with UBIFS

[Smitha Rathnam]

 Hi,

 We have an embedded linux (2.6) system. Since the past few months, we
have been seeing a corruption of the log filesystem.
 The filesystem is using mtd5 device.
 The error looks like this;
 user.notice kernel: klogd started: BusyBox v1.20.0 (2012-09-18 17:42:52 MST)

 Jan  1 00:00:05 user.debug kernel: 00001fa0: 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000
................................
 Jan  1 00:00:05 user.debug kernel: 00001fc0: 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000
................................
 Jan  1 00:00:05 user.debug kernel: 00001fe0: 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000
................................
 *Jan  1 00:00:05 user.err kernel: UBIFS error (pid 1110):
ubifs_recover_leb: LEB 24 scanning failed*

 Is there a way to retrieve data from the partition? How do we fix this error?

 Please help.

 Thanks and regards,
 Smitha.

Re: Trouble with UBIFS

[Richard Weinberger]

Hi!

On Wed, Jun 10, 2015 at 12:08 PM, Smitha Rathnam
<rathnam.smitha@gmail.com> wrote:
>  Hi,
>
>  We have an embedded linux (2.6) system. Since the past few months, we
> have been seeing a corruption of the log filesystem.
>  The filesystem is using mtd5 device.
>  The error looks like this;
>  user.notice kernel: klogd started: BusyBox v1.20.0 (2012-09-18 17:42:52 MST)
>
>  Jan  1 00:00:05 user.debug kernel: 00001fa0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  Jan  1 00:00:05 user.debug kernel: 00001fc0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  Jan  1 00:00:05 user.debug kernel: 00001fe0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  *Jan  1 00:00:05 user.err kernel: UBIFS error (pid 1110):
> ubifs_recover_leb: LEB 24 scanning failed*
>
>  Is there a way to retrieve data from the partition? How do we fix this error?

To fix the problem one has to analyze it first.
The kernel output you've posted is incomplete and most likely unrelated.
Do you have a complete kernel log?

Retrieving data from the filesystem is not easy, it depends how much UBIFS
is corrupted.
What I do in such cases is dumping all nodes and running them thought a
semi-automated script which can deal with UBIFS.

-- 
Thanks,
//richard

Re: Trouble with UBIFS

[Brokhman, Tanya]

Hi Richard

On 6/10/2015 2:25 PM, Richard Weinberger wrote:
> Hi!
>
> On Wed, Jun 10, 2015 at 12:08 PM, Smitha Rathnam
> <rathnam.smitha@gmail.com> wrote:
>>   Hi,
>>
>>   We have an embedded linux (2.6) system. Since the past few months, we
>> have been seeing a corruption of the log filesystem.
>>   The filesystem is using mtd5 device.
>>   The error looks like this;
>>   user.notice kernel: klogd started: BusyBox v1.20.0 (2012-09-18 17:42:52 MST)
>>
>>   Jan  1 00:00:05 user.debug kernel: 00001fa0: 00000000 00000000
>> 00000000 00000000 00000000 00000000 00000000 00000000
>> ................................
>>   Jan  1 00:00:05 user.debug kernel: 00001fc0: 00000000 00000000
>> 00000000 00000000 00000000 00000000 00000000 00000000
>> ................................
>>   Jan  1 00:00:05 user.debug kernel: 00001fe0: 00000000 00000000
>> 00000000 00000000 00000000 00000000 00000000 00000000
>> ................................
>>   *Jan  1 00:00:05 user.err kernel: UBIFS error (pid 1110):
>> ubifs_recover_leb: LEB 24 scanning failed*
>>
>>   Is there a way to retrieve data from the partition? How do we fix this error?
> To fix the problem one has to analyze it first.
> The kernel output you've posted is incomplete and most likely unrelated.
> Do you have a complete kernel log?
>
> Retrieving data from the filesystem is not easy, it depends how much UBIFS
> is corrupted.
> What I do in such cases is dumping all nodes and running them thought a
> semi-automated script which can deal with UBIFS.
Could you please share the script? I'm facing same issues at the moment 
(debugging ubifs corruption). I have a binary dump of the partition. 
Looking for the most user-friendly tool to look at it.

Thanks
-Tanya

Re: Trouble with UBIFS

[Richard Weinberger]

Tanya,

Am 10.06.2015 um 13:55 schrieb Brokhman, Tanya:
>> Retrieving data from the filesystem is not easy, it depends how much UBIFS
>> is corrupted.
>> What I do in such cases is dumping all nodes and running them thought a
>> semi-automated script which can deal with UBIFS.
> Could you please share the script? I'm facing same issues at the moment (debugging ubifs corruption). I have a binary dump of the partition. Looking for the most user-friendly tool
> to look at it.

I'll share it as soon it is a) kind of user friendly and b) all legal stuff has been sorted out.
My plan is turning it into something like a fsck.ubfs with a "fuzzy dump" mode.

Thanks,
//richard

Re: Trouble with UBIFS

[Brokhman, Tanya]

On 6/10/2015 3:43 PM, Richard Weinberger wrote:
> Tanya,
>
> Am 10.06.2015 um 13:55 schrieb Brokhman, Tanya:
>>> Retrieving data from the filesystem is not easy, it depends how much UBIFS
>>> is corrupted.
>>> What I do in such cases is dumping all nodes and running them thought a
>>> semi-automated script which can deal with UBIFS.
>> Could you please share the script? I'm facing same issues at the moment (debugging ubifs corruption). I have a binary dump of the partition. Looking for the most user-friendly tool
>> to look at it.
> I'll share it as soon it is a) kind of user friendly and b) all legal stuff has been sorted out.
> My plan is turning it into something like a fsck.ubfs with a "fuzzy dump" mode.
great. thank you!

-Tanya

>
> Thanks,
> //richard

Re: Trouble with UBIFS

[Smitha Rathnam]

Hi,

With respect to the UBIFS issue.. How do I come to a conclusion that
it is a hardware fault and not a software one?
We are using Micron nand devices in our system.
The UBI rootfs is put on /dev/mtd4 and the ubi logfs is put on
/dev/mtd5. The mtd5 partition shows corruption. When we format the
device and reinstall, the corruption goes away. Of course, we have
lost data.
So should this be enough to conclude that this is an issue with
software and not the actual NAND device?
I ran the nandtest from the mtd_utils set. Before the format the test
showed a lot of ECC errors. After the format and reinstall, the test
showed no ECC failures although there were 4 bad blocks.
How do we understand this? Is this enough to conclude that the issue
is a software one related to the file system?

Regards,
Smitha.

On Wed, Jun 10, 2015 at 8:08 PM, Smitha Rathnam
<rathnam.smitha@gmail.com> wrote:
>  Hi,
>
>  We have an embedded linux (2.6) system. Since the past few months, we
> have been seeing a corruption of the log filesystem.
>  The filesystem is using mtd5 device.
>  The error looks like this;
>  user.notice kernel: klogd started: BusyBox v1.20.0 (2012-09-18 17:42:52 MST)
>
>  Jan  1 00:00:05 user.debug kernel: 00001fa0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  Jan  1 00:00:05 user.debug kernel: 00001fc0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  Jan  1 00:00:05 user.debug kernel: 00001fe0: 00000000 00000000
> 00000000 00000000 00000000 00000000 00000000 00000000
> ................................
>  *Jan  1 00:00:05 user.err kernel: UBIFS error (pid 1110):
> ubifs_recover_leb: LEB 24 scanning failed*
>
>  Is there a way to retrieve data from the partition? How do we fix this error?
>
>  Please help.
>
>  Thanks and regards,
>  Smitha.

Re: Trouble with UBIFS

[Richard Weinberger]

On Thu, Jun 11, 2015 at 6:58 AM, Smitha Rathnam
<rathnam.smitha@gmail.com> wrote:
> Hi,
>
> With respect to the UBIFS issue.. How do I come to a conclusion that
> it is a hardware fault and not a software one?

By analyzing the issue you're facing.

> We are using Micron nand devices in our system.
> The UBI rootfs is put on /dev/mtd4 and the ubi logfs is put on
> /dev/mtd5. The mtd5 partition shows corruption. When we format the
> device and reinstall, the corruption goes away. Of course, we have
> lost data.
> So should this be enough to conclude that this is an issue with
> software and not the actual NAND device?

No way. At least share the full kernel log, including the issue with us.

> I ran the nandtest from the mtd_utils set. Before the format the test
> showed a lot of ECC errors. After the format and reinstall, the test
> showed no ECC failures although there were 4 bad blocks.
> How do we understand this? Is this enough to conclude that the issue
> is a software one related to the file system?

Please verify your MTD driver by running mtd-tests (from kernel source),
and UBI tests (from mtd-utils source).

-- 
Thanks,
//richard