UBIFS automatic recovery

UBIFS automatic recovery

[Johan Borkhuis]

Hello,

On our current system we had some issues with corrupted file systems after
power failures. The system is a TI AM3517 with kernel 2.6.37 and the
backport of UBIFS. The system had one partition, so when the file system
is corrupted the system cannot boot any more.

To improve I change the layout:
- UBI0: 2 partitions, ubi0_0 (rootfs) and ubi0_1 (application), both
read-only
- UBI1: 2 partitions, ubi1_0 (settings) and ubi1_1 (logging)

To test how good this is working I wrote a test application to have a lot
of disk-I/O on the logging partition. Now I cut the power every minute.
I see the erase-counters increase quite fast on UBI1, but UBI0 stays at
the same value (which is what I was looking for)

Once in a while this test results in an error on one of the UBI1
partitions (both settings and logging). The partition cannot be mounted
any more. I can fix this using ubiupdatevol, but I loose the data on this
partition. This was expected and works quite well, the system is always
accessible.

However, when I continue to pull the power on a system with a broken
partition (without running the test-application, this is only started when
all partitions are mounted correctly) after some time (this can be a few
reboots, or a couple of 100 reboots) the system fixes the partition
itself, and I can access the data again, without any indications in the
log.

When it fails it shows the following during a mount (also shown sometimes
for LEB 3 and 6):
UBIFS: recovery needed
UBIFS error (pid 640): ubifs_recover_log_leb: unrecoverable log corruption
in LEB 5

Another UBIFS message I see during a failed mount is:
UBIFS error (pid 637): ubifs_recover_master_node: dumping first master node

As long the mount fails the same message is repeated.

But the first time it succeeds again I get the following output when
mounting:
UBIFS: recovery needed
UBIFS: recovery completed

Now my question is if there is a process in the background that fixes the
problem. If this is the case, how can I trigger or help this process, so
that it will fix the problem on the first time the problem is detected.

Or is there another way to fix/repair a broken partition without loosing
the data that is stored on it?

Regards,
    Johan

Re: UBIFS automatic recovery

[Richard Weinberger]

On Fri, May 13, 2016 at 10:47 AM, Johan Borkhuis <maillist@borkhuis.com> wrote:
> However, when I continue to pull the power on a system with a broken
> partition (without running the test-application, this is only started when
> all partitions are mounted correctly) after some time (this can be a few
> reboots, or a couple of 100 reboots) the system fixes the partition
> itself, and I can access the data again, without any indications in the
> log.

This should not happen. Either the UBIFS is broken and not mountable
or it can recover.
Something else seems to interact here.

> When it fails it shows the following during a mount (also shown sometimes
> for LEB 3 and 6):
> UBIFS: recovery needed
> UBIFS error (pid 640): ubifs_recover_log_leb: unrecoverable log corruption
> in LEB 5
>
> Another UBIFS message I see during a failed mount is:
> UBIFS error (pid 637): ubifs_recover_master_node: dumping first master node
>
> As long the mount fails the same message is repeated.

UBIFS must not break due to power cuts.
So, something else is broken.
Do both MTD and UBI tests pass?

> But the first time it succeeds again I get the following output when
> mounting:
> UBIFS: recovery needed
> UBIFS: recovery completed

This is perfectly fine and only indicates that UBIFS recovered
from an unclean shutdown.
i.e. replayed the journal.

> Now my question is if there is a process in the background that fixes the
> problem. If this is the case, how can I trigger or help this process, so
> that it will fix the problem on the first time the problem is detected.
>
> Or is there another way to fix/repair a broken partition without loosing
> the data that is stored on it?

No. What you need is figuring out *why* UBIFS breaks while doing
power cuts. Both UBI and UBIFS are power cut aware by design.
In most cases UBIFS suffers from MTD problems. May it a faulty driver
or bad hardware...

-- 
Thanks,
//richard

Re: UBIFS automatic recovery

[Johan Borkhuis]

Richard,

Thanks for your quick response!

> On Fri, May 13, 2016 at 10:47 AM, Johan Borkhuis <maillist@borkhuis.com>
> wrote:
>> However, when I continue to pull the power on a system with a broken
>> partition (without running the test-application, this is only started
>> when
>> all partitions are mounted correctly) after some time (this can be a few
>> reboots, or a couple of 100 reboots) the system fixes the partition
>> itself, and I can access the data again, without any indications in the
>> log.
>
> This should not happen. Either the UBIFS is broken and not mountable
> or it can recover.
> Something else seems to interact here.

This was also what I was expecting, and why I was surprised when units
started to recover.

>> When it fails it shows the following during a mount (also shown
>> sometimes for LEB 3 and 6):
>> UBIFS: recovery needed
>> UBIFS error (pid 640): ubifs_recover_log_leb: unrecoverable log
>> corruption
>> in LEB 5
>>
>> Another UBIFS message I see during a failed mount is:
>> UBIFS error (pid 637): ubifs_recover_master_node: dumping first master
>> node
>>
>> As long the mount fails the same message is repeated.
>
> UBIFS must not break due to power cuts.
> So, something else is broken.
> Do both MTD and UBI tests pass?

You mean the kernel startup? Or some specific MTD/UBI test tools?
On startup there is no difference in the console-output between a good and
bad startup, so MTD and UBI are initialised correctly. The first sign is
the fact that a mount fails.

>> Or is there another way to fix/repair a broken partition without loosing
>> the data that is stored on it?
>
> No. What you need is figuring out *why* UBIFS breaks while doing
> power cuts. Both UBI and UBIFS are power cut aware by design.
> In most cases UBIFS suffers from MTD problems. May it a faulty driver
> or bad hardware...

We are using a Micon 2Mb flash chip and the kernel sources from TI. The
output during boot:
omap2-nand driver initializing
NAND device: Manufacturer ID: 0x2c, Chip ID: 0xca (Micron )
Creating 6 MTD partitions on "omap2-nand.0":
....

Regards,
    Johan

Re: UBIFS automatic recovery

[Richard Weinberger]

Johan,

Am 13.05.2016 um 12:27 schrieb Johan Borkhuis:
>> This should not happen. Either the UBIFS is broken and not mountable
>> or it can recover.
>> Something else seems to interact here.
> 
> This was also what I was expecting, and why I was surprised when units
> started to recover.

Maybe a temporary issue with your NAND.

>>> When it fails it shows the following during a mount (also shown
>>> sometimes for LEB 3 and 6):
>>> UBIFS: recovery needed
>>> UBIFS error (pid 640): ubifs_recover_log_leb: unrecoverable log
>>> corruption
>>> in LEB 5
>>>
>>> Another UBIFS message I see during a failed mount is:
>>> UBIFS error (pid 637): ubifs_recover_master_node: dumping first master
>>> node
>>>
>>> As long the mount fails the same message is repeated.
>>
>> UBIFS must not break due to power cuts.
>> So, something else is broken.
>> Do both MTD and UBI tests pass?
> 
> You mean the kernel startup? Or some specific MTD/UBI test tools?
> On startup there is no difference in the console-output between a good and
> bad startup, so MTD and UBI are initialised correctly. The first sign is
> the fact that a mount fails.

In the kernel tree we have a set of tests, MTD tests.
See CONFIG_MTD_TESTS.
UBI tests are part of the mtd-utils source package.

>>> Or is there another way to fix/repair a broken partition without loosing
>>> the data that is stored on it?
>>
>> No. What you need is figuring out *why* UBIFS breaks while doing
>> power cuts. Both UBI and UBIFS are power cut aware by design.
>> In most cases UBIFS suffers from MTD problems. May it a faulty driver
>> or bad hardware...
> 
> We are using a Micon 2Mb flash chip and the kernel sources from TI. The
> output during boot:
> omap2-nand driver initializing
> NAND device: Manufacturer ID: 0x2c, Chip ID: 0xca (Micron )
> Creating 6 MTD partitions on "omap2-nand.0":

*maybe* a driver issue.
First of all you need to find out what exactly is broken.
i.e. analyze the contents of the NAND upon failure.

Thanks,
//richard

Re: UBIFS automatic recovery

[Iwo Mergler]

On 05/13/2016 07:11 PM, Richard Weinberger wrote:
> No. What you need is figuring out*why*  UBIFS breaks while doing
> power cuts. Both UBI and UBIFS are power cut aware by design.
> In most cases UBIFS suffers from MTD problems. May it a faulty driver
> or bad hardware...

Johan,


check if your NAND driver can handle erased pages
with bit errors (0-bits). The drivers from the
old TI Arago tree for the am3x processors didn't,
for instance.

The NAND is expected to contain all '1's after erasure.

Unfortunately, unlike the 1-bit (Hamming) ECC, most
multi-bit (BCH) ECC schemes use a code that flags an
uncorrectable error if both the page and the ECC area
are '1'.

It's annoying - at chip implementation time, a few
extra transistors would have avoided that.

In software, the NAND driver now gets an 'uncorrectable'
error and must check if the page is, in fact, erased. If
it is, it pretends success.

A popular way of checking for erasure is to compare
the ECC syndrome to the known value for an erased page.

Which works fine, as long as you don't have any '0' bits
in it. If you do, the workaround fails and UBI experiences
an "uncorrectable" page.

Most of the time UBIFS doesn't verify erased pages, so
you get away with it. But in times of stress - recovery
or in-the-gaps allocation, for instance - erased pages
are being verified and UBIFS tends to go read-only or
fail the mount on error.

So, make sure that your NAND driver checks the entire
data payload of an erased page for '1' bits, but allows
a small number of '0' bits. Then applies a memset(1)
to the buffer before handing it to MTD.


Best regards,

Iwo