Corrupt empty space

Corrupt empty space

[HOUR Fréderic]

Hello,

I'm working on UBIFS ( linux 3.3, omap138 ). I need some help with a problem on a custom board (omapl138).
I have a software that runs in background and writes data in a UBIFS volume. I can not control when the power cut happens.
Sometimes, when booting my board, I got this "common" error:

UBI: background thread "ubi_bgt0d" started, PID 391
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 0, volume 0, name "log"
UBIFS: file system size:   3174400 bytes (3100 KiB, 3 MiB, 25 LEBs)
UBIFS: journal size:       1015809 bytes (992 KiB, 0 MiB, 6 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: none
UBIFS: reserved for root:  149934 bytes (146 KiB)
UBIFS: recovery needed
UBIFS error (pid 395): ubifs_scan: corrupt empty space at LEB 1534:125170
UBIFS error (pid 395): ubifs_scanned_corruption: corruption at LEB 1534:125170
00000000: ffffff7f ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
00000020: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
[..]
UBIFS error (pid 395): ubifs_scan: LEB 1534 scanning failed

My NAND volume could not be mounted. My problem is that I want to have an access to the files stored in this UBIFS volume.

So I had some questions : 
- What are the risk of commenting out the 'goto corrupted' in the fonction ubifs_scan? (in order to mount my partition and have an access to my files)
- Is this the "unstable bits issue"? (http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits) 
- What could be a proper solution to get my files?
- Is there a solution using U-Boot?

Thanks for your help.

Regards,

Frederic

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#

RE: Corrupt empty space

[Kent Li]

Frederic,

I'm not sure if this is the unstable bit issue, but I just resolved such issue by enhancing my Nand driver in both Uboot code and Linux code.

The main idea is:
Read a page
If ( ECC fail ) ;then
	Check the how many 0 bit there is.
	If ( 0 bit number > tolerance value) //tolerance value is depends on the ECC correctable bits.
		Print "Fail to recover by ECC"
	Else
		Force change the 0 bits to 1
	fi
fi

This idea assume that 
1. there is not ECC fail when reading a normal page.
2. sub-page is disabled.
3. only ECC fail happen when reading an empty page.

BR
Kent Li


-----Original Message-----
From: linux-mtd [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of HOUR Fréderic
Sent: 2014年3月11日 1:35
To: linux-mtd@lists.infradead.org
Subject: Corrupt empty space

Hello,

I'm working on UBIFS ( linux 3.3, omap138 ). I need some help with a problem on a custom board (omapl138).
I have a software that runs in background and writes data in a UBIFS volume. I can not control when the power cut happens.
Sometimes, when booting my board, I got this "common" error:

UBI: background thread "ubi_bgt0d" started, PID 391
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 0, volume 0, name "log"
UBIFS: file system size:   3174400 bytes (3100 KiB, 3 MiB, 25 LEBs)
UBIFS: journal size:       1015809 bytes (992 KiB, 0 MiB, 6 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: none
UBIFS: reserved for root:  149934 bytes (146 KiB)
UBIFS: recovery needed
UBIFS error (pid 395): ubifs_scan: corrupt empty space at LEB 1534:125170 UBIFS error (pid 395): ubifs_scanned_corruption: corruption at LEB 1534:125170
00000000: ffffff7f ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
00000020: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
[..]
UBIFS error (pid 395): ubifs_scan: LEB 1534 scanning failed

My NAND volume could not be mounted. My problem is that I want to have an access to the files stored in this UBIFS volume.

So I had some questions : 
- What are the risk of commenting out the 'goto corrupted' in the fonction ubifs_scan? (in order to mount my partition and have an access to my files)
- Is this the "unstable bits issue"? (http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits)
- What could be a proper solution to get my files?
- Is there a solution using U-Boot?

Thanks for your help.

Regards,

Frederic

#
" Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles, être soumis aux règlementations relatives au contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Toute exportation ou réexportation non autorisée est interdite Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
******
" This e-mail and any attached documents may contain confidential or proprietary information and may be subject to export control laws and regulations. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. Unauthorized export or re-export is prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
#


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

RE: Corrupt empty space

[Gupta, Pekon]

Hi,

>
>Frederic,
>
>I'm not sure if this is the unstable bit issue, but I just resolved such issue by enhancing my Nand driver in both Uboot code and Linux
>code.
>
>The main idea is:
>Read a page
>If ( ECC fail ) ;then
>	Check the how many 0 bit there is.
>	If ( 0 bit number > tolerance value) //tolerance value is depends on the ECC correctable bits.
>		Print "Fail to recover by ECC"
>	Else
>		Force change the 0 bits to 1
>	fi
>fi
>
>This idea assume that
>1. there is not ECC fail when reading a normal page.
>2. sub-page is disabled.
>3. only ECC fail happen when reading an empty page.
>

I'm facing similar problem with some newer technology NAND devices,
And the above approach is similar to what was proposed in OMAP NAND
driver, But its still under discussion [1]
	if (page_is_erased)
		/* return clean read-buffer */
		memset(buf, 0xff, ecc->size);

However, from 3.9 kernel version, MTD and UBI layer already have checks
for bit-flips threshold [2], so this extra check for all(0xff) data in UBIFS is redundant.
I have just sent an RFC patch to remove this all(0xff) check from UBIFS [3],
I'll let Artem (UBIFS Maintainer) comment on this, as such empty-page
corruptions are common in MLC and newer technology NANDs.


[1] http://lists.infradead.org/pipermail/linux-mtd/2014-March/052472.html
[2] http://lists.infradead.org/pipermail/linux-mtd/2012-April/040945.html
[3] http://lists.infradead.org/pipermail/linux-mtd/2014-March/052499.html


with regards, pekon

Re: Corrupt empty space

[Artem Bityutskiy]

On Tue, 2014-03-11 at 05:31 +0000, Kent Li wrote:
> Frederic,
> 
> I'm not sure if this is the unstable bit issue, but I just resolved such issue by enhancing my Nand driver in both Uboot code and Linux code.
> 
> The main idea is:
> Read a page
> If ( ECC fail ) ;then
> 	Check the how many 0 bit there is.
> 	If ( 0 bit number > tolerance value) //tolerance value is depends on the ECC correctable bits.
> 		Print "Fail to recover by ECC"
> 	Else
> 		Force change the 0 bits to 1
> 	fi
> fi

Off the top of my head, this sounds reasonable. Too bad people people do
not just go and fix this in upstream...

-- 
Best Regards,
Artem Bityutskiy

Re: Corrupt empty space

[Brian Norris]

On Tue, Mar 11, 2014 at 12:46 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Tue, 2014-03-11 at 05:31 +0000, Kent Li wrote:
>> Frederic,
>>
>> I'm not sure if this is the unstable bit issue, but I just resolved such issue by enhancing my Nand driver in both Uboot code and Linux code.
>>
>> The main idea is:
>> Read a page
>> If ( ECC fail ) ;then
>>       Check the how many 0 bit there is.
>>       If ( 0 bit number > tolerance value) //tolerance value is depends on the ECC correctable bits.
>>               Print "Fail to recover by ECC"
>>       Else
>>               Force change the 0 bits to 1
>>       fi
>> fi
>
> Off the top of my head, this sounds reasonable. Too bad people people do
> not just go and fix this in upstream...

I have something like this locally, and Huang [1] and Pekon [2] are
both trying to implement this in their upstream drivers (incorrectly,
IMO). As we speak, I'm preparing a patch to send out that can
hopefully kill both [1] and [2]. Unfortunately, I'm discovering some
strange behavior (maybe a HW bug?) on my platform while testing...

I'll try to remember to CC all of you when I send my patch.

Regards,
Brian

[1] http://lists.infradead.org/pipermail/linux-mtd/2014-January/051513.html
[2] http://lists.infradead.org/pipermail/linux-mtd/2014-January/051585.html

RE: Corrupt empty space

[Gupta, Pekon]

Hi,

>From: Artem Bityutskiy
>>On Tue, 2014-03-11 at 05:31 +0000, Kent Li wrote:
>> Frederic,
>>
>> I'm not sure if this is the unstable bit issue, but I just resolved such issue by enhancing my Nand driver in both Uboot code and Linux
>code.
>>
>> The main idea is:
>> Read a page
>> If ( ECC fail ) ;then
>> 	Check the how many 0 bit there is.
>> 	If ( 0 bit number > tolerance value) //tolerance value is depends on the ECC correctable bits.
>> 		Print "Fail to recover by ECC"
>> 	Else
>> 		Force change the 0 bits to 1
>> 	fi
>> fi
>
>Off the top of my head, this sounds reasonable. Too bad people people do
>not just go and fix this in upstream...
>
It's still getting discussed in [a] and [b]. But there are multiple issues to it.

 (1) Many hardware controllers cannot parse ECC errors in blank-pages.
   So, checking number of 0-bits in a page is done in software by comparing
   each byte with 0xff.

(2) Counting number of 0-bits in a page in software, brings down your read-performance.

(3) Some controller required specific sequence for reading pages, hence getting
  a generic driver solution for this problem, is becoming tough. 


*Thread on NAND Driver changes*
[a] http://lists.infradead.org/pipermail/linux-mtd/2014-March/052507.html
[b] http://lists.infradead.org/pipermail/linux-mtd/2014-March/052625.html


*Thread on UBIFS changes*
http://lists.infradead.org/pipermail/linux-mtd/2014-March/052499.html


with regards, pekon

Re: Corrupt empty space

[Artem Bityutskiy]

On Fri, 2014-03-21 at 07:18 +0000, Gupta, Pekon wrote:
> It's still getting discussed in [a] and [b]. But there are multiple issues to it.
> 
>  (1) Many hardware controllers cannot parse ECC errors in blank-pages.
>    So, checking number of 0-bits in a page is done in software by comparing
>    each byte with 0xff.

OK.

> (2) Counting number of 0-bits in a page in software, brings down your read-performance.

UBIFS _only_ needs this when doing recovery, which is _only_ done after
unclean reboots, and this kind of reads would be done _only_ for a
couple of max I/O units and _only_ if there is a corrupted node in the
journal.

IOW, this would be done rarely, and for just few pages on mount.

-- 
Best Regards,
Artem Bityutskiy

Re: Corrupt empty space

[Ricard Wanderlof]

On Fri, 21 Mar 2014, Artem Bityutskiy wrote:

> On Fri, 2014-03-21 at 07:18 +0000, Gupta, Pekon wrote:
>> It's still getting discussed in [a] and [b]. But there are multiple issues to it.
>>
>>  (1) Many hardware controllers cannot parse ECC errors in blank-pages.
>>    So, checking number of 0-bits in a page is done in software by comparing
>>    each byte with 0xff.
>
> OK.
>
>> (2) Counting number of 0-bits in a page in software, brings down your read-performance.
>
> UBIFS _only_ needs this when doing recovery, which is _only_ done after
> unclean reboots, and this kind of reads would be done _only_ for a
> couple of max I/O units and _only_ if there is a corrupted node in the
> journal.
>
> IOW, this would be done rarely, and for just few pages on mount.

What about when attaching a UBI volume to a completely a erased partition? 
I would assume that with the EC and VID headers thus missing, that UBI 
would erase each block prior to writing the headers, and then knowing that 
they were erased not need to check for 0xff in the blocks?

Admittedly this would only happen once in a products lifetime so even if 
UBI did need to scan each block, it won't have much of an impact on 
performance, except possibly that the additional time would be spent in 
production where time is a premium.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: Corrupt empty space

[Gupta, Pekon]

>From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
>
>On Fri, 2014-03-21 at 07:18 +0000, Gupta, Pekon wrote:
>> It's still getting discussed in [a] and [b]. But there are multiple issues to it.
>>
>>  (1) Many hardware controllers cannot parse ECC errors in blank-pages.
>>    So, checking number of 0-bits in a page is done in software by comparing
>>    each byte with 0xff.
>
>OK.
>
>> (2) Counting number of 0-bits in a page in software, brings down your read-performance.
>
>UBIFS _only_ needs this when doing recovery, which is _only_ done after
>unclean reboots, and this kind of reads would be done _only_ for a
>couple of max I/O units and _only_ if there is a corrupted node in the
>journal.
>
>IOW, this would be done rarely, and for just few pages on mount.
>
Sorry. I din't clarify. Read performance drops in NAND driver itself, nothing
related to UBIFS.  
Because of (1) you need to filter-out erased-pages with bit-flips before
driver sends data to hardware engine for ECC error correction. This filtering
of erased-page in software (by comparing each byte for 0-bits) causes
this read performance drop.
However, there is no work-around to it, as it’s a kind of missing feature
in hardware. But we are trying to optimize this implementation.
http://lists.infradead.org/pipermail/linux-mtd/2014-March/052653.html


Also, as there are multiple threads on same topic, If would be helpful
if you can review and continue discussion over there.

*Thread on NAND Driver changes*
http://lists.infradead.org/pipermail/linux-mtd/2014-March/thread.html
	grep "mtd: nand: add erased-page bitflip correction"

*Thread on UBIFS changes*
http://lists.infradead.org/pipermail/linux-mtd/2014-March/thread.html
	grep "UBIFS: remove check for all(0xff) for empty pages"


with regards, pekon

Re: Corrupt empty space

[Artem Bityutskiy]

On Fri, 2014-03-21 at 08:43 +0100, Ricard Wanderlof wrote:
> On Fri, 21 Mar 2014, Artem Bityutskiy wrote:
> 
> > On Fri, 2014-03-21 at 07:18 +0000, Gupta, Pekon wrote:
> >> It's still getting discussed in [a] and [b]. But there are multiple issues to it.
> >>
> >>  (1) Many hardware controllers cannot parse ECC errors in blank-pages.
> >>    So, checking number of 0-bits in a page is done in software by comparing
> >>    each byte with 0xff.
> >
> > OK.
> >
> >> (2) Counting number of 0-bits in a page in software, brings down your read-performance.
> >
> > UBIFS _only_ needs this when doing recovery, which is _only_ done after
> > unclean reboots, and this kind of reads would be done _only_ for a
> > couple of max I/O units and _only_ if there is a corrupted node in the
> > journal.
> >
> > IOW, this would be done rarely, and for just few pages on mount.
> 
> What about when attaching a UBI volume to a completely a erased partition? 
> I would assume that with the EC and VID headers thus missing, that UBI 
> would erase each block prior to writing the headers, and then knowing that 
> they were erased not need to check for 0xff in the blocks?

Hmm, yeah, I missed this case, thanks.

> Admittedly this would only happen once in a products lifetime so even if 
> UBI did need to scan each block, it won't have much of an impact on 
> performance, except possibly that the additional time would be spent in 
> production where time is a premium.

Yeah, I guess you are right.

-- 
Best Regards,
Artem Bityutskiy

Re: Corrupt empty space

[Artem Bityutskiy]

On Fri, 2014-03-21 at 07:50 +0000, Gupta, Pekon wrote:
> Also, as there are multiple threads on same topic, If would be helpful
> if you can review and continue discussion over there.

Pekon, I apologise, I do not have time to review these threads now. If
this is difficult to implement generically on the MTD level, I trust
you.

I think there are only few places where UBI or UBIFS really need to know
an answer to the following question: is this min. I/O unit empty (never
written to) or not?

1. Recovery in UBIFS
2. Auto-formatting decision in UBI/UBIFS
3. Some debugging stuff in UBI where we try to make sure we write only
to empty pages.

If MTD layer provides helpers which UBIFS could call to get the answer,
that could probably work.

I mean, probably it is not necessary to make every read do this empty
page processing. It would be cool, but if the price is read throughput
drop, probably not worth it. All MTD reads can be the old style reads
which give no ECC protection when the page is empty. This could be
documented somewhere, BTW, in the code, and web site, too.

Then when UBI/UBIFS needs to know whether certain min. I/O unit is empty
or not, then UBI/UBIFS calls a special MTD helper function to get the
answer.

This would probably need some work in UBI/UBIFS, but I think this should
not be too difficult.

-- 
Best Regards,
Artem Bityutskiy