Filesystems over UBI can't handle badblocks

Filesystems over UBI can't handle badblocks

[Guilherme de Oliveira Costa]

Hello,

I'm using ubiblk to run cramfs over UBI (the cramfs partition contains my rootfs), but I'm still having trouble handling bad blocks.

I'm using a 128MB NAND, with a 128 kB block size, 2048 B page size.

This is my ubinize config file:

[fs-volume]
mode=ubi
image=cramfs.img
vol_id=1
vol_size=20MiB
vol_type=dynamic
vol_name=fs
vol_flags=autoresize

And I generate the images as such:
mkfs.cramfs fs_dir cramfs.img
ubinize -o fs_on_ubi.img -m 2048 -p 128KiB -O 2048 ubinize.cfg

If I flash these into my system via u-boot, it works without a problem. The issue starts when marking a block as bad (via nand markbad): if I mark the first two blocks as bad, the system boots normally, If I mark any block from the third onwards, the system is able to mount rootfs, by fails soon after with a "Error while decompressing" message. Here is an excerpt from my boot log:

Skip to the important stuff
[    0.710820] UBI: attaching mtd3 to ubi0
[    0.714869] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[    0.721614] UBI: logical eraseblock size:    126976 bytes
[    0.727339] UBI: smallest flash I/O unit:    2048
[    0.732292] UBI: VID header offset:          2048 (aligned 2048)
[    0.738641] UBI: data offset:                4096
[    0.840392] UBI: max. sequence number:       2
[    0.851883] UBI: attached mtd3 to ubi0
[    0.855933] UBI: MTD device name:            "fs"
[    0.860886] UBI: MTD device size:            25 MiB
[    0.866048] UBI: number of good PEBs:        199
[    0.870907] UBI: number of bad PEBs:         1
[    0.875604] UBI: number of corrupted PEBs:   0
[    0.880280] UBI: max. allowed volumes:       128
[    0.885139] UBI: wear-leveling threshold:    4096
[    0.890101] UBI: number of internal volumes: 1
[    0.894776] UBI: number of user volumes:     1
[    0.899462] UBI: available PEBs:             4
[    0.904137] UBI: total number of reserved PEBs: 195
[    0.909280] UBI: number of PEBs reserved for bad PEB handling: 5
[    0.915614] UBI: max/mean erase counter: 1/0
[    0.920107] UBI: image sequence number:  441007200
[    0.925159] UBI: background thread "ubi_bgt0d" started, PID 255
[    0.931418] UBIBLK starting
[    0.934360] UBIBLK: device's major: 254
[    0.938434] Got volume fs: device 0/volume 1 of size 186
[    0.945813] OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[    0.952401] TCP cubic registered
[    0.955883] NET: Registered protocol family 17
[    0.960601] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[    0.968713] ThumbEE CPU extension supported.
[    0.974204] clock: disabling unused clocks to save power
[    0.982964] VFS: Mounted root (cramfs filesystem) readonly on device 254:1.
[    0.990693] Freeing init memory: 120K
INIT: version 2.88 booting
[    1.311990] Error -3 while decompressing!
[    1.316284] c0332084(2516)->c3e9e000(4096)
[    1.320600] Error -3 while decompressing!
[    1.324820] c0332a58(2615)->c3e9f000(4096)
... Same message from here on onwards

As you can see, UBI starts just fine, and I'm able to initialize ubiblk and at least mount the filesystem. I thought UBI was supposed to make any badblocks transparent to the upper layers... Is there a problem with that tought, or did my manual tampering (with nand markbad) got in the way of UBI's bad block management capabilities?

I also tried switching to squashfs, but I ran into a similar problem. . I decided to use cramfs over UBI because we had run into this problem before, and after some research it seemed that is was due to cramfs' inability to handle bad blocks, so I decided to give UBI a try, hoping it would solve my problem...

Regards,
--
Guilherme de Oliveira Costa
Firmware Engineer - Autotrac Comercio Telecomunicacoes
www.autotrac.com.br
guilherme.oliveira(at)autotrac.com.br
Phone: +55 61 3307 2666

Esta mensagem e qualquer anexo a ela são documentos confidenciais e direcionados exclusivamente ao(s) destinatário(s). Qualquer uso, desvio, sonegação, supressão, revelação ou divulgação não autorizada é proibida e sujeita às sanções e/ou reparações legais por ato ilícito (Código Penal, Artigos 151 e 152). Caso não seja um dos destinatários expressamente indicados, por favor entre em contato com o remetente, respondendo este e-mail e destrua quaisquer cópias da mensagem original. Qualquer opinião, crítica ou análise descrita nesta mensagem é de responsabilidade única do remetente, a menos quando estiver explicitamente expresso que seja da empresa remetente.

This message and any attachment are confidential information for the sole use of the intended recipients. Any unauthorized use, deviation, withholdment, suppression, disclosure or distribution is prohibited and is subjected to legal sanctions and/or compensations per illicit act (Penal Code, articles 151 and 152). If you are not one of the intended recipients, please contact the sender by reply e-mail and destroy any copy of the original message. Any view, comment or analysis expressed in this message is sole responsibility from the sender, except when it’s specifically expressed that it’s the view, comment or analysis of the company.

Re: Filesystems over UBI can't handle badblocks

[Richard Weinberger]

On Wed, Feb 24, 2016 at 9:08 PM, Guilherme de Oliveira Costa
<guilherme.oliveira@autotrac.com.br> wrote:
> As you can see, UBI starts just fine, and I'm able to initialize ubiblk and at least mount the filesystem. I thought UBI was supposed to make any badblocks transparent to the upper layers... Is there a problem with that tought, or did my manual tampering (with nand markbad) got in the way of UBI's bad block management capabilities?

So, you marked a block as bad, _after_ setting up UBI and your rootfs?
Well, this cannot work as from UBI's point of view the block is now
gone --> the rootfs is missing something.

-- 
Thanks,
//richard

Re: Filesystems over UBI can't handle badblocks

[Ricard Wanderlof]

On Wed, 24 Feb 2016, Richard Weinberger wrote:

> On Wed, Feb 24, 2016 at 9:08 PM, Guilherme de Oliveira Costa
> <guilherme.oliveira@autotrac.com.br> wrote:
> > As you can see, UBI starts just fine, and I'm able to initialize ubiblk and at least mount the filesystem. I thought UBI was supposed to make any badblocks transparent to the upper layers... Is there a problem with that tought, or did my manual tampering (with nand markbad) got in the way of UBI's bad block management capabilities?
> 
> So, you marked a block as bad, _after_ setting up UBI and your rootfs?
> Well, this cannot work as from UBI's point of view the block is now
> gone --> the rootfs is missing something.

Exactly.

ubiblk will handle existing bad blocks in the system, but can not handle 
new ones. While this might sound like a serious limitation, remember that 
blocks don't just 'go bad' randomly, what happens is that with repeated 
erase and write cycles the blocks will eventually wear out. Thus in a 
read-only file system such as cramfs [over ubiblk] which is never 
rewritten, a block that is good will never 'go bad'.

You could run into an issue when upgrading the filesystem however, if it 
has been done sufficiently often to actually wear out the flash. It is 
highly unlikely that that would be an issue in a traditional rootfs on an 
SLC (the flash is specified to handle a minimum of 100 000 erase/write 
cycles per block; that would correspond to rewriting the file system 27 
times a day over a 10 year period), and at any rate, it would be detected 
by UBI during the rewrite process, so such blocks would never be reused 
for writing, but could eventually lead to running out of available space.

With UBI this could happen in the natural course of things, if a block 
starts to exhibit excessive read errors, UBI will 'scrub' the block and 
copy it (transparently) to another physical location in the flash. Given 
enough time, this could theoretically wear out all the blocks in the 
flash, although in practice, the data retention of SLC flashes means that 
it would not occur for much longer than a reasonable product life cycle 
(meaning that the number of times a block would need to be scrubbed due to 
excessive read errors is much less than the specified number of 
erase/write cycles for the flash chip over a reasonable life span).

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Filesystems over UBI can't handle badblocks

[Artem Bityutskiy]

On Wed, 2016-02-24 at 20:08 +0000, Guilherme de Oliveira Costa wrote:
> As you can see, UBI starts just fine, and I'm able to initialize
> ubiblk and at least mount the filesystem. I thought UBI was supposed
> to make any badblocks transparent to the upper layers... Is there a
> problem with that tought, or did my manual tampering (with nand
> markbad) got in the way of UBI's bad block management capabilities?

It is right that UBI makes things transparent to upper layers. However,
this is true only if you follow a set of reasonable rules, like you do
not go and change the MTD partition directly, all the changes go via
UBI.

Marking an eraseblock as bad is a change, and it should be done via
UBI, not directly via MTD.

Now, how to do it via UBI? I think this is not supported, but it can be
implemented, I believe.

Re: Filesystems over UBI can't handle badblocks

[Richard Weinberger]

On Thu, Feb 25, 2016 at 11:49 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> Now, how to do it via UBI? I think this is not supported, but it can be
> implemented, I believe.

Implementing this is rather easy. But what exactly is the use case?
If one wants to test UBI wrt. to bad blocks I could think of a debugfs
knob to do so,
tst_emulate_bitflips can already trigger bad block marking btw.

-- 
Thanks,
//richard

Re: Filesystems over UBI can't handle badblocks

[Artem Bityutskiy]

On Thu, 2016-02-25 at 22:10 +0100, Richard Weinberger wrote:
> On Thu, Feb 25, 2016 at 11:49 AM, Artem Bityutskiy <dedekind1@gmail.c
> om> wrote:
> > 
> > Now, how to do it via UBI? I think this is not supported, but it
> > can be
> > implemented, I believe.
> Implementing this is rather easy.

I agree.

>  But what exactly is the use case?

I guess the reporter would come up with a good justification.

I can imagine one, but it is not very strong: you have a development
device, you mess with bad blocks by marking/unmarking them for some
research reasons. You put UBI image there, then remember that some of
the blocks were bad and want to mark them as bad without re-flashing
UBI. Kind of a developer convenience.

Re: Filesystems over UBI can't handle badblocks

[Richard Weinberger]

Am 29.02.2016 um 18:53 schrieb Guilherme de Oliveira Costa:
>> I guess the reporter would come up with a good justification.
>>
>> I can imagine one, but it is not very strong: you have a development
>> device, you mess with bad blocks by marking/unmarking them for some
>> research reasons. You put UBI image there, then remember that some
>> of the blocks were bad and want to mark them as bad without re-
>> flashing UBI. Kind of a developer convenience.
> 
> Indeed. Originally, we were using cramfs over mtdblock, but were running on all sorts of problems during testing due to random corruptions. I wanted to mess with the partition blocks to check how cramfs over UBI would handle badblocks and bit-errors.

What you can do is mounting debugfs and set /sys/kernel/debug/ubi/ubi<NUM>/tst_emulate_bitflips to 1.

Thanks,
//richard