Is there possible to integrate mtd ubi ubifs latest version in one git tree?

Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Nancy]

Hi,
    Is there possible to integrate mtd ubi ubifs latest version in one git tree?
    If the answer is no, will ubifs give patches for porting it to old
linux kernel? The patches only contain the ubifs needed file.
    I know little about kernel, just git clone the whole
ubifs-v2.6.24, I found there is a lot of differences between the
original linux-2.6.24.3 and ubifs-v2.6.24. I doulbt all the
linux/kernel ....or linux/mm files' change are all necessary for UBIFS
to run correctly, or just some of them.
     Will you give me some suggestion?


--
Best wishes,
Nancy

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

Hi,

On Fri, 2008-04-11 at 14:48 +0800, Nancy wrote:
>     Is there possible to integrate mtd ubi ubifs latest version in one git tree?
Well, the mainline is where MTD tree and UBI trees are integrated. UBIFS
git tree is not in main line, so it is separate.

But no, we cannot integrate 3 development trees, it just does not make
sense. Better describe your problem and I'll try to help.

>     If the answer is no, will ubifs give patches for porting it to old
> linux kernel? The patches only contain the ubifs needed file.
We maintain 2.6.21, 2.6.22, 2.6.23, and 2.6.24 ports. They contain UBIFS
patches and all the other patches which are needed to make UBIFS work.
For example, 2.6.21 port has tons of UBI patches back-ported from the
ubi-2.6.git tree, it has LZO compressor ported from older versions of
the kernel as well.

>     I know little about kernel, just git clone the whole
> ubifs-v2.6.24, I found there is a lot of differences between the
> original linux-2.6.24.3 and ubifs-v2.6.24.
Sure. But it is easy to make linux-2.6.24.3 + UBIFS. Really. Suppose you
have linux-2.6.24.3 tree at /home/nancy/linux-2.6.24.3 and ubifs-v2.6.24
at /home/nanxy/ubifs-v2.6.24. Do the following:

# cd /home/nanxy/ubifs-v2.6.24
# git diff v2.6.24 > all_ubifs.diff
# cd /home/nancy/linux-2.6.24.3
# patch -p1 < all_ubifs.diff

And that's it.

If you do not want to loose UBIFS history, you could do like this
# cd /home/nanxy/ubifs-v2.6.24
# git format-patch v2.6.24
This will give you huge amount of patches, basically all patches which
we applied to vanilla linux-2.6.24. Now you may apply them to your
linux-2.6.24.3. The easiest way is to use git-am. But first you have to
make sure your linux-2.6.24.3 is a git repository. If it is not (you
downloaded it as a tar.gz file, create a git repository using git
init-db and then git commit).

# cd /home/nancy/linux-2.6.24.3
# git init-db
# git add .
# git-commit -a

Now you have git repository. Then

# mv /home/nanxy/ubifs-v2.6.24/0* .
# git am 0*

And this will apply all the patches in right order (because they have
prefix with number and will be sorted by name). But it you may have some
conflicts, which you will have to fix manually. And after all patches
were applied, you may look what you have with git log.

HTH.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

On Sat, 2008-04-12 at 12:57 +0300, Artem Bityutskiy wrote:
> >     If the answer is no, will ubifs give patches for porting it to old
> > linux kernel? The patches only contain the ubifs needed file.
> We maintain 2.6.21, 2.6.22, 2.6.23, and 2.6.24 ports. They contain UBIFS
> patches and all the other patches which are needed to make UBIFS work.
> For example, 2.6.21 port has tons of UBI patches back-ported from the
> ubi-2.6.git tree, it has LZO compressor ported from older versions of
> the kernel as well.

Corrections:
1. Well, 2.6.21 did not have UBI at all, so ubifs-v2.6.21.git contains
all UBI patches, not just UBIFS.
2. LZO was ported from newer kernels of course, not from older.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

On Sat, 2008-04-12 at 12:57 +0300, Artem Bityutskiy wrote:
> If you do not want to loose UBIFS history, you could do like this
> # cd /home/nanxy/ubifs-v2.6.24
> # git format-patch v2.6.24
> This will give you huge amount of patches, basically all patches which
> we applied to vanilla linux-2.6.24. Now you may apply them to your
> linux-2.6.24.3. The easiest way is to use git-am. But first you have to
> make sure your linux-2.6.24.3 is a git repository. If it is not (you
> downloaded it as a tar.gz file, create a git repository using git
> init-db and then git commit).
> 
> # cd /home/nancy/linux-2.6.24.3
> # git init-db
> # git add .
> # git-commit -a

Well, you might as well use quilt, which is an extremely helpful tool.
So you have many patches in /home/nanxy/ubifs-v2.6.24, like
"0061-UBIFS-use-qstr-for-name-length-pairs.patch". What you can do is:

# ls -1 0* > series
# cd /home/nancy/linux-2.6.24.3
# mkdir patches
# mv /home/nanxy/ubifs-v2.6.24/series patches/
# mv /home/nanxy/ubifs-v2.6.24/0* patches/
# quilt push -a

If there are conflicts, it is easier to resolve them with quilt. Often
it is just about "quilt push; quilt refresh".

Ant after you resolved all conflicts, and all patches apply cleanly (use
quilt refresh), you may incorporate all patches to git:

# quilt pop -q
# git reset --hard
# git quiltimport

And you now may delete patches/ and .pc which is private quilt working
directory.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Nancy]

Hi, Aterm
     First of all, I don't know how to express my deep thankfulness to
you. You are such busy man, but replyed so carefully, patiently and
clearly. That's a big suprise to me :-)  Thank you !

     Hmm..
     I simply use the way you told to port UBI & UBIFS to mine.
> # cd /home/nanxy/ubifs-v2.6.24
> # git diff v2.6.24 > all_ubifs.diff
> # cd /home/nancy/linux-2.6.24.3
> # patch -p1 < all_ubifs.diff

    In fact, I found a serious bug which makes me consider to update
the UBI & UBIFS to the latest version. But it still exist!

# flash_eraseall /dev/mtd5
# nandwrite -a -m -q /dev/mtd5 ubi0414

# modprobe ubi mtd=5
UBI: attached mtd5 to ubi0
UBI: MTD device name:            "NAND VFAT partition"
UBI: MTD device size:            512 MiB
UBI: physical eraseblock size:   262144 bytes (256 KiB)
UBI: logical eraseblock size:    258048 bytes
UBI: number of good PEBs:        2048
UBI: number of bad PEBs:         0
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     2
UBI: available PEBs:             398
UBI: total number of reserved PEBs: 1650
UBI: number of PEBs reserved for bad PEB handling: 20
UBI: max/mean erase counter: 1/0
UBI: background thread "ubi_bgt0d" started, PID 328
# lsmod
Module                  Size  Used by    Not tainted
ubi                    95344  0
# modprobe ubifs
# lsmod
Module                  Size  Used by    Not tainted
ubifs                 271360  0
ubi                    95344  1 ubifs
# mount -t ubifs ubi0:ubifs /mnt/1
UBIFS: mounted UBI device 0, volume 0
UBIFS: minimal I/O unit size:   2048 bytes
UBIFS: logical eraseblock size: 258048 bytes (252 KiB)
UBIFS: file system size:        207212544 bytes (202356 KiB, 197 MiB, 803 LEBs)
UBIFS: journal size:            9420800 bytes (9200 KiB, 8 MiB, 37 LEBs)
UBIFS: data journal heads:      1
UBIFS: default compressor:      LZO
# ls /mnt/1
bin      etc      lib      mnt      proc     sbin     tmp      var
dev      home     linuxrc  opt      root     sys      usr
# umount /mnt/1
UBIFS: un-mount UBI device 0, volume 0
# rmmod ubifs
# rmmod ubi
UBI: mtd5 is detached from ubi0
# modprobe ubi mtd=5
UBI: attached mtd5 to ubi0
UBI: MTD device name:            "NAND VFAT partition"
UBI: MTD device size:            512 MiB
UBI: physical eraseblock size:   262144 bytes (256 KiB)
UBI: logical eraseblock size:    258048 bytes
UBI: number of good PEBs:        2048
UBI: number of bad PEBs:         0
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     2
UBI: available PEBs:             398
UBI: total number of reserved PEBs: 1650
UBI: number of PEBs reserved for bad PEB handling: 20
UBI: max/mean erase counter: 4/0
UBI: background thread "ubi_bgt0d" started, PID 369
# modprobe ubifs
# lsmod
Module                  Size  Used by    Not tainted
ubifs                 271360  0
ubi                    95344  1 ubifs
# mount -t ubifs ubi0:ubifs /mnt/1
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
3:4096, read 258048 bytes
UBIFS error (pid 375): ubifs_scan: corrupt empty space at LEB 1:4096
UBIFS error (pid 375): ubifs_scanned_corruption: corrupted data at LEB 1:4096
UBIFS error (pid 375): ubifs_scanned_corruption: first 4096 bytes from
LEB 1:4096
UBIFS error (pid 375): ubifs_scan: LEB 1 scanning failed
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
NAND: Uncorrectable ECC error
UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
3:4096, read 258048 bytes
UBIFS error (pid 375): ubifs_recover_master_node: failed to recover master node
UBIFS error (pid 375): ubifs_recover_master_node: dumping first master node
CPU 0 Unable to handle kernel paging request at virtual address
c021e003, epc == c014a9e0, ra == c01467d0
Oops[#1]:
Cpu 0
$ 0   : 00000000 10000400 c014a9b4 00000000
$ 4   : 83fb8800 c021e000 00000001 00000000
$ 8   : 8043e280 00000002 00004001 80490000
$12   : 80490000 80490000 000000a0 00000038
$16   : 8003e3a0 c0150000 ffffffea 83fb8800
$20   : 83fb8b70 c0130000 c015c380 c0160000
$24   : 00000002 8009a3d0
$28   : 83efc000 83efdbe0 83fb8800 c01467d0
Hi    : 000000a7
Lo    : 86c91000
epc   : c014a9e0 dbg_dump_node+0x2c/0xd20 [ubifs]     Not tainted
ra    : c01467d0 ubifs_recover_master_node+0xe4/0x2f4 [ubifs]
Status: 10000403    KERNEL EXL IE
Cause : 00800008
BadVA : c021e003
PrId  : 0ad0024f (Ingenic JZRISC)
Modules linked in: ubifs ubi
Process mount (pid: 375, threadinfo=83efc000, task=83c87198)
Stack : ffffffea 00000800 8003e3a0 c0150000 ffffffea 83fb8800 83fb8b70 c0130000
        c015c380 c0160000 8003e3a0 c0150000 ffffffea 83fb8800 83fb8b70 c0130000
        c015c380 c0160000 83fb8800 c01467d0 00001000 00000177 c014d3dc 00000000
        83efdc50 00000001 00000000 00000000 00000000 00000000 c021e000 00000000
        83d90da8 83fb8800 ffffff79 00000000 83fb8b70 c0130464 83fb8b42 83fb8800
        ...
Call Trace:
[<c014a9e0>] dbg_dump_node+0x2c/0xd20 [ubifs]
[<c01467d0>] ubifs_recover_master_node+0xe4/0x2f4 [ubifs]
[<c0130464>] ubifs_read_master+0xb4/0xaec [ubifs]
[<c011f0b8>] ubifs_get_sb+0xe48/0x17c0 [ubifs]
[<800ae134>] vfs_kern_mount+0x64/0x10c
[<800ae238>] do_kern_mount+0x4c/0x134
[<800ce500>] do_mount+0x1c0/0x6e4
[<800ceabc>] sys_mount+0x98/0xfc
[<80021df0>] stack_done+0x20/0x3c


Code: afb40038  afb1002c  afb00028 <88a30003> 3c020610  98a30000
34421831  00a09021  10620022
Segmentation fault
#


pls:

UBIFS UBI: commit 2bee7c9d6d42df3af73d36749760319592c68a96
mkfs.ubifs: commit 20dbf0ae5792e5b13153f676181f83336508d820
mtd-utils: commit fcb52ccc99679460640386c297023f852b108f68

Both root26.img and ubi0414 created on PC
[yrtan@st mkfs.ubifs]$ ./mkfs.ubifs -r /nfsroot/release/rootfs/root26
-m 2048 -e 258048 -c 813 -o root26.img
[yrtan@st new-utils]$ ./ubinize -o ubi0414 ubinize.cfg -p 262144 -m 2048
[yrtan@st new-utils]$ cat ubinize.cfg
[ubifs]
mode=ubi
image=root26.img
vol_id=0
vol_size=200MiB
vol_type=dynamic
vol_name=ubifs
vol_alignment=1
vol_flag=autoresize

[vfat]
mode=ubi
image=vfat.img
vol_id=1
vol_size=298MiB
vol_type=dynamic
vol_name=vfat
vol_alignment=1
vol_flag=autoresize

For original test, I use the origial UBI code, do not add any of the
UBI block layer codes. Though the ubi0414 contain vfat.img, the whole
process do not use it. I think it won't effect the test result.

Maybe there's bug in "ubinize".  Cause if I use "ubiupdatevol",
everything is OK.


--
Best wishes,
Nancy

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

On Mon, 2008-04-14 at 12:46 +0800, Nancy wrote:
>     In fact, I found a serious bug which makes me consider to update
> the UBI & UBIFS to the latest version. But it still exist!

Hmm, good that you wrote about this.

> # flash_eraseall /dev/mtd5

I guess you know that UBIFS stores erase-counter at the beginning of the
eraseblock. What you do with this command you just wipe the flash out
and loose the erase-counters. This is not good.

Partially it is our fault that we did not tell about this loudly at the
web site. I also put examples which use flash_eraseall, but that was for
nandsim. I really did not think about this scenario. We use our own
flasher for flashing images which does preserve erase-counters.

If you do like this often, you are risking to wear-out some eraseblocks
of your flash which is bad.

We have create a separate utility for erasing the flash and preserving
erase-counters (well, actually incrementing them). We will do this ASAP,
may be next week. We will also fix the web site to inform people about
possible consequences of using flash_eraseall for MTD devices which are
used for UBI.

For now, try to avoid using flash_eraseall and use ubiupdatevol for
updating.

> # nandwrite -a -m -q /dev/mtd5 ubi0414

Similarly, here you copy the image which sets all erase-counters to 0.
We will create a utility which you will be able to use instead of
nandwrite soon. Try to use ubiupdatevol so far.


> # modprobe ubi mtd=5
> UBI: attached mtd5 to ubi0
> UBI: MTD device name:            "NAND VFAT partition"
> UBI: MTD device size:            512 MiB
> UBI: physical eraseblock size:   262144 bytes (256 KiB)
> UBI: logical eraseblock size:    258048 bytes
> UBI: number of good PEBs:        2048
> UBI: number of bad PEBs:         0
> UBI: smallest flash I/O unit:    2048
> UBI: VID header offset:          2048 (aligned 2048)
> UBI: data offset:                4096
> UBI: max. allowed volumes:       128
> UBI: wear-leveling threshold:    4096
> UBI: number of internal volumes: 1
> UBI: number of user volumes:     2
> UBI: available PEBs:             398
> UBI: total number of reserved PEBs: 1650
> UBI: number of PEBs reserved for bad PEB handling: 20
> UBI: max/mean erase counter: 1/0
> UBI: background thread "ubi_bgt0d" started, PID 328

I suspect you have MLC NAND, do you? Could you please tell how many
erase-cycles the eraseblock may survive on your flash.


> # mount -t ubifs ubi0:ubifs /mnt/1
> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error

This looks like a bad eraseblock. Try to figure out what is this,
may be putting more information.

> UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
> 3:4096, read 258048 bytes
> UBIFS error (pid 375): ubifs_scan: corrupt empty space at LEB 1:4096
> UBIFS error (pid 375): ubifs_scanned_corruption: corrupted data at LEB 1:4096
> UBIFS error (pid 375): ubifs_scanned_corruption: first 4096 bytes from
> LEB 1:4096
> UBIFS error (pid 375): ubifs_scan: LEB 1 scanning failed

Could you please enable UBIFS debugging. Also, please enable UBI
debugging and enable UBI extra self-checks (not messages, just the
checks). This will make it very slow, but it may help to identify the
problem. Please,do this and send us the dmesg output.

Also, attach your .config next time please.

> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error
> NAND: Uncorrectable ECC error
> UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
> 3:4096, read 258048 bytes
> UBIFS error (pid 375): ubifs_recover_master_node: failed to recover master node
> UBIFS error (pid 375): ubifs_recover_master_node: dumping first master node
> CPU 0 Unable to handle kernel paging request at virtual address
> c021e003, epc == c014a9e0, ra == c01467d0
> Oops[#1]:
> Cpu 0
> $ 0   : 00000000 10000400 c014a9b4 00000000
> $ 4   : 83fb8800 c021e000 00000001 00000000
> $ 8   : 8043e280 00000002 00004001 80490000
> $12   : 80490000 80490000 000000a0 00000038
> $16   : 8003e3a0 c0150000 ffffffea 83fb8800
> $20   : 83fb8b70 c0130000 c015c380 c0160000
> $24   : 00000002 8009a3d0
> $28   : 83efc000 83efdbe0 83fb8800 c01467d0
> Hi    : 000000a7
> Lo    : 86c91000
> epc   : c014a9e0 dbg_dump_node+0x2c/0xd20 [ubifs]     Not tainted

This oops looks like a bug in the dump function, we'll look at this.
But anyway, the root of the error is somewhere at the low level. Those
unrecoverable ECC errors tell about this. Probably you worn out few
eraseblocks.

> ra    : c01467d0 ubifs_recover_master_node+0xe4/0x2f4 [ubifs]
Yeah, the problem is somewhere at master node. This makes my theory that
you worn out an eraseblock more probable. The reason is that you wiped
out your flash all the time and lost erasecounters. And if you have MLC,
this should be very easy to do.

> [yrtan@st new-utils]$ cat ubinize.cfg
> [ubifs]
> mode=ubi
> image=root26.img
> vol_id=0
> vol_size=200MiB
> vol_type=dynamic
> vol_name=ubifs
> vol_alignment=1
> vol_flag=autoresize
> 
> [vfat]
> mode=ubi
> image=vfat.img
> vol_id=1
> vol_size=298MiB
> vol_type=dynamic
> vol_name=vfat
> vol_alignment=1
> vol_flag=autoresize

Hmm, you mark 2 volumes as auto-resize, which is wrong. Only one may
have this flag. I will fix the utility and make it complain about this.
Also, it is strange I do not see UBI messages saying it was doing
auto-resize.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Jamie Lokier]

Artem Bityutskiy wrote:
> > # flash_eraseall /dev/mtd5
> 
> I guess you know that UBIFS stores erase-counter at the beginning of the
> eraseblock. What you do with this command you just wipe the flash out
> and loose the erase-counters. This is not good.
> 
> Partially it is our fault that we did not tell about this loudly at the
> web site. I also put examples which use flash_eraseall, but that was for
> nandsim. I really did not think about this scenario. We use our own
> flasher for flashing images which does preserve erase-counters.
> 
> If you do like this often, you are risking to wear-out some eraseblocks
> of your flash which is bad.
> 
> We have create a separate utility for erasing the flash and preserving
> erase-counters (well, actually incrementing them). We will do this ASAP,
> may be next week. We will also fix the web site to inform people about
> possible consequences of using flash_eraseall for MTD devices which are
> used for UBI.

I'm not sure if UBI stores erase counters for blocks in _other_ blocks.

If erase counters are stored at the start of each block, for the block
itself, what happens in this sequence?

    1. Begin erase block.
    2. Erase succeeds.
    3. Power failure / system reset / watchdog / etc.
    4. ....?

Is the erase counter zeroed as a result of this?

One of the nicest features of JFFS2 and UBIFS is that they are robust
against power failures and other unexpected reboot scenarios.

Some devices use this by not having a shutdown sequence: the normal
method of shutdown is to turn off the power.  It's fast.

It would be unfortunate if UBI's erase levelling info were lost by
doing this.

-- Jamie









> 
> For now, try to avoid using flash_eraseall and use ubiupdatevol for
> updating.
> 
> > # nandwrite -a -m -q /dev/mtd5 ubi0414
> 
> Similarly, here you copy the image which sets all erase-counters to 0.
> We will create a utility which you will be able to use instead of
> nandwrite soon. Try to use ubiupdatevol so far.
> 
> 
> > # modprobe ubi mtd=5
> > UBI: attached mtd5 to ubi0
> > UBI: MTD device name:            "NAND VFAT partition"
> > UBI: MTD device size:            512 MiB
> > UBI: physical eraseblock size:   262144 bytes (256 KiB)
> > UBI: logical eraseblock size:    258048 bytes
> > UBI: number of good PEBs:        2048
> > UBI: number of bad PEBs:         0
> > UBI: smallest flash I/O unit:    2048
> > UBI: VID header offset:          2048 (aligned 2048)
> > UBI: data offset:                4096
> > UBI: max. allowed volumes:       128
> > UBI: wear-leveling threshold:    4096
> > UBI: number of internal volumes: 1
> > UBI: number of user volumes:     2
> > UBI: available PEBs:             398
> > UBI: total number of reserved PEBs: 1650
> > UBI: number of PEBs reserved for bad PEB handling: 20
> > UBI: max/mean erase counter: 1/0
> > UBI: background thread "ubi_bgt0d" started, PID 328
> 
> I suspect you have MLC NAND, do you? Could you please tell how many
> erase-cycles the eraseblock may survive on your flash.
> 
> 
> > # mount -t ubifs ubi0:ubifs /mnt/1
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> 
> This looks like a bad eraseblock. Try to figure out what is this,
> may be putting more information.
> 
> > UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
> > 3:4096, read 258048 bytes
> > UBIFS error (pid 375): ubifs_scan: corrupt empty space at LEB 1:4096
> > UBIFS error (pid 375): ubifs_scanned_corruption: corrupted data at LEB 1:4096
> > UBIFS error (pid 375): ubifs_scanned_corruption: first 4096 bytes from
> > LEB 1:4096
> > UBIFS error (pid 375): ubifs_scan: LEB 1 scanning failed
> 
> Could you please enable UBIFS debugging. Also, please enable UBI
> debugging and enable UBI extra self-checks (not messages, just the
> checks). This will make it very slow, but it may help to identify the
> problem. Please,do this and send us the dmesg output.
> 
> Also, attach your .config next time please.
> 
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> > NAND: Uncorrectable ECC error
> > UBI error: ubi_io_read: error -77 while reading 258048 bytes from PEB
> > 3:4096, read 258048 bytes
> > UBIFS error (pid 375): ubifs_recover_master_node: failed to recover master node
> > UBIFS error (pid 375): ubifs_recover_master_node: dumping first master node
> > CPU 0 Unable to handle kernel paging request at virtual address
> > c021e003, epc == c014a9e0, ra == c01467d0
> > Oops[#1]:
> > Cpu 0
> > $ 0   : 00000000 10000400 c014a9b4 00000000
> > $ 4   : 83fb8800 c021e000 00000001 00000000
> > $ 8   : 8043e280 00000002 00004001 80490000
> > $12   : 80490000 80490000 000000a0 00000038
> > $16   : 8003e3a0 c0150000 ffffffea 83fb8800
> > $20   : 83fb8b70 c0130000 c015c380 c0160000
> > $24   : 00000002 8009a3d0
> > $28   : 83efc000 83efdbe0 83fb8800 c01467d0
> > Hi    : 000000a7
> > Lo    : 86c91000
> > epc   : c014a9e0 dbg_dump_node+0x2c/0xd20 [ubifs]     Not tainted
> 
> This oops looks like a bug in the dump function, we'll look at this.
> But anyway, the root of the error is somewhere at the low level. Those
> unrecoverable ECC errors tell about this. Probably you worn out few
> eraseblocks.
> 
> > ra    : c01467d0 ubifs_recover_master_node+0xe4/0x2f4 [ubifs]
> Yeah, the problem is somewhere at master node. This makes my theory that
> you worn out an eraseblock more probable. The reason is that you wiped
> out your flash all the time and lost erasecounters. And if you have MLC,
> this should be very easy to do.
> 
> > [yrtan@st new-utils]$ cat ubinize.cfg
> > [ubifs]
> > mode=ubi
> > image=root26.img
> > vol_id=0
> > vol_size=200MiB
> > vol_type=dynamic
> > vol_name=ubifs
> > vol_alignment=1
> > vol_flag=autoresize
> > 
> > [vfat]
> > mode=ubi
> > image=vfat.img
> > vol_id=1
> > vol_size=298MiB
> > vol_type=dynamic
> > vol_name=vfat
> > vol_alignment=1
> > vol_flag=autoresize
> 
> Hmm, you mark 2 volumes as auto-resize, which is wrong. Only one may
> have this flag. I will fix the utility and make it complain about this.
> Also, it is strange I do not see UBI messages saying it was doing
> auto-resize.
> 
> -- 
> Best regards,
> Artem Bityutskiy (Битюцкий Артём)
> 
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

On Tue, 2008-04-15 at 12:41 +0100, Jamie Lokier wrote:
> If erase counters are stored at the start of each block, for the block
> itself, what happens in this sequence?
> 
>     1. Begin erase block.
>     2. Erase succeeds.
>     3. Power failure / system reset / watchdog / etc.
>     4. ....?
> 
> Is the erase counter zeroed as a result of this?

It tries to minimize probability of situations like this by writing the
EC right after the erasure. But this still can happen of course. In this
case UBI uses the average erase-counter (calculated while scanning is
done) and writes it to this eraseblock.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Jamie Lokier]

Artem Bityutskiy wrote:
> On Tue, 2008-04-15 at 12:41 +0100, Jamie Lokier wrote:
> > If erase counters are stored at the start of each block, for the block
> > itself, what happens in this sequence?
> > 
> >     1. Begin erase block.
> >     2. Erase succeeds.
> >     3. Power failure / system reset / watchdog / etc.
> >     4. ....?
> > 
> > Is the erase counter zeroed as a result of this?
> 
> It tries to minimize probability of situations like this by writing the
> EC right after the erasure. But this still can happen of course. In this
> case UBI uses the average erase-counter (calculated while scanning is
> done) and writes it to this eraseblock.

Oh, using the averase erase-counter is smart, I like that. :-)

Erase can be quite slow on some kinds of flash, and the race window is
the whole erase time.  And during some activities, many sequential
erases are done.

It would be good if the erase-counters (and indeed metadata needed for
fast mounting) were also records in the log/trees held in other
blocks, but I guess you'd need to closely couple with the filesystems
logging.  And that would be a layer violation, tricky and ugly in the
UBI API.

-- Jamie

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

n Tue, 2008-04-15 at 16:21 +0100, Jamie Lokier wrote:
> It would be good if the erase-counters (and indeed metadata needed for
> fast mounting) were also records in the log/trees held in other
> blocks, but I guess you'd need to closely couple with the filesystems
> logging.  And that would be a layer violation, tricky and ugly in the
> UBI API.

Yeah, that would be nice. Well, if one really wants it is possible to
teach UBI to save erase-counter somewhere else before erasing. Should
not be difficult. UBI has general mechanism of so-called "internal
volumes" which may be added and used for that purposes. But it does not
sound worth the trouble for me :-)

But fast mounting is a separate area which would need full re-design and
many additional complications. As I said before, we are planning to at
lease create a short document about the possible UBI2 design.

The strength of current UBI is its relative simplicity, thus robustness.
It does not have any on-flash trees. UBIFS does - and the complexity is
an order of magnitude higher then UBI's complexity :-)

Imagine you would store some kind of logs and trees on the flash in UBI.
You would refer physical eraseblock from these trees by numbers, surely?
Think about difficulties introduced by randomly distributed bad
eraseblocks. With current UBI, one can prepare an image and send it to
factory, where it will be flashed to all devices. The same image,
irrespectively on how bad blocks are distributed. Simple. But in case
UBI2 (imaginable) which would have some trees, where a tree could refer,
say, an eraseblock 1006, things would have been much more difficult.
This eraseblock 1006 could be a bad block for example. This is only one
example.

But that stuff is doable I believe, although trickier that current UBI.
And we hope, once UBI/UBIFS gets stabilized and used, we or someone else
may do the tricks.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Nancy]

>  > I'm looking forward to the new tools come out :-)
>  >
>  > > I suspect you have MLC NAND, do you? Could you please tell how many
>  > > erase-cycles the eraseblock may survive on your flash.
>  > Yes, MLC Nand,  Samsung K9G8G08U0M , 10K program/erase cycles(with
>  > 4bit/512byte ECC).
>
>  Then first of all you cannot use the default 4096 UBI WL threshold. Make
>  it 256 for example. 4K is almost lifetime of your eraseblock!
       Why 4K is almost lifetime of my eraseblock? the specs says 10K,
how you result that number 4?
       How to set the WL threshold? any equation?
       When eraseblock counter exceed 2*WL threshold, it will be less
used, move its data to seldom used eraseblock?
       Why WL can't just simply pick up the min used PEB to be the
next block to use?

-- 
Best wishes,
Nancy

Re: Is there possible to integrate mtd ubi ubifs latest version in one git tree?

[Artem Bityutskiy]

On Tue, 2008-04-15 at 08:04 -0400, Nancy wrote:
> >  Then first of all you cannot use the default 4096 UBI WL threshold. Make
> >  it 256 for example. 4K is almost lifetime of your eraseblock!
>        Why 4K is almost lifetime of my eraseblock? the specs says 10K,
> how you result that number 4?
I meant they are close and you should use something an order of
magnitude less then 4K.

>        How to set the WL threshold? any equation?
No equation. I'd suggest 256 for you. There is a configuration option
for this.

>        When eraseblock counter exceed 2*WL threshold, it will be less
> used, move its data to seldom used eraseblock?
>        Why WL can't just simply pick up the min used PEB to be the
> next block to use?

Consider a situation when you have static data, which never changes, and
it sits a a subset of eraseblock N, while the rest of the flash is
intensively used. In this case UBI moves data from N to more worn-out
eraseblocks, and start using N. And the threshold says which should be
the difference between erase-counter of eraseblocks N and the other
ones, roughly speaking.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)