ubifs error when do bonnie++ test

ubifs error when do bonnie++ test

[Jason Liu]

I'm working on the 2.6.26 with ubifs enabled, but after some stress
test, ubifs give some err. The following is the steps to reproduce.

$ ubiattach /dev/ubi_ctrl -m 2
UBI: attaching mtd2 to ubi1
UBI: physical eraseblock size:   1048576 bytes (1024 KiB)
UBI: logical eraseblock size:    1032192 bytes
UBI: smallest flash I/O unit:    8192
UBI: VID header offset:          8192 (aligned 8192)
UBI: data offset:                16384
UBI: empty MTD device detected
UBI: create volume table (copy #1)
UBI: create volume table (copy #2)
UBI: attached mtd2 to ubi1
UBI: MTD device name:            "nand.rootfs"
UBI: MTD device size:            128 MiB
UBI: number of good PEBs:        128
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             122
UBI: total number of reserved PEBs: 6
UBI: number of PEBs reserved for bad PEB handling: 2
UBI: max/mean erase counter: 0/0
UBI: background thread "ubi_bgt1d" started, PID 2166
UBI device number 1, total 128 LEBs (132120576 bytes, 126.0 MiB),
available 122 LEBs (125927424 byt
es, 120.1 MiB), LEB size 1032192 bytes (1008.0 KiB)

$ ubimkvol /dev/ubi1 -N test -m
Set volume size to 125927424
Volume ID 0, size 122 LEBs (125927424 bytes, 120.1 MiB), LEB size
1032192 bytes (1008.0 KiB), dynam
ic, name "test", alignment 1

$ mount -t ubifs ubi1:test /mnt/ubifs
UBIFS: default file-system created
UBIFS: mounted UBI device 1, volume 0, name "test"
UBIFS: file system size:   116637696 bytes (113904 KiB, 111 MiB, 113 LEBs)
UBIFS: journal size:       8257537 bytes (8064 KiB, 7 MiB, 6 LEBs)
UBIFS: media format:       4 (latest is 4)
UBIFS: default compressor: LZO
UBIFS: reserved for root:  5182151 bytes (5060 KiB)

$ ./bonnie\+\+ -d /mnt/ubifs -s 32 -r 16 -u 0:0 -m FSL -x 1000

when running at 390 times, the following error shows up:

Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...UBI: scrubbed PEB 54, data moved to PEB 69
UBIFS error (pid 4500): ubifs_check_node: bad magic 0x4811831,
expected 0x6101831
UBIFS error (pid 4500): ubifs_check_node: bad node at LEB 22:163392
UBIFS error (pid 4500): ubifs_read_node: expected node type 2
UBIFS warning (pid 4500): ubifs_ro_mode: switched to read-only mode, error -117
Can't delete file Q0010289
Cleaning up test directory after error.
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 384 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 384 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 384 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 392 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30
UBIFS error (pid 4500): make_reservation: cannot reserve 400 bytes in
jhead 1, error -30

When power down and re-attach the mtd2, I get the following information:
$ ubiattach /dev/ubi_ctrl -m 2
UBI: attaching mtd2 to ubi1
UBI: physical eraseblock size:   1048576 bytes (1024 KiB)
UBI: logical eraseblock size:    1032192 bytes
UBI: smallest flash I/O unit:    8192
UBI: VID header offset:          8192 (aligned 8192)
UBI: data offset:                16384
UBI: attached mtd2 to ubi1
UBI: MTD device name:            "nand.rootfs"
UBI: MTD device size:            128 MiB
UBI: number of good PEBs:        128
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 128
UBI: number of PEBs reserved for bad PEB handling: 2
UBI: max/mean erase counter: 1342/101
UBI: background thread "ubi_bgt1d" started, PID 1994
UBI device number 1, total 128 LEBs (132120576 bytes, 126.0 MiB),
available 0 LEBs (0 bytes), LEB s
ize 1032192 bytes (1008.0 KiB)

I found the available PEBs became 0 and total number of reserved PEBs:
128?  What happened and why happen?
Please give your kind help on this? Thanks,

Re: ubifs error when do bonnie++ test

[Adrian Hunter]

Jason Liu wrote:
> I'm working on the 2.6.26 with ubifs enabled, but after some stress
> test, ubifs give some err. The following is the steps to reproduce.

Please apply all recent UBIFS patches and enable debugging and try
again.

You could also enable debugging checks, but that will really slow it down.

Re: ubifs error when do bonnie++ test

[Jason Liu]

2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
> Jason Liu wrote:
>>
>> I'm working on the 2.6.26 with ubifs enabled, but after some stress
>> test, ubifs give some err. The following is the steps to reproduce.
>
> Please apply all recent UBIFS patches and enable debugging and try
> again.
>
> You could also enable debugging checks, but that will really slow it down.
>
>
After apply all the patches from users/dedekind/ubifs-v2.6.26.git
I still get errors with bonnie++ test and only run 3times compared 390
times without the patch

Using uid:0, gid:0.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...Expected 16384 files but only got 16385
Cleaning up test directory after error.

Could you please help on this?Does this means UBIFS has some bugs?

Re: ubifs error when do bonnie++ test

[Adrian Hunter]

Jason Liu wrote:
> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
>> Jason Liu wrote:
>>> I'm working on the 2.6.26 with ubifs enabled, but after some stress
>>> test, ubifs give some err. The following is the steps to reproduce.
>> Please apply all recent UBIFS patches and enable debugging and try
>> again.
>>
>> You could also enable debugging checks, but that will really slow it down.
>>
>>
> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
> I still get errors with bonnie++ test and only run 3times compared 390
> times without the patch
> 
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...Expected 16384 files but only got 16385
> Cleaning up test directory after error.
> 
> Could you please help on this?Does this means UBIFS has some bugs?
> 

I will see if I can reproduce this.

Maybe you could try with debugging checks enabled.

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
> > Jason Liu wrote:
> >>
> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
> >> test, ubifs give some err. The following is the steps to reproduce.
> >
> > Please apply all recent UBIFS patches and enable debugging and try
> > again.
> >
> > You could also enable debugging checks, but that will really slow it down.
> >
> >
> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
> I still get errors with bonnie++ test and only run 3times compared 390
> times without the patch
> 
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...Expected 16384 files but only got 16385
> Cleaning up test directory after error.
> 
> Could you please help on this?Does this means UBIFS has some bugs?

Your eraseblock size is rather not-typical. We did not test UBIFS
on such big eraseblocks, so yes, there may be bugs.

Please, enable UBIFS extra checks and run the tests. Then when it
fails, type dmesg and send the messages please.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
> > Jason Liu wrote:
> >>
> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
> >> test, ubifs give some err. The following is the steps to reproduce.
> >
> > Please apply all recent UBIFS patches and enable debugging and try
> > again.
> >
> > You could also enable debugging checks, but that will really slow it down.
> >
> >
> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
> I still get errors with bonnie++ test and only run 3times compared 390
> times without the patch
> 
> Using uid:0, gid:0.
> Writing with putc()...done
> Writing intelligently...done
> Rewriting...done
> Reading with getc()...done
> Reading intelligently...done
> start 'em...done...done...done...
> Create files in sequential order...done.
> Stat files in sequential order...Expected 16384 files but only got 16385
> Cleaning up test directory after error.

So have you tried to enable UBIFS extra checks?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs error when do bonnie++ test

[Jason Liu]

2009/4/10 Artem Bityutskiy <dedekind@infradead.org>:
> On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
>> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
>> > Jason Liu wrote:
>> >>
>> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
>> >> test, ubifs give some err. The following is the steps to reproduce.
>> >
>> > Please apply all recent UBIFS patches and enable debugging and try
>> > again.
>> >
>> > You could also enable debugging checks, but that will really slow it down.
>> >
>> >
>> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
>> I still get errors with bonnie++ test and only run 3times compared 390
>> times without the patch
>>
>> Using uid:0, gid:0.
>> Writing with putc()...done
>> Writing intelligently...done
>> Rewriting...done
>> Reading with getc()...done
>> Reading intelligently...done
>> start 'em...done...done...done...
>> Create files in sequential order...done.
>> Stat files in sequential order...Expected 16384 files but only got 16385
>> Cleaning up test directory after error.
>
> So have you tried to enable UBIFS extra checks?
Yes, The test is in progress. It's really slow after enable UBIFS
debug and extra checks.It will cost half hour for one test.And the
test case failed without debug off after 600 round of test.So it will
cost 600 * 0,5 = 300 hours with debug on, longer than 10 days.

>
> --
> Best regards,
> Artem Bityutskiy (Битюцкий Артём)
>
>

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Mon, 2009-04-13 at 14:50 +0800, Jason Liu wrote:
> 2009/4/10 Artem Bityutskiy <dedekind@infradead.org>:
> > On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
> >> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
> >> > Jason Liu wrote:
> >> >>
> >> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
> >> >> test, ubifs give some err. The following is the steps to reproduce.
> >> >
> >> > Please apply all recent UBIFS patches and enable debugging and try
> >> > again.
> >> >
> >> > You could also enable debugging checks, but that will really slow it down.
> >> >
> >> >
> >> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
> >> I still get errors with bonnie++ test and only run 3times compared 390
> >> times without the patch
> >>
> >> Using uid:0, gid:0.
> >> Writing with putc()...done
> >> Writing intelligently...done
> >> Rewriting...done
> >> Reading with getc()...done
> >> Reading intelligently...done
> >> start 'em...done...done...done...
> >> Create files in sequential order...done.
> >> Stat files in sequential order...Expected 16384 files but only got 16385
> >> Cleaning up test directory after error.
> >
> > So have you tried to enable UBIFS extra checks?
> Yes, The test is in progress. It's really slow after enable UBIFS
> debug and extra checks.It will cost half hour for one test.And the
> test case failed without debug off after 600 round of test.So it will
> cost 600 * 0,5 = 300 hours with debug on, longer than 10 days.

When you reproduce this, please, do not forget to type dmesg
and save the output. Also, it is good idea to save an image
of your flash.

Could you please tell what is your flash device? I see this:
UBI: physical eraseblock size:   1048576 bytes (1024 KiB)
UBI: logical eraseblock size:    1032192 bytes
UBI: smallest flash I/O unit:    8192
UBI: VID header offset:          8192 (aligned 8192)
UBI: data offset:                16384

So your flash have huge PEB size of 1MiB. Min I/O unit size is
also huge - 8KiB. And no sub-pages. What is this device? Some
kind of NAND?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs error when do bonnie++ test

[Jason Liu]

2009/4/14 Artem Bityutskiy <dedekind@infradead.org>:
> On Mon, 2009-04-13 at 14:50 +0800, Jason Liu wrote:
>> 2009/4/10 Artem Bityutskiy <dedekind@infradead.org>:
>> > On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
>> >> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
>> >> > Jason Liu wrote:
>> >> >>
>> >> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
>> >> >> test, ubifs give some err. The following is the steps to reproduce.
>> >> >
>> >> > Please apply all recent UBIFS patches and enable debugging and try
>> >> > again.
>> >> >
>> >> > You could also enable debugging checks, but that will really slow it down.
>> >> >
>> >> >
>> >> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
>> >> I still get errors with bonnie++ test and only run 3times compared 390
>> >> times without the patch
>> >>
>> >> Using uid:0, gid:0.
>> >> Writing with putc()...done
>> >> Writing intelligently...done
>> >> Rewriting...done
>> >> Reading with getc()...done
>> >> Reading intelligently...done
>> >> start 'em...done...done...done...
>> >> Create files in sequential order...done.
>> >> Stat files in sequential order...Expected 16384 files but only got 16385
>> >> Cleaning up test directory after error.
>> >
>> > So have you tried to enable UBIFS extra checks?
>> Yes, The test is in progress. It's really slow after enable UBIFS
>> debug and extra checks.It will cost half hour for one test.And the
>> test case failed without debug off after 600 round of test.So it will
>> cost 600 * 0,5 = 300 hours with debug on, longer than 10 days.
>
> When you reproduce this, please, do not forget to type dmesg
> and save the output. Also, it is good idea to save an image
> of your flash.
>
> Could you please tell what is your flash device? I see this:
> UBI: physical eraseblock size:   1048576 bytes (1024 KiB)
> UBI: logical eraseblock size:    1032192 bytes
> UBI: smallest flash I/O unit:    8192
> UBI: VID header offset:          8192 (aligned 8192)
> UBI: data offset:                16384
>
> So your flash have huge PEB size of 1MiB. Min I/O unit size is
> also huge - 8KiB. And no sub-pages. What is this device? Some
> kind of NAND?
>
We use 2 MLC nand flash to simulate this virtual device. Each NAND
chip is 4K page size with 218 OOB size.
We can parallel program each NAND chip in order to improve the write
performance.
> --
> Best regards,
> Artem Bityutskiy (Битюцкий Артём)
>
>

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Wed, 2009-04-15 at 16:44 +0800, Jason Liu wrote:
> We use 2 MLC nand flash to simulate this virtual device. Each NAND
> chip is 4K page size with 218 OOB size.
> We can parallel program each NAND chip in order to improve the write
> performance.

I see. Then you want to teach your setup understand sub-pages.
In your case the sub-page will be 2K. Then you'll waste much
less space in UBI. See here:
http://www.linux-mtd.infradead.org/doc/ubi.html#L_subpage

Note, with this all I/O will be in page size units (8K), only
UBI headers I/O will be in sub-page size units. So with this
you will preserve I/O speed, but lessen flash space wastage.

And just in case, please, make sure you have read this:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_mlc

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Wed, 2009-04-15 at 11:50 +0300, Artem Bityutskiy wrote:
> On Wed, 2009-04-15 at 16:44 +0800, Jason Liu wrote:
> > We use 2 MLC nand flash to simulate this virtual device. Each NAND
> > chip is 4K page size with 218 OOB size.
> > We can parallel program each NAND chip in order to improve the write
> > performance.
> 
> I see. Then you want to teach your setup understand sub-pages.
> In your case the sub-page will be 2K. 

Sorry, s/2K/4K/

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs error when do bonnie++ test

[Artem Bityutskiy]

On Mon, 2009-04-13 at 14:50 +0800, Jason Liu wrote:
> 2009/4/10 Artem Bityutskiy <dedekind@infradead.org>:
> > On Thu, 2009-04-02 at 17:49 +0800, Jason Liu wrote:
> >> 2009/4/2 Adrian Hunter <adrian.hunter@nokia.com>:
> >> > Jason Liu wrote:
> >> >>
> >> >> I'm working on the 2.6.26 with ubifs enabled, but after some stress
> >> >> test, ubifs give some err. The following is the steps to reproduce.
> >> >
> >> > Please apply all recent UBIFS patches and enable debugging and try
> >> > again.
> >> >
> >> > You could also enable debugging checks, but that will really slow it down.
> >> >
> >> >
> >> After apply all the patches from users/dedekind/ubifs-v2.6.26.git
> >> I still get errors with bonnie++ test and only run 3times compared 390
> >> times without the patch
> >>
> >> Using uid:0, gid:0.
> >> Writing with putc()...done
> >> Writing intelligently...done
> >> Rewriting...done
> >> Reading with getc()...done
> >> Reading intelligently...done
> >> start 'em...done...done...done...
> >> Create files in sequential order...done.
> >> Stat files in sequential order...Expected 16384 files but only got 16385
> >> Cleaning up test directory after error.
> >
> > So have you tried to enable UBIFS extra checks?
> Yes, The test is in progress. It's really slow after enable UBIFS
> debug and extra checks.It will cost half hour for one test.And the
> test case failed without debug off after 600 round of test.So it will
> cost 600 * 0,5 = 300 hours with debug on, longer than 10 days.

BTW, I guess you could try to look where the problem happens,
and then make the test go straight to that? This would save time?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)