[RFC] slight UBI scan time improvement

[RFC] slight UBI scan time improvement

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

Hi,

I've prepared 2 UBI patches which may slightly improve the scan time. I
am not sure though. Would you guys please try them and tell if UBI scan
time changed? Thanks in advance.

Attached.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

[-- Attachment #2: 0001-UBI-separate-out-header-checking.patch --]
[-- Type: application/mbox, Size: 14244 bytes --]

[-- Attachment #3: 0002-UBI-optimize-scanning-a-bit.patch --]
[-- Type: application/mbox, Size: 10723 bytes --]

Re: [RFC] slight UBI scan time improvement

[Bruce_Leonard]

> 
> I've prepared 2 UBI patches which may slightly improve the scan time. I
> am not sure though. Would you guys please try them and tell if UBI scan
> time changed? Thanks in advance.
> 

Hi Artem,

I'll be happy to do so, but it may be a couple of days before I can get 
results to you.  I'm on the road for my company for a few days.  If I can, 
I'll try it out before I leave tonight, but if I can't get to it, it'll be 
tommorrow night before I can get my equipment set up and running again.

See 'ya!

Bruce

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Tue, 2008-04-22 at 10:28 -0700, Bruce_Leonard@selinc.com wrote:
> I'll be happy to do so, but it may be a couple of days before I can get 
> results to you.  I'm on the road for my company for a few days.  If I can, 
> I'll try it out before I leave tonight, but if I can't get to it, it'll be 
> tommorrow night before I can get my equipment set up and running again.

Thanks Bruce,

no problem, this is not urgent and not that important.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Nancy]

> I've prepared 2 UBI patches which may slightly improve the scan time. I
> am not sure though. Would you guys please try them and tell if UBI scan
> time changed? Thanks in advance.
I have tried, as you said slightly improve the scan time, slightly,
even don't feel it exist. en....I don't like the two stages displaying
when ubi attach to mtd. First time I see it, I doubt the NFS server is
too busy. It is just like a kid playing sliding board who do not like
being block in the middle, or a group of men "cheers!", then drink off
the beer. Oh, don't make it stop in the half way.

You still do not add patch to handle write fail issue, a little disappointment.

-- 
Best wishes,
Nancy

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 15:15 +0800, Nancy wrote:
> I have tried, as you said slightly improve the scan time, slightly,
> even don't feel it exist. en....I don't like the two stages displaying
> when ubi attach to mtd. First time I see it, I doubt the NFS server is
> too busy. It is just like a kid playing sliding board who do not like
> being block in the middle, or a group of men "cheers!", then drink off
> the beer. Oh, don't make it stop in the half way.

Hmm, may be. But the reason for this is that we want to see this
information even if UBI refuses your MTD device later.

> You still do not add patch to handle write fail issue, a little disappointment.

I do not see any UBI/UBIFS issue in your case. The issue was that you
abused your flash using wrong tools, I think. Use the "ubiformat"
utility which we have recently implemented - the issue should go.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Tue, Apr 22, 2008 at 07:42:32PM +0300, Artem Bityutskiy wrote:
> Hi,
> 
> I've prepared 2 UBI patches which may slightly improve the scan time. I
> am not sure though. Would you guys please try them and tell if UBI scan
> time changed? Thanks in advance.

Hi Artem,

Thanks for your patch.. unfortunately I don't see a significant
difference, although it is slightly faster.

Before:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.900000] UBI: attached mtd9 to ubi0

After:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.890000] UBI: attached mtd9 to ubi0



Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [RFC] slight UBI scan time improvement

[Nancy]

On 4/23/08, Artem Bityutskiy <dedekind@infradead.org> wrote:
> On Wed, 2008-04-23 at 15:15 +0800, Nancy wrote:
> > I have tried, as you said slightly improve the scan time, slightly,
> > even don't feel it exist. en....I don't like the two stages displaying
> > when ubi attach to mtd. First time I see it, I doubt the NFS server is
> > too busy. It is just like a kid playing sliding board who do not like
> > being block in the middle, or a group of men "cheers!", then drink off
> > the beer. Oh, don't make it stop in the half way.
>
> Hmm, may be. But the reason for this is that we want to see this
> information even if UBI refuses your MTD device later.
   Thank you for your explaination.

> > You still do not add patch to handle write fail issue, a little disappointment.
>
> I do not see any UBI/UBIFS issue in your case. The issue was that you
> abused your flash using wrong tools, I think. Use the "ubiformat"
> utility which we have recently implemented - the issue should go.
    Thanks for the "ubiformat " tool.
    Yes, the issue is go away by properly use MLC nand.
    But the write fail issue in UBI still exist! Suppose you are
properly used your Nand, when it return write fail, but can erase
successfully, UBI still group this block in good block heap. That is
wrong and dangerous.


-- 
Best wishes,
Nancy

Re: [RFC] slight UBI scan time improvement

[Matthieu CASTET]

Hamish Moffatt wrote:
> On Tue, Apr 22, 2008 at 07:42:32PM +0300, Artem Bityutskiy wrote:
>> Hi,
>>
>> I've prepared 2 UBI patches which may slightly improve the scan time. I
>> am not sure though. Would you guys please try them and tell if UBI scan
>> time changed? Thanks in advance.
> 
> Hi Artem,
> 
> Thanks for your patch.. unfortunately I don't see a significant
> difference, although it is slightly faster.
> 
> Before:
> 
> [    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> [    0.960000] Scanning device for bad blocks
> [    1.000000] Bad eraseblock 494 at 0x03dc0000
> [    1.050000] Bad eraseblock 1300 at 0x0a280000
> [    1.140000] Bad eraseblock 2554 at 0x13f40000
> [    1.160000] Bad eraseblock 2923 at 0x16d60000
> [    1.200000] Bad eraseblock 3349 at 0x1a2a0000
> [    1.230000] Bad eraseblock 3790 at 0x1d9c0000
> [    6.900000] UBI: attached mtd9 to ubi0
> 
> After:
> 
> [    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> [    0.960000] Scanning device for bad blocks
> [    1.000000] Bad eraseblock 494 at 0x03dc0000
> [    1.050000] Bad eraseblock 1300 at 0x0a280000
> [    1.140000] Bad eraseblock 2554 at 0x13f40000
> [    1.160000] Bad eraseblock 2923 at 0x16d60000
> [    1.200000] Bad eraseblock 3349 at 0x1a2a0000
> [    1.230000] Bad eraseblock 3790 at 0x1d9c0000
> [    6.890000] UBI: attached mtd9 to ubi0
> 
> 
> 
> Hamish

Do you know when the bad block scanning finish and the ubi scan start ?


Matthieu

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 16:01 +0800, Nancy wrote:
>     Yes, the issue is go away by properly use MLC nand.
>     But the write fail issue in UBI still exist! Suppose you are
> properly used your Nand, when it return write fail, but can erase
> successfully, UBI still group this block in good block heap. That is
> wrong and dangerous.

The driver may return I/O errors for many reasons, including bugs in the
driver. Or this may be an dumb user who screwed up his flash. It might
be something else. We do not wand to mark the eraseblock bad when it is
_not_ bad.

And it's not just erase successfully. We torture the eraseblock. We
erase it, read it back, check it has all 0xFFs, then write 0xA5 pattern,
read it back, check, erase again, read back, check, write 0x5A pattern,
read back, check, erase, read back, check, write 0x00 pattern, read
back, check, erase, read back check. Enough? If it is not enough, feel
free to send a patch for better torturing.

Please, refer the torture_peb() function for more details.

So, we do not mark eraseblock as bad if it passed the torture test.
Note, a bit-flip during torture test is enough to mark the eraseblock
bad as well.

In your particular case, you would have _all_ eraseblocks marked as bad
if UBI did what you want.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

Hi,

On Wed, 2008-04-23 at 10:13 +0200, Matthieu CASTET wrote:
> > [    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> > [    0.960000] Scanning device for bad blocks
> > [    1.000000] Bad eraseblock 494 at 0x03dc0000
> > [    1.050000] Bad eraseblock 1300 at 0x0a280000
> > [    1.140000] Bad eraseblock 2554 at 0x13f40000
> > [    1.160000] Bad eraseblock 2923 at 0x16d60000
> > [    1.200000] Bad eraseblock 3349 at 0x1a2a0000
> > [    1.230000] Bad eraseblock 3790 at 0x1d9c0000
> > [    6.890000] UBI: attached mtd9 to ubi0
> > 
> > 
> > 
> > Hamish
> 
> Do you know when the bad block scanning finish and the ubi scan start ?

Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver
for scanning against bad eraseblocks to build in-memory bad block table
(BBT). And it is probably more than 1.23 sec. If you start using
on-flash bad block table, this should go away. I never used on-flash
BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Nancy]

On 4/23/08, Artem Bityutskiy <dedekind@infradead.org> wrote:
> On Wed, 2008-04-23 at 16:01 +0800, Nancy wrote:
> >     Yes, the issue is go away by properly use MLC nand.
> >     But the write fail issue in UBI still exist! Suppose you are
> > properly used your Nand, when it return write fail, but can erase
> > successfully, UBI still group this block in good block heap. That is
> > wrong and dangerous.
>
> The driver may return I/O errors for many reasons, including bugs in the
> driver. Or this may be an dumb user who screwed up his flash. It might
> be something else. We do not wand to mark the eraseblock bad when it is
> _not_ bad.
>
> And it's not just erase successfully. We torture the eraseblock. We
> erase it, read it back, check it has all 0xFFs, then write 0xA5 pattern,
> read it back, check, erase again, read back, check, write 0x5A pattern,
> read back, check, erase, read back, check, write 0x00 pattern, read
> back, check, erase, read back check. Enough? If it is not enough, feel
> free to send a patch for better torturing.
>
> Please, refer the torture_peb() function for more details.
>
> So, we do not mark eraseblock as bad if it passed the torture test.
> Note, a bit-flip during torture test is enough to mark the eraseblock
> bad as well.
>
> In your particular case, you would have _all_ eraseblocks marked as bad
> if UBI did what you want.

Torure is enough : )
Will you please explain why pattern 0x00 is not enough, it still need
0xA5 and 0x5A. What these information come from? Thanks in advance.



-- 
Best wishes,
Nancy

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 17:07 +0800, Nancy wrote:
> Torure is enough : )
> Will you please explain why pattern 0x00 is not enough, it still need
> 0xA5 and 0x5A. What these information come from? Thanks in advance.

Dunno, i just decided it might be good. And I just picked these values.
If we fail  on the first pattern, we do not continue.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Matthieu CASTET]

Artem Bityutskiy wrote:
> Hi,
>>>
>>> Hamish
>> Do you know when the bad block scanning finish and the ubi scan start ?
> 
> Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver
> for scanning against bad eraseblocks to build in-memory bad block table
> (BBT). And it is probably more than 1.23 sec. If you start using
> on-flash bad block table, this should go away. I never used on-flash
> BBT, but I know MTD supports this and for example OLPC has on-flash BBT.
> 
May be UBI should print a message at startup.


Matthieu

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 11:21 +0200, Matthieu CASTET wrote:
> May be UBI should print a message at startup.

Yes, it will, the change is already in my tree and should go to 2.6.26.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Nancy]

On 4/23/08, Artem Bityutskiy <dedekind@infradead.org> wrote:
> On Wed, 2008-04-23 at 17:07 +0800, Nancy wrote:
> > Torure is enough : )
> > Will you please explain why pattern 0x00 is not enough, it still need
> > 0xA5 and 0x5A. What these information come from? Thanks in advance.
>
> Dunno, i just decided it might be good. And I just picked these values.
> If we fail  on the first pattern, we do not continue.
Okay.
Another question:
      Will you please explain why need leb_read_lock()?
      If I ummap a leb(only operate the eba table, do not do
ubi_wl_put_peb() ), then, what will that PEB belong to? wl->free,
wl->used, wl->prot.pnum, wl->prot.aec, wl->scrub? What will happen to
that PEB?


--
Best wishes,
Nancy

Re: [RFC] slight UBI scan time improvement

[Nancy]

Does volume size must be aligned to LEB size?
If I pass a unaligned LEB size to ubimkvol -s, then it will
automatically make it aligned or what?

-- 
Best wishes,
Nancy

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 18:51 +0800, Nancy wrote:
>       Will you please explain why need leb_read_lock()?

This locks the LEB and prevent concurrent I/O to this LEB. E.g., we do
not want a situation when one process unmaps an LEB while another
process is writing to it.

That was probably overkill, but I implemented per-LEB r/w locking. This
is similar to rw semaphore in Linux. It means that several processes may
simultaneously read for this LEB, but only one process at a time may
write (or unmap).

>       If I ummap a leb(only operate the eba table, do not do
> ubi_wl_put_peb() ), then, what will that PEB belong to? wl->free,
> wl->used, wl->prot.pnum, wl->prot.aec, wl->scrub? What will happen to
> that PEB?

If you unmap it you have to clean the corresponding EBA table record and
return it to the WL unit.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 18:57 +0800, Nancy wrote:
> Does volume size must be aligned to LEB size?
Volume size is always multiple of LEB size.

> If I pass a unaligned LEB size to ubimkvol -s, then it will
> automatically make it aligned or what?
Try and find out please.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Wed, Apr 23, 2008 at 11:21:04AM +0300, Artem Bityutskiy wrote:
> Hi,
> 
> On Wed, 2008-04-23 at 10:13 +0200, Matthieu CASTET wrote:
> > > [    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> > > [    0.960000] Scanning device for bad blocks
> > > [    1.000000] Bad eraseblock 494 at 0x03dc0000
> > > [    1.050000] Bad eraseblock 1300 at 0x0a280000
> > > [    1.140000] Bad eraseblock 2554 at 0x13f40000
> > > [    1.160000] Bad eraseblock 2923 at 0x16d60000
> > > [    1.200000] Bad eraseblock 3349 at 0x1a2a0000
> > > [    1.230000] Bad eraseblock 3790 at 0x1d9c0000
> > > [    6.890000] UBI: attached mtd9 to ubi0
> > > 
> > > 
> > > 
> > > Hamish
> > 
> > Do you know when the bad block scanning finish and the ubi scan start ?
> 
> Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver
> for scanning against bad eraseblocks to build in-memory bad block table
> (BBT). And it is probably more than 1.23 sec. If you start using
> on-flash bad block table, this should go away. I never used on-flash
> BBT, but I know MTD supports this and for example OLPC has on-flash BBT.

Well I think from past use of "time ubiattach ..." that most of
the missing time is in the attach. 

I've added a message at start of ubi_attach_mtd_dev(), but I can't test it 
from home as I left my test hardware at the office switched off tonight..
I'll confirm tomorrow.

Would combining the bad block scan with the UBI scan save time? I guess
it would be a bad layering violation however :-)


What sort of speed do you get using
dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64
(where mtdblock9 is your raw mtd NAND device). I'm seeing about 6
seconds to read that 8Mb, which is quite long I guess.

Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 22:40 +1000, Hamish Moffatt wrote:
> Well I think from past use of "time ubiattach ..." that most of
> the missing time is in the attach. 
Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec,
you may try to play with on-flash BBT.

> Would combining the bad block scan with the UBI scan save time? I guess
> it would be a bad layering violation however :-)
Yeah, it could be done as custom hack but not for mainstream.

> What sort of speed do you get using
> dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64
> (where mtdblock9 is your raw mtd NAND device). I'm seeing about 6
> seconds to read that 8Mb, which is quite long I guess.

I have busybox so stuff like 128K does not work. Here are my results:

# time dd if=/dev/mtd4 of=/dev/null bs=131072 count=64
64+0 records in
64+0 records out
real    0m 0.28s
user    0m 0.00s
sys     0m 0.28s

# time dd if=/dev/mtd4 of=/dev/null bs=131072 count=256
256+0 records in
256+0 records out
real    0m 1.12s
user    0m 0.00s
sys     0m 1.11s

Its OneNAND which has a controller and it is quite quick.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Wed, Apr 23, 2008 at 03:57:47PM +0300, Artem Bityutskiy wrote:
> On Wed, 2008-04-23 at 22:40 +1000, Hamish Moffatt wrote:
> > Well I think from past use of "time ubiattach ..." that most of
> > the missing time is in the attach. 
> Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec,
> you may try to play with on-flash BBT.

I'm not sure what this means.. ? Instead of having to scan each block
to check the marker, it has a central table? And that table is created
once by an initial scan and then added to when UBI declares a block bad?
How do I access this feature?

> > What sort of speed do you get using
> > dd if=/dev/mtdblock9 of=/tmp/foo bs=128K count=64
> > (where mtdblock9 is your raw mtd NAND device). I'm seeing about 6
> > seconds to read that 8Mb, which is quite long I guess.
> I have busybox so stuff like 128K does not work. Here are my results:

Hmm I have busybox (1.9.x) also, 128K works ok for me.

> # time dd if=/dev/mtd4 of=/dev/null bs=131072 count=64
> 64+0 records in
> 64+0 records out
> real    0m 0.28s
> user    0m 0.00s
> sys     0m 0.28s

Very good. It's about 6.1 seconds for me :-(

(Worse on my previous hardware which had compact flash on IDE without a
hardware IDE controller: over 11 seconds.)


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Wed, 2008-04-23 at 23:42 +1000, Hamish Moffatt wrote:
> On Wed, Apr 23, 2008 at 03:57:47PM +0300, Artem Bityutskiy wrote:
> > On Wed, 2008-04-23 at 22:40 +1000, Hamish Moffatt wrote:
> > > Well I think from past use of "time ubiattach ..." that most of
> > > the missing time is in the attach. 
> > Sure, UBI takes most of the time. Its just if you want to save 1.2+ sec,
> > you may try to play with on-flash BBT.
> 
> I'm not sure what this means.. ? Instead of having to scan each block
> to check the marker, it has a central table? And that table is created
> once by an initial scan and then added to when UBI declares a block bad?
> How do I access this feature?
Yes.

When your NAND driver is initialized, it scans whole flash and builds
the BBT. Basically, it reads OOB area of each first (and may be second)
page of each eraseblock. The BBT is simply an array with 1 bit per
eraseblock.

You may make it save the BBT on the flash media. So next time you boot,
it reads the BBT from a pre-defined place (e.g., the last eraseblock)
and that's it. It does not scan and does not waste time.

This all is happening in the driver, before UBI starts.

Yes, if UBI marks a block bad, which basically means it calls an MTD
function, the NAND infrastructure changes the on-flash BBT.

But I never used this myself, so my knowledge is theoretical. I just
know others use this and I saw the code in nand_base.c.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Wed, Apr 23, 2008 at 10:40:46PM +1000, Hamish Moffatt wrote:
> On Wed, Apr 23, 2008 at 11:21:04AM +0300, Artem Bityutskiy wrote:
> > Hi,
> > 
> > On Wed, 2008-04-23 at 10:13 +0200, Matthieu CASTET wrote:
> > > > [    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> > > > [    0.960000] Scanning device for bad blocks
> > > > [    1.000000] Bad eraseblock 494 at 0x03dc0000
> > > > [    1.050000] Bad eraseblock 1300 at 0x0a280000
> > > > [    1.140000] Bad eraseblock 2554 at 0x13f40000
> > > > [    1.160000] Bad eraseblock 2923 at 0x16d60000
> > > > [    1.200000] Bad eraseblock 3349 at 0x1a2a0000
> > > > [    1.230000] Bad eraseblock 3790 at 0x1d9c0000
> > > > [    6.890000] UBI: attached mtd9 to ubi0
> > > > 
> > > > 
> > > > 
> > > > Hamish
> > > 
> > > Do you know when the bad block scanning finish and the ubi scan start ?
> > 
> > Good point Matthieu. Indeed, _at least_ 1.23 sec is spend in the driver
> > for scanning against bad eraseblocks to build in-memory bad block table
> > (BBT). And it is probably more than 1.23 sec. If you start using
> > on-flash bad block table, this should go away. I never used on-flash
> > BBT, but I know MTD supports this and for example OLPC has on-flash BBT.
> 
> Well I think from past use of "time ubiattach ..." that most of
> the missing time is in the attach. 

Here's the evidence:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    1.250000] UBI: attaching mtd9 to ubi-1
[    6.890000] UBI: attached mtd9 to ubi0
[    6.900000] UBI: MTD device name:            "gen_nand.0"
[    6.900000] UBI: MTD device size:            512 MiB
[    6.910000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[    6.910000] UBI: logical eraseblock size:    129024 bytes
[    6.920000] UBI: number of good PEBs:        4090
[    6.920000] UBI: number of bad PEBs:         6
[    6.930000] UBI: smallest flash I/O unit:    2048
[    6.930000] UBI: VID header offset:          512 (aligned 512)
[    6.940000] UBI: data offset:                2048
[    6.940000] UBI: max. allowed volumes:       128
[    6.950000] UBI: wear-leveling threshold:    4096
[    6.950000] UBI: number of internal volumes: 1
[    6.960000] UBI: number of user volumes:     4
[    6.960000] UBI: available PEBs:             0
[    6.970000] UBI: total number of reserved PEBs: 4090
[    6.970000] UBI: number of PEBs reserved for bad PEB handling: 40
[    6.980000] UBI: max/mean erase counter: 29/1
[    6.980000] UBI: background thread "ubi_bgt0d" started, PID 641


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Wed, Apr 23, 2008 at 05:09:19PM +0300, Artem Bityutskiy wrote:
> You may make it save the BBT on the flash media. So next time you boot,
> it reads the BBT from a pre-defined place (e.g., the last eraseblock)
> and that's it. It does not scan and does not waste time.

Thanks for the info. It looks like this would not save me very much
time so I don't think I will bother.

[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    1.250000] UBI: attaching mtd9 to ubi-1
[    6.890000] UBI: attached mtd9 to ubi0

I notice in your patch that you read a whole min_io_size block, even
though you only need the EC and VID headers (total of 128 bytes each, or
576 bytes as a single read according to my calculation):

+	/*
+	 * During scanning we read EC and VID headers at one go in to the
+	 * @si->read_buf, and then check EC and VID header. This must be faster
+	 * than doing 2 small read operations.
+	 */
+	si->read_len = ubi->vid_hdr_offset + UBI_VID_HDR_SIZE;
+	si->read_len = ALIGN(si->read_len, ubi->min_io_size);
+	si->read_buf = kmalloc(si->read_len, GFP_KERNEL);
+	if (!si->read_buf) {
+		err = -ENOMEM;
 		goto out_si;
+	}
 
Won't reading 2K bytes be slower than 576 in some cases? If you have
soft ECC then you have to read the whole page anyway, but if you have
hardware ECC then you have no need to read the whole page into RAM.

Hmm. The software ECC seems to work internally on 256 byte blocks.
However it appears that nand_base will always read in a whole page (2K
on my flash). It should be ok to read only a 256-byte block as that's
all you need for ECC calculation? Not a whole 2K which requires 8 ECC
calculations.


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [RFC] slight UBI scan time improvement

[Artem Bityutskiy]

On Thu, 2008-04-24 at 11:53 +1000, Hamish Moffatt wrote:
> Thanks for the info. It looks like this would not save me very much
> time so I don't think I will bother.

Ok.

> I notice in your patch that you read a whole min_io_size block, even
> though you only need the EC and VID headers (total of 128 bytes each, or
> 576 bytes as a single read according to my calculation):

Err, 64 bytes each even.
 
> Won't reading 2K bytes be slower than 576 in some cases? If you have
> soft ECC then you have to read the whole page anyway, but if you have
> hardware ECC then you have no need to read the whole page into RAM.

Yes, I guess. Current MTD implementation reads whole page in any case,
though.

> Hmm. The software ECC seems to work internally on 256 byte blocks.
> However it appears that nand_base will always read in a whole page (2K
> on my flash). It should be ok to read only a 256-byte block as that's
> all you need for ECC calculation? Not a whole 2K which requires 8 ECC
> calculations.

However, there was a patch from Alexey which may certainly help you. It
was not looked at properly, unfortunately. I'll try to find it in my
mailbox and will send to you.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [RFC] slight UBI scan time improvement

[Hamish Moffatt]

On Thu, Apr 24, 2008 at 09:21:20AM +0300, Artem Bityutskiy wrote:
> On Thu, 2008-04-24 at 11:53 +1000, Hamish Moffatt wrote:
> > Hmm. The software ECC seems to work internally on 256 byte blocks.
> > However it appears that nand_base will always read in a whole page (2K
> > on my flash). It should be ok to read only a 256-byte block as that's
> > all you need for ECC calculation? Not a whole 2K which requires 8 ECC
> > calculations.
> 
> However, there was a patch from Alexey which may certainly help you. It
> was not looked at properly, unfortunately. I'll try to find it in my
> mailbox and will send to you.

Thanks. Yes I found the patch in my linux-mtd folder too. It's very
nice!

Here's my attach time before:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.960000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.050000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.160000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    6.900000] UBI: attached mtd9 to ubi0

... and now:

[    0.950000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.970000] Scanning device for bad blocks
[    1.000000] Bad eraseblock 494 at 0x03dc0000
[    1.060000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.170000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    3.880000] UBI: attached mtd9 to ubi0


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>