ubi and u-boot

ubi and u-boot

[Bernard Blackham]

[apologies for the cross post]

Is anyone working on ubi/ubifs support in u-boot?

Kind regards,
Bernard.

Re: [U-Boot-Users] ubi and u-boot

[Josh Boyer]

On Fri, 2008-04-18 at 22:21 +0800, Bernard Blackham wrote:
> [apologies for the cross post]
> 
> Is anyone working on ubi/ubifs support in u-boot?

I am not aware of anyone actively working on porting UBI to upstream
U-Boot at this exact moment.

josh

Re: [U-Boot-Users] ubi and u-boot

[Jamie Lokier]

Josh Boyer wrote:
> On Fri, 2008-04-18 at 22:21 +0800, Bernard Blackham wrote:
> > [apologies for the cross post]
> > Is anyone working on ubi/ubifs support in u-boot?
> 
> I am not aware of anyone actively working on porting UBI to upstream
> U-Boot at this exact moment.

Is it even a good idea?  The UBI (version 1 :-) initial scan is not
fast for large flash, and if the bootloader does it too, that's twice
as much time to boot.

However, if there was a protocol for bootloader to pass the scan
results to the booted kernel, that would be very nice.

-- Jamie

Re: [U-Boot-Users] ubi and u-boot

[Bernard Blackham]

Jamie Lokier wrote:
> Josh Boyer wrote:
>>> Is anyone working on ubi/ubifs support in u-boot?
>> I am not aware of anyone actively working on porting UBI to upstream
>> U-Boot at this exact moment.
> 
> Is it even a good idea?

 From what I've seen, UBI is the first solution with the potential for 
reliably booting from NAND flash - specifically, dealing with 
read-disturbance from adjacent blocks causing bit errors in critical 
boot blocks. This could save useful things like your kernel image (where 
it's rarely written, but it and blocks around it are read often enough 
to cause an annoying bitflip or two). The rewriting can be done once 
userspace boots, but having it in UBI to begin with makes this easier.

(Unless there's a much easier way to deal with read-disturbance that 
I've missed, in which case please lart me).

Wolfgang showed some interest briefly too.[1]

 > The UBI (version 1 :-) initial scan is not
> fast for large flash, and if the bootloader does it too, that's twice
> as much time to boot.
 >
> However, if there was a protocol for bootloader to pass the scan
> results to the booted kernel, that would be very nice.

If there's no compelling reasons not to, I'll try and find some time to 
work on ubi in uboot. I'm not expecting it to be easy though :)

Cheers,
Bernard.

[1] http://article.gmane.org/gmane.comp.boot-loaders.u-boot/32602

Re: [U-Boot-Users] ubi and u-boot

[Josh Boyer]

On Fri, 2008-04-18 at 17:59 +0100, Jamie Lokier wrote:
> Josh Boyer wrote:
> > On Fri, 2008-04-18 at 22:21 +0800, Bernard Blackham wrote:
> > > [apologies for the cross post]
> > > Is anyone working on ubi/ubifs support in u-boot?
> > 
> > I am not aware of anyone actively working on porting UBI to upstream
> > U-Boot at this exact moment.
> 
> Is it even a good idea?  The UBI (version 1 :-) initial scan is not
> fast for large flash, and if the bootloader does it too, that's twice
> as much time to boot.

It's a good idea, yes.  Particularly if you want to boot from NAND
flash.

Can you define "large device"?  It only scans the first 1 or 2 pages for
each eraseblock to build up the volume tables.  That really isn't that
slow...

> However, if there was a protocol for bootloader to pass the scan
> results to the booted kernel, that would be very nice.

Maybe.  I was never fully convinced of that.

josh

Re: ubi and u-boot

[Artem Bityutskiy]

On Fri, 2008-04-18 at 22:21 +0800, Bernard Blackham wrote:
> [apologies for the cross post]
> 
> Is anyone working on ubi/ubifs support in u-boot?

Someone mailed me some question and he said he is trying to make u-boot
UBI-aware, but that guy has not showed up anymore.

Yeah, it would be nice to have u-boot understand UBI. At leaset, it
would be nice to teach it to preserve erase counters when erasing flash.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [U-Boot-Users] ubi and u-boot

[Jamie Lokier]

Josh Boyer wrote:
> > Is it even a good idea?  The UBI (version 1 :-) initial scan is not
> > fast for large flash, and if the bootloader does it too, that's twice
> > as much time to boot.
> 
> It's a good idea, yes.  Particularly if you want to boot from NAND
> flash.
> 
> Can you define "large device"?  It only scans the first 1 or 2 pages for
> each eraseblock to build up the volume tables.  That really isn't that
> slow...

I was thinking this:

Hamish Moffatt wrote (Message-ID: <20080407073227.GA6317@cloud.net.au>):
> Sorry I should've said 512MiB perhaps: 512 megabytes.
> UBI attach time appears to be about 6 seconds.

If 6 seconds is as fast as it can be done, annoying but fair enough.

Adding _another_ 6 seconds to the boot time seems a lot to me.

The only ways I see to improve the speed in general are:

   1. Partition the NAND into multiple independent UBIs, so the boot
      loader doesn't have the scan the whole chip, but that is
      obviously not so good for wear levelling.  (It's probably
      alright, though, if it's like the /boot partition on a standard
      Linux distro - not updated very often.)

   2. Change UBI's data structure so that the number of pages it needs
      to read is a sub-linear function of the number of erase blocks.
      I think this is what's meant by 'UBI 2'.

To remove the double scan:

> > However, if there was a protocol for bootloader to pass the scan
> > results to the booted kernel, that would be very nice.
> 
> Maybe.  I was never fully convinced of that.

I can understand the hesitation, but I think 6 seconds just to find
the kernel - especially when doing a 'disk resume' - is quite a lot.

Note that I haven't tried UBI myself yet.  I'm going on what has been
written to the list so far, as quoted above.

-- Jamie

Re: [U-Boot-Users] ubi and u-boot

[Josh Boyer]

On Sun, 2008-04-20 at 17:04 +0100, Jamie Lokier wrote:
> Josh Boyer wrote:
> > > Is it even a good idea?  The UBI (version 1 :-) initial scan is not
> > > fast for large flash, and if the bootloader does it too, that's twice
> > > as much time to boot.
> > 
> > It's a good idea, yes.  Particularly if you want to boot from NAND
> > flash.
> > 
> > Can you define "large device"?  It only scans the first 1 or 2 pages for
> > each eraseblock to build up the volume tables.  That really isn't that
> > slow...
> 
> I was thinking this:
> 
> Hamish Moffatt wrote (Message-ID: <20080407073227.GA6317@cloud.net.au>):
> > Sorry I should've said 512MiB perhaps: 512 megabytes.
> > UBI attach time appears to be about 6 seconds.
> 
> If 6 seconds is as fast as it can be done, annoying but fair enough.

You should read that thread a bit more carefully.  The scan time is
highly dependent upon the NAND driver beneath UBI.  For example, a UBI
scan/attach on a 1GiB device on OLPC was 2 seconds.

> 
> Adding _another_ 6 seconds to the boot time seems a lot to me.

You mean adding another "X amount of time depending on factors outside
of UBI's control."

> The only ways I see to improve the speed in general are:
> 
>    1. Partition the NAND into multiple independent UBIs, so the boot
>       loader doesn't have the scan the whole chip, but that is
>       obviously not so good for wear levelling.  (It's probably
>       alright, though, if it's like the /boot partition on a standard
>       Linux distro - not updated very often.)
> 
>    2. Change UBI's data structure so that the number of pages it needs
>       to read is a sub-linear function of the number of erase blocks.
>       I think this is what's meant by 'UBI 2'.
> 
> To remove the double scan:
> 
> > > However, if there was a protocol for bootloader to pass the scan
> > > results to the booted kernel, that would be very nice.
> > 
> > Maybe.  I was never fully convinced of that.
> 
> I can understand the hesitation, but I think 6 seconds just to find
> the kernel - especially when doing a 'disk resume' - is quite a lot.

You should really stop quoting this 6 second number.

Anyway, passing the scan results from a bootloader becomes very
involved.  You have to store it somewhere, probably preserving a section
of DRAM, which isn't always easy.

> Note that I haven't tried UBI myself yet.  I'm going on what has been
> written to the list so far, as quoted above.

Maybe you should try it :).

josh

Re: [U-Boot-Users] ubi and u-boot

[Jamie Lokier]

Josh Boyer wrote:
> > Hamish Moffatt wrote (Message-ID: <20080407073227.GA6317@cloud.net.au>):
> > > Sorry I should've said 512MiB perhaps: 512 megabytes.
> > > UBI attach time appears to be about 6 seconds.
> > 
> > If 6 seconds is as fast as it can be done, annoying but fair enough.
> 
> You should read that thread a bit more carefully.  The scan time is
> highly dependent upon the NAND driver beneath UBI.  For example, a UBI
> scan/attach on a 1GiB device on OLPC was 2 seconds.

Ah, I intended to quote the 2 seconds too but forgot, sorry.  I think
2 seconds per gigabyte is a significant time, too, but not so much.

The followup suggested it was due to the speed of the chip, not so
much the driver.

> > Adding _another_ 6 seconds to the boot time seems a lot to me.
> 
> You mean adding another "X amount of time depending on factors outside
> of UBI's control."

Well, yes, that would be a reason to consider whether doing it is a
good idea :-)

> > I can understand the hesitation, but I think 6 seconds just to find
> > the kernel - especially when doing a 'disk resume' - is quite a lot.
> 
> You should really stop quoting this 6 second number.

Let's call it 2 seconds per gigabyte, then.

> > Note that I haven't tried UBI myself yet.  I'm going on what has been
> > written to the list so far, as quoted above.
> 
> Maybe you should try it :).

I will when good looking figures are being quoted on the list ;-)

-- Jamie

Re: [U-Boot-Users] ubi and u-boot

[Wolfgang Denk]

In message <4808DF3B.2080702@largestprime.net> you wrote:
> 
> Wolfgang showed some interest briefly too.[1]

I am definitely interested.

>  > The UBI (version 1 :-) initial scan is not
> > fast for large flash, and if the bootloader does it too, that's twice
> > as much time to boot.

We have the same (at at least similar) problems when using JFFS2 file
systems in U-Boot. Yet, being able to do this is quite useful in many
situations.

> > However, if there was a protocol for bootloader to pass the scan
> > results to the booted kernel, that would be very nice.

That would be of course a good thing to have :-)

> If there's no compelling reasons not to, I'll try and find some time to 
> work on ubi in uboot. I'm not expecting it to be easy though :)

Thanks in advance, and good luck!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
All these theories, diverse as they are, have two things  in  common:
they explain the observed facts, and they are completeley and utterly
wrong.                       - Terry Pratchett, _The Light Fantastic_

Re: [U-Boot-Users] ubi and u-boot

[Hamish Moffatt]

On Sun, Apr 20, 2008 at 05:04:14PM +0100, Jamie Lokier wrote:
> I was thinking this:
> 
> Hamish Moffatt wrote (Message-ID: <20080407073227.GA6317@cloud.net.au>):
> > Sorry I should've said 512MiB perhaps: 512 megabytes.
> > UBI attach time appears to be about 6 seconds.
> 
> If 6 seconds is as fast as it can be done, annoying but fair enough.
> 
> Adding _another_ 6 seconds to the boot time seems a lot to me.

Yes. I have a prototype system here booting a minimal system from NOR
flash then booting the full system from NAND via kexec. I'm suffering
my 6 seconds of UBI attach time twice each boot.

Six seconds does seem to be abnormal though - I don't have a hardware
NAND controller so it's all software ECC, GPIO for chip selects etc. The
performance is said to be much better with a hardware controller.

> To remove the double scan:
> 
> > > However, if there was a protocol for bootloader to pass the scan
> > > results to the booted kernel, that would be very nice.

That would certainly help my case. Fortunately my device is usually
always-on and therefore boot time is not critical, and we don't
suspend/resume.


Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: [U-Boot-Users] ubi and u-boot

[Artem Bityutskiy]

On Mon, 2008-04-21 at 00:22 +0200, Wolfgang Denk wrote:
> In message <4808DF3B.2080702@largestprime.net> you wrote:
> > 
> > Wolfgang showed some interest briefly too.[1]
> 
> I am definitely interested.

What I would suggest to start with is to teach U-boot to erase flash and
preserve erase counters, because now it just wipes the erase counters
out. Also, it would be nice to teach it to flash images properly in case
of NAND flash.

I've just written a documentation section which I hope would help to do
this. Please, see
http://www.linux-mtd.infradead.org/doc/ubi.html#L_format

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [U-Boot-Users] ubi and u-boot

[Ricard Wanderlof]

On Mon, 21 Apr 2008, Artem Bityutskiy wrote:

> On Mon, 2008-04-21 at 00:22 +0200, Wolfgang Denk wrote:
>> In message <4808DF3B.2080702@largestprime.net> you wrote:
>>>
>>> Wolfgang showed some interest briefly too.[1]
>>
>> I am definitely interested.
>
> What I would suggest to start with is to teach U-boot to erase flash and
> preserve erase counters, because now it just wipes the erase counters
> out. Also, it would be nice to teach it to flash images properly in case
> of NAND flash.
>

I was under the impression from past emails on the MTD list that the 
internal meta format for UBI has changed a few times over its lifetime.

Assuming this continues, I would think this would cause problems for 
bootloaders such as U-Boot whose flash partitions are not necessarily 
rewritten in the case of a product upgrade - i.e. a firmware upgrade with 
a new kernel would have a new UBI implementation and spec causing an
older bootloader to fail to operate properly with the newer image.

/Ricard
--
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: [U-Boot-Users] ubi and u-boot

[Josh Boyer]

On Mon, 2008-04-21 at 15:36 +0200, Ricard Wanderlof wrote:
> On Mon, 21 Apr 2008, Artem Bityutskiy wrote:
> 
> > On Mon, 2008-04-21 at 00:22 +0200, Wolfgang Denk wrote:
> >> In message <4808DF3B.2080702@largestprime.net> you wrote:
> >>>
> >>> Wolfgang showed some interest briefly too.[1]
> >>
> >> I am definitely interested.
> >
> > What I would suggest to start with is to teach U-boot to erase flash and
> > preserve erase counters, because now it just wipes the erase counters
> > out. Also, it would be nice to teach it to flash images properly in case
> > of NAND flash.
> >
> 
> I was under the impression from past emails on the MTD list that the 
> internal meta format for UBI has changed a few times over its lifetime.
> 
> Assuming this continues, I would think this would cause problems for 
> bootloaders such as U-Boot whose flash partitions are not necessarily 
> rewritten in the case of a product upgrade - i.e. a firmware upgrade with 
> a new kernel would have a new UBI implementation and spec causing an
> older bootloader to fail to operate properly with the newer image.

I think there was only 1 time the on-flash format changed, and that was
before UBI was merged upstream.  New features can be introduced with
additional flags, however older versions should be able to ignore those.

Artem, is that right?  My memory is fuzzy here.

josh

Re: [U-Boot-Users] ubi and u-boot

[Artem Bityutskiy]

On Mon, 2008-04-21 at 08:44 -0500, Josh Boyer wrote:
> I think there was only 1 time the on-flash format changed, and that was
> before UBI was merged upstream.  New features can be introduced with
> additional flags, however older versions should be able to ignore those.
> 
> Artem, is that right?  My memory is fuzzy here.

No, this is not right. We never changed the media format since UBI was
merged.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [U-Boot-Users] ubi and u-boot

[Artem Bityutskiy]

On Mon, 2008-04-21 at 16:50 +0300, Artem Bityutskiy wrote:
> On Mon, 2008-04-21 at 08:44 -0500, Josh Boyer wrote:
> > I think there was only 1 time the on-flash format changed, and that was
> > before UBI was merged upstream.  New features can be introduced with
> > additional flags, however older versions should be able to ignore those.
> > 
> > Artem, is that right?  My memory is fuzzy here.
> 
> No, this is not right. We never changed the media format since UBI was
> merged.

Err, I meant what you said is right, but it is not right that we changed
media format.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [U-Boot-Users] ubi and u-boot

[Ricard Wanderlof]

On Mon, 21 Apr 2008, Artem Bityutskiy wrote:

> On Mon, 2008-04-21 at 08:44 -0500, Josh Boyer wrote:
>> I think there was only 1 time the on-flash format changed, and that was
>> before UBI was merged upstream.  New features can be introduced with
>> additional flags, however older versions should be able to ignore those.
>>
>> Artem, is that right?  My memory is fuzzy here.
>
> [...] We never changed the media format since UBI was merged.

Ok, that sounds reassuring. I think I've been confusing UBI with UBIFS in 
this respect.

/Ricard
--
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: [U-Boot-Users] ubi and u-boot

[Jamie Lokier]

Josh Boyer wrote:
> > I was under the impression from past emails on the MTD list that the 
> > internal meta format for UBI has changed a few times over its lifetime.
> > 
> > Assuming this continues, I would think this would cause problems for 
> > bootloaders such as U-Boot whose flash partitions are not necessarily 
> > rewritten in the case of a product upgrade - i.e. a firmware upgrade with 
> > a new kernel would have a new UBI implementation and spec causing an
> > older bootloader to fail to operate properly with the newer image.
> 
> I think there was only 1 time the on-flash format changed, and that was
> before UBI was merged upstream.  New features can be introduced with
> additional flags, however older versions should be able to ignore those.

Quite.  As long as the format is changed in an upward compatible way,
it should be fine.  Hopefully the existing format is designed to
ensure that's always possible - at least for everything called 'UBI 1'
since the upstream merge.

-- Jamie