U-Boot and UBI

U-Boot and UBI

[Josh Boyer]

Hi All,

As has been stated before by various people, IBM has added some form of
UBI support to an older version of U-Boot.  While we would love to be
able to forward port this to the latest version of U-Boot, we have
simply not found the time and don't see that being the case in the near
future.

So as an alternative, I have cleaned up the code slightly and pushed out
a version that applies to U-Boot 1.1.6.  This way there is at least some
form of code release that people can use as a reference.  I am not
pushing this for inclusion into the current U-Boot, as we are well aware
of the need for further adaptation and cleanup.

A few caveats:

- This is a quick forward port from a patched 1.1.4 tree to 1.1.6.  It
compiles for the example board (PowerPC 440EP Bamboo), but that example
code may not actually work due to the differences in how flash was read
between the code bases.  When moving to a newer U-Boot, I highly
recommend using the generic MTD interfaces.

- There might be slight differences in the UBI format that this code
base supports and what is in the upstream Linux kernel.  Comparison of
the data structures in the current kernel would be advised if you are
forward porting.

- There is no write support for UBI in this port.

- The interfaces have two methods of getting scan information.  The
first is having a bit of code that runs before U-Boot do the scan and
pass it.  The second is to have a user/script use the "ubiscan" command
to have U-Boot manually perform the scan itself.  The latter method is
likely more flexible for new boards.

All of that being said, we hope this code is useful as a guide.  I will
certainly answer questions as I can.  If for some unknown reason I find
a massive amount of spare time, I might even try to forward port it, but
that seems unlikely these days.

I would like to thank Frank Haverkamp, Thomas Gleixner, Andreas Arnez,
and Artem Bityutskiy for their work on UBI and the majority of the
U-Boot code we have.

The git tree can be found here:

git fetch git://git.infradead.org/users/jwboyer/u-boot.git ubi

http://git.infradead.org/users/jwboyer/u-boot.git?a=shortlog;h=refs/heads/ubi

josh

Re: [U-Boot] U-Boot and UBI

[Wolfgang Denk]

Dear Josh,

in message <1218220016.2328.109.camel@localhost.localdomain> you wrote:
> 
> As has been stated before by various people, IBM has added some form of
> UBI support to an older version of U-Boot.  While we would love to be
> able to forward port this to the latest version of U-Boot, we have
> simply not found the time and don't see that being the case in the near
> future.
> 
> So as an alternative, I have cleaned up the code slightly and pushed out
> a version that applies to U-Boot 1.1.6.  This way there is at least some
> form of code release that people can use as a reference.  I am not
> pushing this for inclusion into the current U-Boot, as we are well aware
> of the need for further adaptation and cleanup.
...
> The git tree can be found here:
> 
> git fetch git://git.infradead.org/users/jwboyer/u-boot.git ubi
> 
> http://git.infradead.org/users/jwboyer/u-boot.git?a=shortlog;h=refs/heads/ubi

Thanks a lot.

I guess we will pick up this work ASAP.

If anybody else has intentions to work in this area, I'd appreciate if
you contact me so we can coordinate efforts.

Thanks again.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Lead me not into temptation... I can find it myself.

Re: [U-Boot] U-Boot and UBI

[Kyungmin Park]

Dear Josh,

First thank you for announcing UBI support on U-boot.

Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

I saw your source codes and ubi commands. I think it supports minimal
features of UBI, no volume create, remove, read/write and so on like
these.

u-boot $ help ubi
ubi info [l[ayout]] - Display volume and ubi layout information
ubi create[vol] volume [size] [type] - create volume name with size
ubi write[vol] address volume size - Write volume from address with size
ubi read[vol] address volume [size] - Read volume to address with size
ubi remove[vol] volume - Remove volume
[Legends]
 volume: character name
 size: KiB, MiB, GiB, and bytes
 type: s[tatic] or d[ynamic] (default=dynamic)

I want to know what's the better way if there are two code bases.

Please give any options and comments

Thank you,
Kyungmin Park

P.S., we will post the patches ASAP


>> As has been stated before by various people, IBM has added some form of
>> UBI support to an older version of U-Boot.  While we would love to be
>> able to forward port this to the latest version of U-Boot, we have
>> simply not found the time and don't see that being the case in the near
>> future.
>>
>> So as an alternative, I have cleaned up the code slightly and pushed out
>> a version that applies to U-Boot 1.1.6.  This way there is at least some
>> form of code release that people can use as a reference.  I am not
>> pushing this for inclusion into the current U-Boot, as we are well aware
>> of the need for further adaptation and cleanup.
> ...

> Thanks a lot.
>
> I guess we will pick up this work ASAP.
>
> If anybody else has intentions to work in this area, I'd appreciate if
> you contact me so we can coordinate efforts.
>

Re: [U-Boot] U-Boot and UBI

[Wolfgang Denk]

Dear Kyungmin Park,

In message <9c9fda240808102218y6461fb6el192dcff7e47c2536@mail.gmail.com> you wrote:
> 
> Actually the Samsung implemented the UBI support on U-boot already and
> has used it internally. The big difference is the code base. It's

The big problem  is  always  that  such  ports  are  done  completely
internally, and nothing is given back to the community. This causes a
lot of duplicated (and thus avoidavle) efforts. Samsung is one of the
companies hich have a notoriously bad record on this :-(

> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
> it created the ubi wrapper for u-boot.

It would have been nice if such issued whould have been discussed
brfore implementing the stuff behind closed doors.

> I saw your source codes and ubi commands. I think it supports minimal
> features of UBI, no volume create, remove, read/write and so on like
> these.
> 
> u-boot $ help ubi
> ubi info [l[ayout]] - Display volume and ubi layout information
> ubi create[vol] volume [size] [type] - create volume name with size
> ubi write[vol] address volume size - Write volume from address with size
> ubi read[vol] address volume [size] - Read volume to address with size
> ubi remove[vol] volume - Remove volume
> [Legends]
>  volume: character name
>  size: KiB, MiB, GiB, and bytes
>  type: s[tatic] or d[ynamic] (default=dynamic)
> 
> I want to know what's the better way if there are two code bases.

The first thing that should be done is publishing the Samsung code
without further delay. Even if it is against an old code base.

> P.S., we will post the patches ASAP

Please do this *now*.


Thanks.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Hacking's just another word for nothing left to kludge.

Re: [U-Boot] U-Boot and UBI

[Josh Boyer]

On Mon, 11 Aug 2008 14:18:53 +0900
"Kyungmin Park" <kmpark@infradead.org> wrote:

> Dear Josh,
> 
> First thank you for announcing UBI support on U-boot.
> 
> Actually the Samsung implemented the UBI support on U-boot already and
> has used it internally. The big difference is the code base. It's
> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
> it created the ubi wrapper for u-boot.

Interesting.

> I saw your source codes and ubi commands. I think it supports minimal
> features of UBI, no volume create, remove, read/write and so on like
> these.

That is correct.  Simple scan and static volume read support are
present in the code base I pushed out.  That was all the project we
were doing really required in U-Boot at the time.


> u-boot $ help ubi
> ubi info [l[ayout]] - Display volume and ubi layout information
> ubi create[vol] volume [size] [type] - create volume name with size
> ubi write[vol] address volume size - Write volume from address with size
> ubi read[vol] address volume [size] - Read volume to address with size
> ubi remove[vol] volume - Remove volume
> [Legends]
>  volume: character name
>  size: KiB, MiB, GiB, and bytes
>  type: s[tatic] or d[ynamic] (default=dynamic)
> 
> I want to know what's the better way if there are two code bases.
> 
> Please give any options and comments

If your code base is more feature complete, it is certainly worth
looking at.  Particularly if it is for a newer version of U-Boot.

josh

Re: [U-Boot] U-Boot and UBI

[Stefan Roese]

On Monday 11 August 2008, Kyungmin Park wrote:
> Actually the Samsung implemented the UBI support on U-boot already and
> has used it internally. The big difference is the code base. It's
> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
> it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I assume 
that you have some size restrictions for the IPL/SPL on your platforms as 
well.

<snip>

> I want to know what's the better way if there are two code bases.
>
> Please give any options and comments

As Josh already mentioned, your version might be better suited as a basis for 
mainline inclusion because its more feature complete and probably is for a 
newer U-Boot version.

So best would be to publish your UBI support too, then we can compare and 
decide where to go.

> Thank you,
> Kyungmin Park
>
> P.S., we will post the patches ASAP

Do you have a rough idea when you could publish your patches?

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de
=====================================================================

Re: [U-Boot] U-Boot and UBI

[Kyungmin Park]

[-- Attachment #1: Type: text/plain, Size: 3849 bytes --]

Hi,

On Tue, Aug 12, 2008 at 5:02 AM, Stefan Roese <sr@denx.de> wrote:
> On Monday 11 August 2008, Kyungmin Park wrote:
>> Actually the Samsung implemented the UBI support on U-boot already and
>> has used it internally. The big difference is the code base. It's
>> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
>> it created the ubi wrapper for u-boot.
>
> And how does NAND/OneNAND booting with UBI support fit into this? I assume
> that you have some size restrictions for the IPL/SPL on your platforms as
> well.
>

It's not yet covered. it's TODO

Anyway, here's UBI patch.
The real implementation was done by another board but to publish I
make a patch for apollon board for usage. (tested and verified)

As see it's all-in-one patch and contains several parts, apollon
board, OneNAND, mtdpartition, and UBI.

After the feedback, I will generate right patches for purpose.

Please give any comments and feedbacks

Thank you,
Kyungmin Park

P.S., it's bzip2ed since mailing list has size limitation.

 Makefile                           |    1 +
 board/apollon/Makefile             |    2 +-
 board/apollon/config.mk            |    2 +-
 board/apollon/ubi.c                |   55 ++
 common/Makefile                    |    1 +
 common/cmd_jffs2.c                 |   69 ++-
 common/cmd_ubi.c                   |  532 ++++++++++++
 drivers/mtd/Makefile               |    1 +
 drivers/mtd/mtdcore.c              |  140 +++
 drivers/mtd/mtdpart.c              |  531 ++++++++++++
 drivers/mtd/nand/nand_base.c       |    2 +
 drivers/mtd/onenand/onenand_base.c |   98 ++-
 drivers/mtd/ubi/Makefile           |   48 +
 drivers/mtd/ubi/build.c            | 1182 +++++++++++++++++++++++++
 drivers/mtd/ubi/cdev.c             |  819 ++++++++++++++++++
 drivers/mtd/ubi/crc32.c            |  510 +++++++++++
 drivers/mtd/ubi/crc32defs.h        |   32 +
 drivers/mtd/ubi/crc32table.h       |  136 +++
 drivers/mtd/ubi/debug.c            |  192 +++++
 drivers/mtd/ubi/debug.h            |  152 ++++
 drivers/mtd/ubi/eba.c              | 1256 +++++++++++++++++++++++++++
 drivers/mtd/ubi/gluebi.c           |  347 ++++++++
 drivers/mtd/ubi/io.c               | 1274 +++++++++++++++++++++++++++
 drivers/mtd/ubi/kapi.c             |  638 ++++++++++++++
 drivers/mtd/ubi/misc.c             |  106 +++
 drivers/mtd/ubi/scan.c             | 1361 +++++++++++++++++++++++++++++
 drivers/mtd/ubi/scan.h             |  165 ++++
 drivers/mtd/ubi/ubi-media.h        |  372 ++++++++
 drivers/mtd/ubi/ubi.h              |  641 ++++++++++++++
 drivers/mtd/ubi/upd.c              |  441 ++++++++++
 drivers/mtd/ubi/vmt.c              |  862 +++++++++++++++++++
 drivers/mtd/ubi/vtbl.c             |  837 ++++++++++++++++++
 drivers/mtd/ubi/wl.c               | 1670 ++++++++++++++++++++++++++++++++++++
 fs/jffs2/jffs2_1pass.c             |  118 +++
 include/configs/apollon.h          |   21 +-
 include/jffs2/load_kernel.h        |   11 +-
 include/linux/crc32.h              |   27 +
 include/linux/mtd/compat.h         |    5 +
 include/linux/mtd/mtd-abi.h        |    1 +
 include/linux/mtd/mtd.h            |   27 +
 include/linux/mtd/nand.h           |   14 -
 include/linux/mtd/nand_legacy.h    |   12 -
 include/linux/mtd/onenand.h        |   19 +-
 include/linux/mtd/partitions.h     |   84 ++
 include/linux/mtd/ubi.h            |  186 ++++
 include/linux/rbtree.h             |  161 ++++
 include/linux/types.h              |   24 +
 include/mtd/ubi-header.h           |  372 ++++++++
 include/mtd/ubi-user.h             |  268 ++++++
 include/mtd_uboot.h                |    7 +
 include/onenand_uboot.h            |   11 +-
 include/ubi_uboot.h                |  222 +++++
 lib_generic/Makefile               |    1 +
 lib_generic/rbtree.c               |  397 +++++++++
 54 files changed, 16383 insertions(+), 80 deletions(-)

[-- Attachment #2: ubi-uboot.patch.bz2 --]
[-- Type: application/x-bzip2, Size: 89437 bytes --]

Re: [U-Boot] U-Boot and UBI

[Stefan Roese]

On Tuesday 12 August 2008, Kyungmin Park wrote:
> > On Monday 11 August 2008, Kyungmin Park wrote:
> >> Actually the Samsung implemented the UBI support on U-boot already and
> >> has used it internally. The big difference is the code base. It's
> >> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
> >> it created the ubi wrapper for u-boot.
> >
> > And how does NAND/OneNAND booting with UBI support fit into this? I
> > assume that you have some size restrictions for the IPL/SPL on your
> > platforms as well.
>
> It's not yet covered. it's TODO

I see.

> Anyway, here's UBI patch.
> The real implementation was done by another board but to publish I
> make a patch for apollon board for usage. (tested and verified)
>
> As see it's all-in-one patch and contains several parts, apollon
> board, OneNAND, mtdpartition, and UBI.

Thanks. Really appreciated.

> After the feedback, I will generate right patches for purpose.
>
> Please give any comments and feedbacks

Will come... :)

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office@denx.de
=====================================================================

Re: [U-Boot] U-Boot and UBI

[Kyungmin Park]

Hi,

On Tue, Aug 12, 2008 at 8:11 PM, Stefan Roese <sr@denx.de> wrote:
> On Tuesday 12 August 2008, Kyungmin Park wrote:
>> > On Monday 11 August 2008, Kyungmin Park wrote:
>> >> Actually the Samsung implemented the UBI support on U-boot already and
>> >> has used it internally. The big difference is the code base. It's
>> >> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
>> >> it created the ubi wrapper for u-boot.
>> >
>> > And how does NAND/OneNAND booting with UBI support fit into this? I
>> > assume that you have some size restrictions for the IPL/SPL on your
>> > platforms as well.
>>
>> It's not yet covered. it's TODO
>
> I see.
>

Yes, we should make a u-boot within one block size since flash only
guarantees the first one block as bad block free. e.g., exactly
(128KiB - 2KiB) size if OneNAND case.
For this, we need to code or size optimization.
If this is solved, we can use flash as bad block free device with UBI.

Also there are some issues related with ubi partitioning.

It's just rough ideas. For make a system as reasonable it creates two
UBI devices, one for boot-loader and another for kernel. If we make
only one UBI device, we should read whole flash blocks to read kernel.
but it's not reasonable.

1. UBI image for bootloader.
    boot params (rw), kernel(ro), [Optional] initrd (ro)

Even though it has r/w area, we regards it as static volume for
performance since boot params almost read only. I think If UBI has
only static volumes we can optimize it more. we don't need to mirror
layout volume. no wear leveling and so on. For this, of course, we
need special handling for boot params.

2. UBI image for kernel.
    rootfs(ro/rw), datafs(rw), and others(??)

Feel free, you make it for your purposes.

Thank you,
Kyungmin Park

Re: [U-Boot] U-Boot and UBI

[Josh Boyer]

On Wed, 2008-08-13 at 08:34 +0900, Kyungmin Park wrote:
> Hi,
> 
> On Tue, Aug 12, 2008 at 8:11 PM, Stefan Roese <sr@denx.de> wrote:
> > On Tuesday 12 August 2008, Kyungmin Park wrote:
> >> > On Monday 11 August 2008, Kyungmin Park wrote:
> >> >> Actually the Samsung implemented the UBI support on U-boot already and
> >> >> has used it internally. The big difference is the code base. It's
> >> >> based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
> >> >> it created the ubi wrapper for u-boot.
> >> >
> >> > And how does NAND/OneNAND booting with UBI support fit into this? I
> >> > assume that you have some size restrictions for the IPL/SPL on your
> >> > platforms as well.
> >>
> >> It's not yet covered. it's TODO
> >
> > I see.
> >
> 
> Yes, we should make a u-boot within one block size since flash only
> guarantees the first one block as bad block free. e.g., exactly
> (128KiB - 2KiB) size if OneNAND case.
> For this, we need to code or size optimization.
> If this is solved, we can use flash as bad block free device with UBI.

An alternative to that is to have a very small IPL that loads U-Boot to
DRAM as the SPL.  That would require some relocatable support in U-Boot.

josh

Re: [U-Boot] U-Boot and UBI

[Enrico Scholz]

Josh Boyer <jwboyer@linux.vnet.ibm.com> writes:

>> Yes, we should make a u-boot within one block size since flash
>> only guarantees the first one block as bad block free. e.g.,
>> exactly (128KiB - 2KiB) size if OneNAND case.
>> For this, we need to code or size optimization.  If this is
>> solved, we can use flash as bad block free device with UBI.
>
> An alternative to that is to have a very small IPL that loads
> U-Boot to DRAM as the SPL.

Yes; our implementation of such an IPL (for PXA320) can be found
at

   https://www.cvg.de/people/ensc/elito-mobm320-0.0.4.tar.bz2

This allows to jump into an image either located in a 'bootloader'
volume (which contains U-Boot) or in a 'kernel' image.

Latter is faster (by about 3-5 seconds plus the duplicate UBI
scan within U-Boot) which can make a difference as the IPL itself
needs 3.5 seconds to scan a 1GiB NAND :(


> That would require some relocatable support in U-Boot.

Depends on the system resp. how U-Boot was built (e.g. set its
TEXTADDR so that it does not conflict with the IPL).


Enrico