Bootloader support for UBI images

Bootloader support for UBI images

[Richard Titmuss]

Hi,

The system I am developing keeps two kernel and root filesystem images 
to allow easy recovery from any errors. I would like to use UBI to store 
both the kernel and filesystem, to take advantage of the error recovery, 
wear leveling, etc. I see searching the list adding support for UBI into 
a bootloader has been discussed before, but no patches are available so 
far.

I have now ported a limited read-only version of UBI to redboot, to 
allow loading static volumes. This is at a stage where I can now boot 
the system after loading the kernel from the UBI volume. It should be 
easy to also extend this to work with u-boot (but I have no immediate 
need for this myself). I will be adding erase and writing for UBI 
images, but not updating volumes as this is significantly more complex. 
If people are interested I can post the redboot package.

Before I proceed any further I have some questions. To upgrade the 
system I would like to do something like:

# Remove backup volumes
ubirmvol /dev/ubi0 -N kernel.old
ubirmvol /dev/ubi0 -N rootfs.old
# Create new volumes
ubimkvol /dev/ubi0 -N kernel.new -s ...
ubimkvol /dev/ubi0 -N rootfs.old -s ...
# Write new volume contents
ubiupdatevol /dev/ubi0 -N kernel.new zImage
ubiupdatevol /dev/ubi0 -N rootfs.new rootfs
# Atomically rename volumes, switch to the new images
ubirenamevol /dev/ubi0 kernel kernel.old kernel.new kernel rootfs 
rootfs.old rootfs.new rootfs

This would require adding support for -N to ubiupdatevol to allow the 
volume name to be used (just syntactic sugar) . Also I need support for 
renaming multiple volumes atomically when switching images after the 
upgrade. If these sound sensible and useful changes I can look at 
creating patches to add these features.

The initial scanning of the flash in the bootloader seems to be taking a 
long time (longer than mounting the volume from linux). I need to look 
at why this is taking longer, but it would be great to avoid doing two 
full table scans. Passing the scan information from the bootloader to 
the kernel seems tricky, especially as the kernel builds multiple trees 
and lists during the scan. Bootloader support for UBI imagesA possible 
alternative approach would be to include an LER to PER index table for 
static volumes only. This table would only need updating when the static 
volume was rewritten or a PER moved due to a bit error or wear leveling. 
If the index table and the volume table were always located near the 
start or end of the flash then a full volume scan could be avoided in 
the bootloader. Does this sound like a reasonable optimization, or is 
another solution preferred?

Richard

Re: Bootloader support for UBI images

[Artem Bityutskiy]

Hi,

On Tue, 2008-07-01 at 12:43 +0100, Richard Titmuss wrote:
> Before I proceed any further I have some questions. To upgrade the 
> system I would like to do something like:
> 
> # Remove backup volumes
> ubirmvol /dev/ubi0 -N kernel.old
> ubirmvol /dev/ubi0 -N rootfs.old
> # Create new volumes
> ubimkvol /dev/ubi0 -N kernel.new -s ...
> ubimkvol /dev/ubi0 -N rootfs.old -s ...
> # Write new volume contents
> ubiupdatevol /dev/ubi0 -N kernel.new zImage
> ubiupdatevol /dev/ubi0 -N rootfs.new rootfs
> # Atomically rename volumes, switch to the new images
> ubirenamevol /dev/ubi0 kernel kernel.old kernel.new kernel rootfs 
> rootfs.old rootfs.new rootfs
> 
> This would require adding support for -N to ubiupdatevol to allow the 
> volume name to be used (just syntactic sugar).
Sounds fine and useful.

>  Also I need support for 
> renaming multiple volumes atomically when switching images after the 
> upgrade. If these sound sensible and useful changes I can look at 
> creating patches to add these features.
Also sounds as a good and useful idea. And implementation should not
difficult - it is just about changing the volume table. You may count on
my assistance.

I'm just not sure about the interface. Renaming one volume is fine. But
you need to rename 2, so the interface should be more complex. And if 2,
why not 3, or 5? Or 16? Or any amount from 1 to UBI_MAX_VOLUMES?

> The initial scanning of the flash in the bootloader seems to be taking a 
> long time (longer than mounting the volume from linux). I need to look 
> at why this is taking longer, but it would be great to avoid doing two 
> full table scans.

So you use the sub-page read patch BTW?

>  Passing the scan information from the bootloader to 
> the kernel seems tricky, especially as the kernel builds multiple trees 
> and lists during the scan.
>  Bootloader support for UBI imagesA possible 
> alternative approach would be to include an LER to PER index table for 
> static volumes only. This table would only need updating when the static 
> volume was rewritten or a PER moved due to a bit error or wear leveling. 
> If the index table and the volume table were always located near the 
> start or end of the flash then a full volume scan could be avoided in 
> the bootloader. Does this sound like a reasonable optimization, or is 
> another solution preferred?

Well, surely the boot-loader may create a per-PEB (physical eraseblock)
array and put all information about each PEB there (mapped/unmapped, LEB
number, erase-counter, etc)? Then in UBI it will be easy to scan this
array instead of the flash media. I am just not aware how such kind of
bootloader->kernel data passing is done in Linux. Can you come up with a
nice and mainline-acceptible way?

I did not actually catch the second part (LER and PER) tables - this is
something I am not aware of. UBI has only volume table on flash, the
other things are in-memory and build by scanning.

But in general, the idea not to do double scan but pass the scanning
information from boot-loader to the kernel sounds good. We just need to
figure out how to do this nicely.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Bootloader support for UBI images

[Artem Bityutskiy]

On Tue, 2008-07-01 at 15:26 +0300, Artem Bityutskiy wrote:
> >  Also I need support for 
> > renaming multiple volumes atomically when switching images after the 
> > upgrade. If these sound sensible and useful changes I can look at 
> > creating patches to add these features.
> Also sounds as a good and useful idea. And implementation should not
> difficult - it is just about changing the volume table. You may count on
> my assistance.
> 
> I'm just not sure about the interface. Renaming one volume is fine. But
> you need to rename 2, so the interface should be more complex. And if 2,
> why not 3, or 5? Or 16? Or any amount from 1 to UBI_MAX_VOLUMES?

Just a note: I was talking about how would the corresponding UBI
ioctl(s) interface(s) look like. Kernel people are very picky at this.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: Bootloader support for UBI images

[Richard Titmuss]

Artem Bityutskiy wrote:

>>  Also I need support for 
>> renaming multiple volumes atomically when switching images after the 
>> upgrade. If these sound sensible and useful changes I can look at 
>> creating patches to add these features.
>>     
> Also sounds as a good and useful idea. And implementation should not
> difficult - it is just about changing the volume table. You may count on
> my assistance.
>
> I'm just not sure about the interface. Renaming one volume is fine. But
> you need to rename 2, so the interface should be more complex. And if 2,
> why not 3, or 5? Or 16? Or any amount from 1 to UBI_MAX_VOLUMES?
>   
Right, I was thinking that it would allow any number of volumes for 
renaming. I am also not sure about the interface, and was hoping to get 
some suggestions here. I will think about it and propose something in a 
few days.

> So you use the sub-page read patch BTW?
No, I'd missed that. Where is it?

> Well, surely the boot-loader may create a per-PEB (physical eraseblock)
> array and put all information about each PEB there (mapped/unmapped, LEB
> number, erase-counter, etc)? Then in UBI it will be easy to scan this
> array instead of the flash media. I am just not aware how such kind of
> bootloader->kernel data passing is done in Linux. Can you come up with a
> nice and mainline-acceptible way?
>   
Ok, that is possible. This would also have to include the bad block and 
bit flip status for each PEB. At the moment the Redboot flash api does 
not make this available, so it would mean extending Redboot or writing 
custom flash drivers to access this information.

> I did not actually catch the second part (LER and PER) tables - this is
> something I am not aware of. UBI has only volume table on flash, the
> other things are in-memory and build by scanning.
>   
It was a proposal to _add_ an addition table, that provided a LER to PER 
mapping for static volumes. If this table was stored near the beginning 
of the flash, then it could be accessed without a full scan. This could 
only be used if the UBI volume was mounted read-only, any write 
operations would require scanning all the flash blocks. This is an 
alternative approach to passing data from the bootloader to the kernel.

Richard

Re: Bootloader support for UBI images

[Artem Bityutskiy]

On Wed, 2008-07-02 at 08:57 +0100, Richard Titmuss wrote:
> Right, I was thinking that it would allow any number of volumes for 
> renaming. I am also not sure about the interface, and was hoping to get 
> some suggestions here. I will think about it and propose something in a 
> few days.

Do you absolutely need multiple renames? Could you live with one at
a time?

> > So you use the sub-page read patch BTW?
> No, I'd missed that. Where is it?

Here is the relevant thread. Some people already use that patch. dwmw2
was positive about it and told he would accept it.

http://lists.infradead.org/pipermail/linux-mtd/2008-April/021412.html

> Ok, that is possible. This would also have to include the bad block and 
> bit flip status for each PEB. At the moment the Redboot flash api does 
> not make this available, so it would mean extending Redboot or writing 
> custom flash drivers to access this information.

Well, bit-flip information is not mandatory and may be done later as
an improvement. Bad-block information have to be available.

> > I did not actually catch the second part (LER and PER) tables - this is
> > something I am not aware of. UBI has only volume table on flash, the
> > other things are in-memory and build by scanning.
> >   
> It was a proposal to _add_ an addition table, that provided a LER to PER 
> mapping for static volumes.

Ah, this is s/LER/LEB/, s/PER/PEB/.

>  If this table was stored near the beginning 
> of the flash, then it could be accessed without a full scan. This could 
> only be used if the UBI volume was mounted read-only, any write 
> operations would require scanning all the flash blocks. This is an 
> alternative approach to passing data from the bootloader to the kernel.

Well, may be done, but the R/O limitation seems to be too hard.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)