[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Artem Bityutskiy]

On Wed, 2011-06-01 at 00:57 -0700, Lars Michael wrote:
> Hello,
> 
> Using mkfs.ubifs and ubinize I have created an image with 5 volumes.
> In Linux I could flash it using "ubiformat /dev/mtd1 -f /root/ubi.img"
> and attach it using "ubiattach /dev/ubi_ctrl -m 1". It worked.

OK.

> In production I would prefer to write the image from U-Boot. So in
> U-Boot I used "nand erase" then "nand write" to flash the image. Then
> I want to attach by "ubi part" command - but this fails: "UBI error:
> ubi_init: UBI error: cannot initialize UBI, error -17"

Sorry, I'm not familiar with U-boot, so probably cc-ing U-boot mailing
list would be a good idea.

> Then I booted Linux (via nfs) and tried to re-format+attach, but now
> it fails. Format complains about eraseblock contents and missing erase
> counters: 
> 
> ubiformat: warning!: 1099 of 2036 eraseblocks contain non-ubifs data
> ubiformat: warning!: only 937 of 2036 eraseblocks have valid erase
> counter

OK, this means that many eraseblocks in the flash contain some data
which does not look like UBI headers.

> 
> The ubiattach now reports lots of:
> ...
> UBI error: check_corruption: PEB 0 contains corrupted VID header, and
> the data does not contain all 0xFF, this may be a non-UBI PEB or a
> severe VID header corruption which requires manual inspection
> 
> UBI warning: process_eb: valid VID header but corrupted EC header at
> PEB 4
> ...

OK.

> I realize that the U-Boot 'nand erase' does not preserve the erase
> counters. So what would be the correct way to erase and write in
> U-Boot?

I do not know, you need to ask U-boot-aware people. I've CCed some.

>  And why is the ubiformat not able to correct the problems?

It should be able to format any flash. Probably this is a bug. Can you
reproduce this with nandsim please? Just insmod nandsim on your PC (see
http://www.linux-mtd.infradead.org/faq/nand.html#L_nand_nandsim or
modinfo nandsim), write some garbage there or your image, and try to
format it.

If you can describe how I can reproduce this in my PC, I might look at
this. Or you can just debug ubiformat yourself.

-- 
Best Regards,
Artem Bityutskiy (????? ????????)

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Ben Gardiner]

Hi Lars

On Wed, Jun 1, 2011 at 3:57 AM, Lars Michael <lh_post@yahoo.com> wrote:
> Hello,
>
> Using mkfs.ubifs and ubinize I have created an image with 5 volumes. In Linux I could flash it using "ubiformat /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach /dev/ubi_ctrl -m 1". It worked.
>
> In production I would prefer to write the image from U-Boot. So in U-Boot I used "nand erase" then "nand write" to flash the image. Then I want to attach by "ubi part" command - but this fails: "UBI error: ubi_init: UBI error: cannot initialize UBI, error -17"
> [...]
> I realize that the U-Boot 'nand erase' does not preserve the erase counters. So what would be the correct way to erase and write in U-Boot? And why is the ubiformat not able to correct the problems?

It's true that the ECs are not preserved. But it's also true that
trailing pages of 0xff are not trimmed either. You definitely need to
drop trailing 0xff pages when writing ubinized images containing UBIFS
volumes. To do this you can use either the new 'nand write' variant,
.trimffs [1] which is currently patches that have been Acked. Or you
can use the free-space-fixup feature which is available in the
ubifs-2.6 tree and requires that you use a mkfs.ubifs built from the
the mtd-utils tree.

I've never seen the particular error "ubi_init: UBI error: cannot
initialize UBI, error -17" when I have erroneously written a ubinized
image with nand write. But maybe that is due to the paritculars of my
platform drivers or it may be due to the version of u-boot you are
running there -- see below.

I reccomend giving both the 'nand write.trimffs' patches and the
free-space-fixup feature a try (indivually) and reporting back with
your success. After you update u-boot, see below:

> My kernel is 2.6.29 patched with 165 patched from the ubi back port tree.
> U-Boot is 2009.08 with ubifs support, but I am not sure if there are relevant patches. The CPU is a v4 ColdFire and 256MB Micron NAND.

2009.08 is maybe too old. There have been several fixes to both ubi
and ubis in u-boot since then:

$git log --oneline v2009.08..u-boot/master  -- fs/ubifs/ drivers/mtd/ubi/
c7b7d45 UBIFS: Fix dereferencing type-punned pointer compiler warning
64b6817 ubifs.c: BUG: Error following links
b1a14f8 UBIFS: Change ubifsload to not read beyond the requested size
cb9c09d UBIFS: Add ubifsumount command to unmount an active volume
6d8962e Switch from archive libraries to partial linking
2e5167c Replace CONFIG_RELOC_FIXUP_WORKS by CONFIG_NEEDS_MANUAL_RELOC
86af10c Fix "ubi part" cmd re-entrancy
ceeba00 UBI: initialise update marker
1b1f9a9 UBI: Ensure that "background thread" operations are really executed
6865168 ubifsmount fails due to not initialized list
87d93a1 move prototypes for gunzip() and zunzip() to common.h
4b142fe common: delete CONFIG_SYS_64BIT_VSPRINTF and CONFIG_SYS_64BIT_STRTOUL
521af04 Conditionally perform common relocation fixups
ef37c68 ubifs: Correct dereferencing of files-after-symlinks
9d7952e ubifs: Add support for looking up directory and relative symlinks
02f9990 Move __set/clear_bit from ubifs.h to bitops.h


Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Lars Michael]

Hello Ben,

--- On Wed, 1/6/11, Ben Gardiner <bengardiner@nanometrics.ca> wrote:
> > Using mkfs.ubifs and ubinize I have created an image
> with 5 volumes. In Linux I could flash it using "ubiformat
> /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach
> /dev/ubi_ctrl -m 1". It worked.
> >
> > In production I would prefer to write the image from
> U-Boot. So in U-Boot I used "nand erase" then "nand write"
> to flash the image. Then I want to attach by "ubi part"
> command - but this fails: "UBI error: ubi_init: UBI error:
> cannot initialize UBI, error -17"
> > [...]
> > I realize that the U-Boot 'nand erase' does not
> preserve the erase counters. So what would be the correct
> way to erase and write in U-Boot? And why is the ubiformat
> not able to correct the problems?
> 
> It's true that the ECs are not preserved. But it's also
> true that
> trailing pages of 0xff are not trimmed either. You
> definitely need to
> drop trailing 0xff pages when writing ubinized images
> containing UBIFS
> volumes. To do this you can use either the new 'nand write'
> variant,
> .trimffs [1] which is currently patches that have been
> Acked. Or you
> can use the free-space-fixup feature which is available in
> the
> ubifs-2.6 tree and requires that you use a mkfs.ubifs built
> from the
> the mtd-utils tree.

Ok, I have updated my mtd-utils and recompiled the mkfs.ubifs to get
support for the free-space-fixup feature. Next I have pulled
all ubi updates from the 2.6.29 backport tree which gave med 154 patches
including the fixup.

Now I can build the image, ubiformat, ubiattach and mount with success.
After a reboot I can ubiattach and mount with success.

Next step is to get it to work in U-Boot too.

Thanks and regards,
- Lars

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Aaron Williams]

Hi Lars,

Were you ever successful in getting UBI to work in U-Boot? I am in a similar 
boat. I have UBI working beautifully in Linux (2.6.32.27) but so far I've had 
no luck in U-Boot.

-Aaron

On Monday, July 18, 2011 02:29:08 AM Lars Michael wrote:
> Hello Ben,
> 
> --- On Wed, 1/6/11, Ben Gardiner <bengardiner@nanometrics.ca> wrote:
> > > Using mkfs.ubifs and ubinize I have created an image
> > 
> > with 5 volumes. In Linux I could flash it using "ubiformat
> > /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach
> > /dev/ubi_ctrl -m 1". It worked.
> > 
> > > In production I would prefer to write the image from
> > 
> > U-Boot. So in U-Boot I used "nand erase" then "nand write"
> > to flash the image. Then I want to attach by "ubi part"
> > command - but this fails: "UBI error: ubi_init: UBI error:
> > cannot initialize UBI, error -17"
> > 
> > > [...]
> > > I realize that the U-Boot 'nand erase' does not
> > 
> > preserve the erase counters. So what would be the correct
> > way to erase and write in U-Boot? And why is the ubiformat
> > not able to correct the problems?
> > 
> > It's true that the ECs are not preserved. But it's also
> > true that
> > trailing pages of 0xff are not trimmed either. You
> > definitely need to
> > drop trailing 0xff pages when writing ubinized images
> > containing UBIFS
> > volumes. To do this you can use either the new 'nand write'
> > variant,
> > .trimffs [1] which is currently patches that have been
> > Acked. Or you
> > can use the free-space-fixup feature which is available in
> > the
> > ubifs-2.6 tree and requires that you use a mkfs.ubifs built
> > from the
> > the mtd-utils tree.
> 
> Ok, I have updated my mtd-utils and recompiled the mkfs.ubifs to get
> support for the free-space-fixup feature. Next I have pulled
> all ubi updates from the 2.6.29 backport tree which gave med 154 patches
> including the fixup.
> 
> Now I can build the image, ubiformat, ubiattach and mount with success.
> After a reboot I can ubiattach and mount with success.
> 
> Next step is to get it to work in U-Boot too.
> 
> Thanks and regards,
> - Lars
> 
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

-- 
Aaron Williams <Aaron.Williams@cavium.com>
 (408) 943-7198

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Lars Michael]

--- On Thu, 29/9/11, Aaron Williams <Aaron.Williams@cavium.com> wrote:
> From: Aaron Williams <Aaron.Williams@cavium.com>
> Subject: Re: [U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails
> To: u-boot at lists.denx.de
> Cc: "Lars Michael" <lh_post@yahoo.com>
> Date: Thursday, 29 September, 2011, 4:11
> Hi Lars,
> 
> Were you ever successful in getting UBI to work in U-Boot?
> I am in a similar 
> boat. I have UBI working beautifully in Linux (2.6.32.27)
> but so far I've had 
> no luck in U-Boot.
> 
> -Aaron

Hi Aaron,

Good question, I am not able to write a ubi image (created by ubinize) 
and then attatch to it from U-Boot. I found that I need the free space
fixup patch. My U-Boot is rather old (2009.08) and I can not easily
apply that patch due to too many other changes. So for now, I must
write the image from Linux, do the fixup and then U-Boot is able to
attach. 

Regards,
- Lars 

> On Monday, July 18, 2011 02:29:08 AM Lars Michael wrote:
> > Hello Ben,
> > 
> > --- On Wed, 1/6/11, Ben Gardiner <bengardiner@nanometrics.ca>
> wrote:
> > > > Using mkfs.ubifs and ubinize I have created
> an image
> > > 
> > > with 5 volumes. In Linux I could flash it using
> "ubiformat
> > > /dev/mtd1 -f /root/ubi.img" and attach it using
> "ubiattach
> > > /dev/ubi_ctrl -m 1". It worked.
> > > 
> > > > In production I would prefer to write the
> image from
> > > 
> > > U-Boot. So in U-Boot I used "nand erase" then
> "nand write"
> > > to flash the image. Then I want to attach by "ubi
> part"
> > > command - but this fails: "UBI error: ubi_init:
> UBI error:
> > > cannot initialize UBI, error -17"
> > > 
> > > > [...]
> > > > I realize that the U-Boot 'nand erase' does
> not
> > > 
> > > preserve the erase counters. So what would be the
> correct
> > > way to erase and write in U-Boot? And why is the
> ubiformat
> > > not able to correct the problems?
> > > 
> > > It's true that the ECs are not preserved. But
> it's also
> > > true that
> > > trailing pages of 0xff are not trimmed either.
> You
> > > definitely need to
> > > drop trailing 0xff pages when writing ubinized
> images
> > > containing UBIFS
> > > volumes. To do this you can use either the new
> 'nand write'
> > > variant,
> > > .trimffs [1] which is currently patches that have
> been
> > > Acked. Or you
> > > can use the free-space-fixup feature which is
> available in
> > > the
> > > ubifs-2.6 tree and requires that you use a
> mkfs.ubifs built
> > > from the
> > > the mtd-utils tree.
> > 
> > Ok, I have updated my mtd-utils and recompiled the
> mkfs.ubifs to get
> > support for the free-space-fixup feature. Next I have
> pulled
> > all ubi updates from the 2.6.29 backport tree which
> gave med 154 patches
> > including the fixup.
> > 
> > Now I can build the image, ubiformat, ubiattach and
> mount with success.
> > After a reboot I can ubiattach and mount with
> success.
> > 
> > Next step is to get it to work in U-Boot too.
> > 
> > Thanks and regards,
> > - Lars
> > 
> > _______________________________________________
> > U-Boot mailing list
> > U-Boot at lists.denx.de
> > http://lists.denx.de/mailman/listinfo/u-boot
> 
> -- 
> Aaron Williams <Aaron.Williams@cavium.com>
>  (408) 943-7198
> 

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Aaron Williams]

Thanks.

I finally found the problem on our end and now I have ubifs working
beautifully.

-Aaron

On 09/29/2011 08:33 PM, Lars Michael wrote:
> --- On Thu, 29/9/11, Aaron Williams <Aaron.Williams@cavium.com> wrote:
>> From: Aaron Williams <Aaron.Williams@cavium.com>
>> Subject: Re: [U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails
>> To: u-boot at lists.denx.de
>> Cc: "Lars Michael" <lh_post@yahoo.com>
>> Date: Thursday, 29 September, 2011, 4:11
>> Hi Lars,
>>
>> Were you ever successful in getting UBI to work in U-Boot?
>> I am in a similar 
>> boat. I have UBI working beautifully in Linux (2.6.32.27)
>> but so far I've had 
>> no luck in U-Boot.
>>
>> -Aaron
> Hi Aaron,
>
> Good question, I am not able to write a ubi image (created by ubinize) 
> and then attatch to it from U-Boot. I found that I need the free space
> fixup patch. My U-Boot is rather old (2009.08) and I can not easily
> apply that patch due to too many other changes. So for now, I must
> write the image from Linux, do the fixup and then U-Boot is able to
> attach. 
>
> Regards,
> - Lars 
>
>> On Monday, July 18, 2011 02:29:08 AM Lars Michael wrote:
>>> Hello Ben,
>>>
>>> --- On Wed, 1/6/11, Ben Gardiner <bengardiner@nanometrics.ca>
>> wrote:
>>>>> Using mkfs.ubifs and ubinize I have created
>> an image
>>>> with 5 volumes. In Linux I could flash it using
>> "ubiformat
>>>> /dev/mtd1 -f /root/ubi.img" and attach it using
>> "ubiattach
>>>> /dev/ubi_ctrl -m 1". It worked.
>>>>
>>>>> In production I would prefer to write the
>> image from
>>>> U-Boot. So in U-Boot I used "nand erase" then
>> "nand write"
>>>> to flash the image. Then I want to attach by "ubi
>> part"
>>>> command - but this fails: "UBI error: ubi_init:
>> UBI error:
>>>> cannot initialize UBI, error -17"
>>>>
>>>>> [...]
>>>>> I realize that the U-Boot 'nand erase' does
>> not
>>>> preserve the erase counters. So what would be the
>> correct
>>>> way to erase and write in U-Boot? And why is the
>> ubiformat
>>>> not able to correct the problems?
>>>>
>>>> It's true that the ECs are not preserved. But
>> it's also
>>>> true that
>>>> trailing pages of 0xff are not trimmed either.
>> You
>>>> definitely need to
>>>> drop trailing 0xff pages when writing ubinized
>> images
>>>> containing UBIFS
>>>> volumes. To do this you can use either the new
>> 'nand write'
>>>> variant,
>>>> .trimffs [1] which is currently patches that have
>> been
>>>> Acked. Or you
>>>> can use the free-space-fixup feature which is
>> available in
>>>> the
>>>> ubifs-2.6 tree and requires that you use a
>> mkfs.ubifs built
>>>> from the
>>>> the mtd-utils tree.
>>> Ok, I have updated my mtd-utils and recompiled the
>> mkfs.ubifs to get
>>> support for the free-space-fixup feature. Next I have
>> pulled
>>> all ubi updates from the 2.6.29 backport tree which
>> gave med 154 patches
>>> including the fixup.
>>>
>>> Now I can build the image, ubiformat, ubiattach and
>> mount with success.
>>> After a reboot I can ubiattach and mount with
>> success.
>>> Next step is to get it to work in U-Boot too.
>>>
>>> Thanks and regards,
>>> - Lars
>>>
>>> _______________________________________________
>>> U-Boot mailing list
>>> U-Boot at lists.denx.de
>>> http://lists.denx.de/mailman/listinfo/u-boot
>> -- 
>> Aaron Williams <Aaron.Williams@cavium.com>
>>  (408) 943-7198
>>

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Lars Michael]

--- On Fri, 30/9/11, Aaron Williams <Aaron.Williams@cavium.com> wrote:
> I finally found the problem on our end and now I have ubifs
> working
> beautifully.
> 
> -Aaron

Good, do you mind sharing what the solution to your problem is and what version of U-Boot you are running?

Thanks,
- Lars 

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Matthew L. Creech]

On Wed, Jun 1, 2011 at 3:57 AM, Lars Michael <lh_post@yahoo.com> wrote:
> Hello,
>
> Using mkfs.ubifs and ubinize I have created an image with 5 volumes. In Linux I could flash it using "ubiformat /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach /dev/ubi_ctrl -m 1". It worked.
>
> In production I would prefer to write the image from U-Boot. So in U-Boot I used "nand erase" then "nand write" to flash the image. Then I want to attach by "ubi part" command - but this fails: "UBI error: ubi_init: UBI error: cannot initialize UBI, error -17"
>

Basically, this isn't a valid combination: "nand write" doesn't know
anything about UBI, and a UBIFS image isn't suitable for flashing
directly to NAND - it expects to sit on top of UBI.

There are 2 ways to get it working:

1. Don't use "nand write".  If U-Boot is built with UBI support, you
can use "ubi write" to write a UBI image to flash.  I _think_ you can
use this to overwrite an existing volume with a new image, preserving
ECs (haven't tried it myself).  It certainly works fine to write a new
image to blank flash.

2. Use the "ubinize" tool to create an image suitable for writing
directly to NAND.  This takes a UBIFS image and "wraps" it to turn it
into a UBI image, which you can then use with "nand write".

I've used both these methods to write UBIFS images to flash within
U-Boot, and the 2nd method also works with industrial NAND
programmers.

Hope this helps

-- 
Matthew L. Creech

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Lars Michael]

--- On Thu, 29/9/11, Matthew L. Creech <mlcreech@gmail.com> wrote:
> On Wed, Jun 1, 2011 at 3:57 AM, Lars
> Michael <lh_post@yahoo.com> wrote:
> > Using mkfs.ubifs and ubinize I have created an image
> with 5 volumes. In Linux I could flash it using "ubiformat
> /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach
> /dev/ubi_ctrl -m 1". It worked.
> >
> > In production I would prefer to write the image from
> U-Boot. So in U-Boot I used "nand erase" then "nand write"
> to flash the image. Then I want to attach by "ubi part"
> command - but this fails: "UBI error: ubi_init: UBI error:
> cannot initialize UBI, error -17"
> >
> 
> Basically, this isn't a valid combination: "nand write"
> doesn't know
> anything about UBI, and a UBIFS image isn't suitable for
> flashing
> directly to NAND - it expects to sit on top of UBI.

Hi Matthew,

I agree, which is why I used ubinize to create a UBI image (with 5 UBIFS images) and then tried to write that image.

> There are 2 ways to get it working:
> 
> 1. Don't use "nand write".? If U-Boot is built with
> UBI support, you
> can use "ubi write" to write a UBI image to flash.? I
> _think_ you can
> use this to overwrite an existing volume with a new image,
> preserving
> ECs (haven't tried it myself).? It certainly works
> fine to write a new
> image to blank flash.
>
> 2. Use the "ubinize" tool to create an image suitable for
> writing
> directly to NAND.? This takes a UBIFS image and
> "wraps" it to turn it
> into a UBI image, which you can then use with "nand
> write".

This is what I am doing (or plan to do). But to get it to work, I need
the space fixup feature for the nand write in U-Boot. I got that feature in Linux, so in Linux it works. Unfortunately it is not easy to patch 
that one in my 2009.08 U-Boot.

Thanks for your comments.

Regards
Lars

[U-Boot] U-Boot: using nand write to flash an ubi image, ubi part fails

[Scott Wood]

On 09/29/2011 10:44 PM, Lars Michael wrote:
> --- On Thu, 29/9/11, Matthew L. Creech <mlcreech@gmail.com> wrote:
>> 2. Use the "ubinize" tool to create an image suitable for
>> writing
>> directly to NAND.  This takes a UBIFS image and
>> "wraps" it to turn it
>> into a UBI image, which you can then use with "nand
>> write".
> 
> This is what I am doing (or plan to do). But to get it to work, I need
> the space fixup feature for the nand write in U-Boot. I got that feature in Linux, so in Linux it works. Unfortunately it is not easy to patch 
> that one in my 2009.08 U-Boot.

Try using a current U-Boot, and use "nand write.trimffs".

-Scott