What if the UBI flasher can't skip all-FF pages?

What if the UBI flasher can't skip all-FF pages?

[J Barlow]

Hi everyone,

I am developing an embedded system with a NAND flash.  We're intending to 
program all of the NAND flashes in a factory programmer.  I've read "how the UBI 
flash should work" and I suspect that the UBI flasher doesn't work that way.  
Specifically, I'm pretty sure it doesn't know how to skip pages containing only 
FFs.  Unfortunately, the flasher's software is proprietary, not to mention only 
available in Chinese.  It might well work as expected, but testing this would 
also be difficult and expensive since the flasher is in a factory in China. 

I believe that UBIFS is doing the right thing.  I blame the flasher, but I don't 
think I can do anything about it.

I'd prefer to use UBIFS as a writable root file system, if possible.  But maybe 
this won't work in my situation.

Is there any reasonable way to let UBIFS know all pages have been programmed 
initially?  For example, maybe I could deliberate replace all-FF pages with all-
00 pages in the flashed image.  Or will that just cause UBI/UBIFS to find 
corrupt empty space and fail?  Or perhaps someone could direct me how to make a 
patch that would help with UBIFS/UBI/ubinize with this specific situation.  It 
does seem like other people have run into problems with poorly written flashers 
problem, and in some cases like mine changing the flasher is not an option.

Also, if I use a read-only UBIFS in a dynamic UBI volume, will UBIFS ever 
attempt to double-program an all-FF page?  I would imagine if I used a read-only 
UBIFS in a static UBI volume I won't have trouble -- except that I also need to 
update the firmware.  Is it possible to ubiupdatevol a static UBI volume, or do 
I need something more aggressive to modify a static UBI volume?


Thanks.
J Barlow

Re: What if the UBI flasher can't skip all-FF pages?

[Artem Bityutskiy]

Hi,

On Mon, 2011-01-10 at 22:47 +0000, J Barlow wrote:
> Hi everyone,
> 
> I am developing an embedded system with a NAND flash.  We're intending to 
> program all of the NAND flashes in a factory programmer.  I've read "how the UBI 
> flash should work" and I suspect that the UBI flasher doesn't work that way.  
> Specifically, I'm pretty sure it doesn't know how to skip pages containing only 
> FFs.  Unfortunately, the flasher's software is proprietary, not to mention only 
> available in Chinese.  It might well work as expected, but testing this would 
> also be difficult and expensive since the flasher is in a factory in China.

First of all, is this a problem for your flash? If yes, why is it a
problem and why you cannot test this?

The original reason for writing this article was that some flashes have
write non-0xFF ECC to the OOB area when programmed with all 0xFFs. But
this case is easily testable.

> Is there any reasonable way to let UBIFS know all pages have been programmed 
> initially? For example, maybe I could deliberate replace all-FF pages with all-
> 00 pages in the flashed image.  Or will that just cause UBI/UBIFS to find 
> corrupt empty space and fail?  Or perhaps someone could direct me how to make a 
> patch that would help with UBIFS/UBI/ubinize with this specific situation.  It 
> does seem like other people have run into problems with poorly written flashers 
> problem, and in some cases like mine changing the flasher is not an option.

The easiest way is to introduce a bit to UBI EC headers which would mean
"I'm an LEB flashed in production, and empty pages were programmed with
0xFFs, do something". Then for each of such LEBs UBI would do full
re-write and would clean the flag. It is very easy to implement, but
would mean that the first boot basically needs full re-write of the
flashed data, which may be very slow.

With more work and thought it is also possible to invent better
solution, though. But I do not have time to think about this and
describe this right now (vacation) :-)

> Also, if I use a read-only UBIFS in a dynamic UBI volume, will UBIFS ever 
> attempt to double-program an all-FF page?

Depends. But yes, for some of them, not all.

>   I would imagine if I used a read-only 
> UBIFS in a static UBI volume I won't have trouble -- except that I also need to 
> update the firmware.

Yes, in case of R/O usage I do not see issues.

>   Is it possible to ubiupdatevol a static UBI volume, or do 
> I need something more aggressive to modify a static UBI volume?

You can use ubiupdatevol

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: What if the UBI flasher can't skip all-FF pages?

[J Barlow]

Artem Bityutskiy <dedekind1 <at> gmail.com> writes:

> 
> Hi,
> 
> On Mon, 2011-01-10 at 22:47 +0000, J Barlow wrote:
> > I am developing an embedded system with a NAND flash.  We're intending to 
> > program all of the NAND flashes in a factory programmer.
[snip]
> 
> First of all, is this a problem for your flash? If yes, why is it a
> problem and why you cannot test this?

Good point.  But unfortunately I did find out it was a problem.  I have
development tools which didn't follow the UBI flasher instructions, so I
ran into various ECC errors and found the solution on this list.  I've fixed
my development flasher tool, but I suspect the production flasher won't do the
right thing.  For other reasons we do need the production flasher.

> The original reason for writing this article was that some flashes have
> write non-0xFF ECC to the OOB area when programmed with all 0xFFs. But
> this case is easily testable.

Isn't that possible for all ECC algorithms?  If the page has a bit stuck at 0,
and is programmed to all 0xFFs, the ECC cannot possibly be all 0xFFs.

> > Is there any reasonable way to let UBIFS know all pages have been programmed 
> > initially? For example, maybe I could deliberate replace all-FF pages with 
all-
> > 00 pages in the flashed image.  Or will that just cause UBI/UBIFS to find 
> > corrupt empty space and fail?  Or perhaps someone could direct me how to 
make a 
> > patch that would help with UBIFS/UBI/ubinize with this specific situation.  
It 
> > does seem like other people have run into problems with poorly written 
flashers 
> > problem, and in some cases like mine changing the flasher is not an option.
> 
> The easiest way is to introduce a bit to UBI EC headers which would mean
> "I'm an LEB flashed in production, and empty pages were programmed with
> 0xFFs, do something". Then for each of such LEBs UBI would do full
> re-write and would clean the flag. It is very easy to implement, but
> would mean that the first boot basically needs full re-write of the
> flashed data, which may be very slow.

A long initial boot is not really a problem.  I was seriously considering this,
but I realized that I could set up a platform specific workaround.  My 
bootloader can write UBIFS correctly, so all I need to do is reprogram every
PEB in the UBI volume on the first boot before attempting to load Linux.

> With more work and thought it is also possible to invent better
> solution, though. But I do not have time to think about this and
> describe this right now (vacation) 

Enjoy your vacation.  I really appreciate your comments and work on UBI/UBIFS.
It's a great system.

J Barlow

RE: What if the UBI flasher can't skip all-FF pages?

[Jon Povey]

linux-mtd-bounces@lists.infradead.org wrote:
> Artem Bityutskiy <dedekind1 <at> gmail.com> writes:

>> The original reason for writing this article was that some flashes
>> have write non-0xFF ECC to the OOB area when programmed with all
>> 0xFFs. But this case is easily testable.
>
> Isn't that possible for all ECC algorithms?  If the page has
> a bit stuck at 0,
> and is programmed to all 0xFFs, the ECC cannot possibly be all 0xFFs.

If the page has a bit which won't erase, the block should be marked bad and retired, so correct ECC is not relevant.

An ECC algorithm that produced all-FF for all-FF input would be nice. I wonder if some controllers implement this. You could do it just by XORing with the binary-not of the underlying all-FF ECC signature. (This is just me thinking out loud about hardware I'd like to see, it doesn't help you).

> A long initial boot is not really a problem.  I was seriously
> considering this, but I realized that I could set up a platform
> specific workaround.  My bootloader can write UBIFS correctly, so all
> I need to do is reprogram every PEB in the UBI volume on the first
> boot before attempting to
> load Linux.

If that is a fresh mkfs.ubifs filesystem not many blocks will have a mix of all-FF and not-all-FF pages, you only need to rewrite those blocks.
all-FF blocks I think you can just erase completely and UBI will put headers back on later (someone else may like to comment..)

Bear in mind what happens if you lose power or get reset while doing this process. If you are planning to do it in the factory with a test after, shouldn't be a problem.

--
Jon Povey
jon.povey@racelogic.co.uk

Racelogic is a limited company registered in England. Registered number 2743719 .
Registered Office Unit 10, Swan Business Centre, Osier Way, Buckingham, Bucks, MK18 1TB .

The information contained in this electronic mail transmission is intended by Racelogic Ltd for the use of the named individual or entity to which it is directed and may contain information that is confidential or privileged. If you have received this electronic mail transmission in error, please delete it from your system without copying or forwarding it, and notify the sender of the error by reply email so that the sender's address records can be corrected. The views expressed by the sender of this communication do not necessarily represent those of Racelogic Ltd. Please note that Racelogic reserves the right to monitor e-mail communications passing through its network

RE: What if the UBI flasher can't skip all-FF pages?

[Artem Bityutskiy]

On Fri, 2011-01-14 at 05:45 +0000, Jon Povey wrote:
> > A long initial boot is not really a problem.  I was seriously
> > considering this, but I realized that I could set up a platform
> > specific workaround.  My bootloader can write UBIFS correctly, so
> all
> > I need to do is reprogram every PEB in the UBI volume on the first
> > boot before attempting to
> > load Linux.
> 
> If that is a fresh mkfs.ubifs filesystem not many blocks will have a
> mix of all-FF and not-all-FF pages, you only need to rewrite those
> blocks.
> all-FF blocks I think you can just erase completely and UBI will put
> headers back on later (someone else may like to comment..)

Right.

> Bear in mind what happens if you lose power or get reset while doing
> this process. If you are planning to do it in the factory with a test
> after, shouldn't be a problem.

We have UBI atomic leb change function for this - it should take care of
power cuts. But it has some overhead - it calculates CRC32 of the data.

So basically, my idea was:

1. Add a bit to the EC header which tells: this PEB needs re-freshing.
2. After scanning and initializing, do 'ubi_eba_atomic_leb_change()' for
all LEBs which are mapped to PEBs with that flag.
3. 'ubi_eba_atomic_leb_change()' will atomically move the data to
another PEB and clean the flag.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: What if the UBI flasher can't skip all-FF pages?

[Artem Bityutskiy]

On Mon, 2011-01-10 at 22:47 +0000, J Barlow wrote:
> Hi everyone,
> 
> I am developing an embedded system with a NAND flash.  We're intending to 
> program all of the NAND flashes in a factory programmer.  I've read "how the UBI 
> flash should work" and I suspect that the UBI flasher doesn't work that way.  
> Specifically, I'm pretty sure it doesn't know how to skip pages containing only 
> FFs.  Unfortunately, the flasher's software is proprietary, not to mention only 
> available in Chinese.  It might well work as expected, but testing this would 
> also be difficult and expensive since the flasher is in a factory in China. 
> 
> I believe that UBIFS is doing the right thing.  I blame the flasher, but I don't 
> think I can do anything about it.

But yes, UBI/UBIFS stack could do better job in this respect, but I do
not have time to implement this. But it you wish to do this - I can only
help you by suggestions.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)