[U-Boot] NAND bad environment block handling

[U-Boot] NAND bad environment block handling

[Frans Meulenbroeks]

Hi,

I'm wondering what the best way is to handle bad environment blocks in NAND.

According to the spec of our supplier a delivered component is considered to be OK if less than 2% of the blocks are not bad.
This means that for our products we need to take into account that worst case 2% of the blocks are bad. But even with 1% bad blocks we have an issue:
If the bad blocks are distributed randomly we have a chance of 1/100 * 1/100 so 1 out of 10.000 that both U-Boot environment blocks are bad. 
And actually things could even be worse if the bad blocks are caused by some manufacturing defect at our supplier and both environment blocks happen to be bad in a whole batch of ICs.

Is there a solution for this?
(obviously I am not considering the situation for a single system, where I just would relocate the environment block; this concerns a production situation).

E.g. it would be nice if U-Boot could read the env from the next non-bad block at/after the env address.

Has someone experience in this area? Ideas? Suggestions?

Thanks alot!
Frans Meulenbroeks


      

[U-Boot] NAND bad environment block handling

[Cote, Sylvain]

My understanding is that u-boot already support bad block for the environment config.   When reading or writing the config from/to the nand, u-boot skip bad block.  However, you need to reserve some spare blocks in nand for environment config.  This is done in u-boot-nand.lds:

/* Align to next NAND block.  The NAND has block of 128K ==> 0x20000*/
    . = ALIGN(0x20000);
    common/environment.o  (.ppcenv)
    /* Keep some space here for redundant env and potential bad env blocks */
    . = ALIGN(0x80000);


Regards 

Sylvain C?t?

-----Original Message-----
From: u-boot-bounces@lists.denx.de [mailto:u-boot-bounces at lists.denx.de] On Behalf Of Frans Meulenbroeks
Sent: Thursday, January 08, 2009 10:19 AM
To: u-boot at lists.denx.de
Subject: [U-Boot] NAND bad environment block handling

Hi,

I'm wondering what the best way is to handle bad environment blocks in NAND.

According to the spec of our supplier a delivered component is considered to be OK if less than 2% of the blocks are not bad.
This means that for our products we need to take into account that worst case 2% of the blocks are bad. But even with 1% bad blocks we have an issue:
If the bad blocks are distributed randomly we have a chance of 1/100 * 1/100 so 1 out of 10.000 that both U-Boot environment blocks are bad. 
And actually things could even be worse if the bad blocks are caused by some manufacturing defect at our supplier and both environment blocks happen to be bad in a whole batch of ICs.

Is there a solution for this?
(obviously I am not considering the situation for a single system, where I just would relocate the environment block; this concerns a production situation).

E.g. it would be nice if U-Boot could read the env from the next non-bad block at/after the env address.

Has someone experience in this area? Ideas? Suggestions?

Thanks alot!
Frans Meulenbroeks


      
_______________________________________________
U-Boot mailing list
U-Boot at lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] NAND bad environment block handling

[Frans Meulenbroeks]

Sylvain, Scott,

Thanks for the info.
If the bad env blocks are skipped that at least would solve my problem.
I hope that it does not break fw_setenv. 
Storing the pointer in oob in block 0 is an interesting idea. I need to think about the impact on that one a little bit more.

With large blocks definitely the waste is big. Ideal solution would be that the hw does bad block mapping. Somewhat less ideal would be to position partitions dynamically  and specify the mtd partition start points at the kernel cmd line. Issue is that this is complicates the software upgrade. Even less ideal but common practice today seems to be to reserve slack room for every partition (or even worse, ignore the problem and assume perfect NAND)

Best regards, Frans


--- On Thu, 1/8/09, Cote, Sylvain <Sylvain.Cote@verint.com> wrote:

> From: Cote, Sylvain <Sylvain.Cote@verint.com>
> Subject: RE: [U-Boot] NAND bad environment block handling
> To: fransmeulenbroeks at yahoo.com, u-boot at lists.denx.de
> Date: Thursday, January 8, 2009, 5:06 PM
> My understanding is that u-boot already support bad block
> for the environment config.   When reading or writing the
> config from/to the nand, u-boot skip bad block.  However,
> you need to reserve some spare blocks in nand for
> environment config.  This is done in u-boot-nand.lds:
> 
> /* Align to next NAND block.  The NAND has block of 128K
> ==> 0x20000*/
>     . = ALIGN(0x20000);
>     common/environment.o  (.ppcenv)
>     /* Keep some space here for redundant env and potential
> bad env blocks */
>     . = ALIGN(0x80000);
> 
> 
> Regards 
> 
> Sylvain C?t?
> 
> -----Original Message-----
> From: u-boot-bounces at lists.denx.de
> [mailto:u-boot-bounces at lists.denx.de] On Behalf Of Frans
> Meulenbroeks
> Sent: Thursday, January 08, 2009 10:19 AM
> To: u-boot at lists.denx.de
> Subject: [U-Boot] NAND bad environment block handling
> 
> Hi,
> 
> I'm wondering what the best way is to handle bad
> environment blocks in NAND.
> 
> According to the spec of our supplier a delivered component
> is considered to be OK if less than 2% of the blocks are not
> bad.
> This means that for our products we need to take into
> account that worst case 2% of the blocks are bad. But even
> with 1% bad blocks we have an issue:
> If the bad blocks are distributed randomly we have a chance
> of 1/100 * 1/100 so 1 out of 10.000 that both U-Boot
> environment blocks are bad. 
> And actually things could even be worse if the bad blocks
> are caused by some manufacturing defect at our supplier and
> both environment blocks happen to be bad in a whole batch of
> ICs.
> 
> Is there a solution for this?
> (obviously I am not considering the situation for a single
> system, where I just would relocate the environment block;
> this concerns a production situation).
> 
> E.g. it would be nice if U-Boot could read the env from the
> next non-bad block at/after the env address.
> 
> Has someone experience in this area? Ideas? Suggestions?
> 
> Thanks alot!
> Frans Meulenbroeks
> 
> 
>       
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot


      

[U-Boot] NAND bad environment block handling

[Scott Wood]

On Thu, Jan 08, 2009 at 07:19:29AM -0800, Frans Meulenbroeks wrote:
> Is there a solution for this? (obviously I am not considering the
> situation for a single system, where I just would relocate the
> environment block; this concerns a production situation).
> 
> E.g. it would be nice if U-Boot could read the env from the next non-bad block at/after the env address.
> 
> Has someone experience in this area? Ideas? Suggestions?

As others pointed out, this is supported; just make the environment range
larger than the environment size.

However, this can be wasteful, especially with large block sizes; someone
a while ago proposed putting a pointer to the environment in the OOB of
the first block.

-Scott

[U-Boot] NAND bad environment block handling

[Marcel Ziswiler]

Scott Wood <scottwood <at> freescale.com> writes:

> On Thu, Jan 08, 2009 at 07:19:29AM -0800, Frans Meulenbroeks wrote:
> > Is there a solution for this? (obviously I am not considering the
> > situation for a single system, where I just would relocate the
> > environment block; this concerns a production situation).
> > 
> > E.g. it would be nice if U-Boot could read the env from the next non-bad
block at/after the env address.
> > 
> > Has someone experience in this area? Ideas? Suggestions?
> 
> As others pointed out, this is supported; just make the environment range
> larger than the environment size.
> 
> However, this can be wasteful, especially with large block sizes; someone
> a while ago proposed putting a pointer to the environment in the OOB of
> the first block.

The openmoko guys do have an alternative implementation which dynamically sizes
all partitions in NAND according to what bad blocks are found therefore not
wasting any space. And they are using it in full-scale production environment
for the GTA02. Unfortunately I don't think this made it into mainline just yet.

[U-Boot] NAND bad environment block handling

[Wolfgang Denk]

Dear Marcel Ziswiler,

In message <loom.20090109T071730-900@post.gmane.org> you wrote:
>
> The openmoko guys do have an alternative implementation which dynamically sizes
> all partitions in NAND according to what bad blocks are found therefore not
> wasting any space. And they are using it in full-scale production environment
> for the GTA02. Unfortunately I don't think this made it into mainline just yet.

This has other problems - bad blocks are not a static thing: new ones
develop over time, and shifting partitions around is not exactly nice.

You probably want to look into UBI support...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Advice is seldom welcome; and those who want it the most always like
it the least.                         -- Philip Earl of Chesterfield

[U-Boot] NAND bad environment block handling

[Schlaegl Manfred jun.]

Hi!
Am Donnerstag, den 08.01.2009, 07:19 -0800 schrieb Frans Meulenbroeks:
> 
> Has someone experience in this area? Ideas? Suggestions?

1. For our systems we assume, that the env-block never gets bad in
productive use, because the env-block is never written again. So we
reduced the problem to number of initial/factory bad blocks.
2. the partitioning and environment-saving is done at production-time.
3. we put the environment in the last good block of the device. The
search starts at the end of device and ends at the last block of the
last data-partition with an error.


Example-Partition-Table on an 32MB-NAND, with blocksize 16kb and max. 45
factory bad-blocks:
 * the part-size means the possible data size (without bad-blocks). 
 * the real partition-size on device depends on the bad-blocks in this
the partition and is calculated automatically at production-time
(partition generation).

part-size      partition
 200kb:         uboot
 1000kb:        kernel
 15000kb:       rootfs
 15832kb:       datafs
rest of device: rest (maximal 45*16kb (max bad blocks) + 16bk (envblock)
= 736kb

with an increasing number of initial/factory bad-blocks the
rest-partition shrinks. So the env-block can be written on any
nand-device without less than 45 (initial/factory) bad-blocks.


> 
> Thanks alot!
> Frans Meulenbroeks
> 

- Manfred

[U-Boot] NAND bad environment block handling

[Frans Meulenbroeks]

Manfred,

Thanks for your suggestions.
Did you modify u-boot to search from the last block of the device backwards?
I feel putting the env at the end is quite a good idea.

Unfortunately in our system we need to modify the env at runtime.
For software upgrade we aim to go for a ping pong strategy (so having two kernels and two rfs-es in flash, and upgrade the non active kernel, then modify u-boot env so the next time we boot from the upgraded kernel).
Unless of course there is a better strategy for this.

Best regards, Frans.


--- On Sun, 1/11/09, Schlaegl Manfred jun. <manfred.schlaegl@gmx.at> wrote:

> From: Schlaegl Manfred jun. <manfred.schlaegl@gmx.at>
> Subject: Re: [U-Boot] NAND bad environment block handling
> To: fransmeulenbroeks at yahoo.com
> Cc: u-boot at lists.denx.de
> Date: Sunday, January 11, 2009, 12:10 PM
> Hi!
> Am Donnerstag, den 08.01.2009, 07:19 -0800 schrieb Frans
> Meulenbroeks:
> > 
> > Has someone experience in this area? Ideas?
> Suggestions?
> 
> 1. For our systems we assume, that the env-block never gets
> bad in
> productive use, because the env-block is never written
> again. So we
> reduced the problem to number of initial/factory bad
> blocks.
> 2. the partitioning and environment-saving is done at
> production-time.
> 3. we put the environment in the last good block of the
> device. The
> search starts at the end of device and ends at the last
> block of the
> last data-partition with an error.
> 
> 
> Example-Partition-Table on an 32MB-NAND, with blocksize
> 16kb and max. 45
> factory bad-blocks:
>  * the part-size means the possible data size (without
> bad-blocks). 
>  * the real partition-size on device depends on the
> bad-blocks in this
> the partition and is calculated automatically at
> production-time
> (partition generation).
> 
> part-size      partition
>  200kb:         uboot
>  1000kb:        kernel
>  15000kb:       rootfs
>  15832kb:       datafs
> rest of device: rest (maximal 45*16kb (max bad blocks) +
> 16bk (envblock)
> = 736kb
> 
> with an increasing number of initial/factory bad-blocks the
> rest-partition shrinks. So the env-block can be written on
> any
> nand-device without less than 45 (initial/factory)
> bad-blocks.
> 
> 
> > 
> > Thanks alot!
> > Frans Meulenbroeks
> > 
> 
> - Manfred


      

[U-Boot] NAND bad environment block handling

[Schlaegl Manfred jun.]

Hi!
Am Sonntag, den 11.01.2009, 05:26 -0800 schrieb Frans Meulenbroeks:
> Manfred,
> 
> Thanks for your suggestions.
> Did you modify u-boot to search from the last block of the device backwards?
> I feel putting the env at the end is quite a good idea.
> 
Yes. First I wanted to implement the openmoku-solution: putting the
address of env-block in first block oob.
But then I thought it would be much simpler to put the env at the second
distinctive position on a nand device - the end.

Yes, I had to do some modification on uboot. But I had to do this on
uboot-1.1.3(delivered by our hardware-supplier) which is quite old. I'm
currenty on work to port our architecture to 2009q1.

> 
> Unfortunately in our system we need to modify the env at runtime.
The solution to put the environment at end of device may also be
combined with redundant environment.

> For software upgrade we aim to go for a ping pong strategy (so having two kernels and two rfs-es in flash, and upgrade the non active kernel, then modify u-boot env so the next time we boot from the upgraded kernel).
> Unless of course there is a better strategy for this.
Yes. Its a very common strategie. We also use two kernel-partitions.

- Manfred

[U-Boot] NAND bad environment block handling

[Wolfgang Denk]

Dear Manfred,

In message <1231672254.3130.29.camel@mobil.alm.archives.at> you wrote:
> 
> 1. For our systems we assume, that the env-block never gets bad in
> productive use, because the env-block is never written again. So we
> reduced the problem to number of initial/factory bad blocks.

That's a pretty risky setup because  it  is  based  on  an  incorrect
assumption  - on NAND, block become not only bad when being erased or
written to, but also after a certain number  of  read  operations  is
exceeded.  Yes, there is a maximum number of reads per block on NAND!
People like to ignore that... See for example  the  Micron  Technical
Note   "Design   and  Use  Considerations  for  NAND  Flash  Memory",
http://download.micron.com/pdf/technotes/nand/tn2917.pdf

Time to look into UBI, me thinks.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
No question is too silly to ask. Of course, some  questions  are  too
silly to to answer...  - L. Wall & R. L. Schwartz, _Programming Perl_

[U-Boot] NAND bad environment block handling

[Frans Meulenbroeks]

> 
> That's a pretty risky setup because  it  is  based  on 
> an  incorrect
> assumption  - on NAND, block become not only bad when being
> erased or
> written to, but also after a certain number  of  read 
> operations  is
> exceeded.  Yes, there is a maximum number of reads per
> block on NAND!
> People like to ignore that... See for example  the  Micron 
> Technical
> Note   "Design   and  Use  Considerations  for  NAND 
> Flash  Memory",
> http://download.micron.com/pdf/technotes/nand/tn2917.pdf

Thanks for the link. I was not aware of read degradation. That is what most manufacturers do not tell you.
> 
> Time to look into UBI, me thinks.

For u-boot env settings??

My problem that the designated env blocks can be bad or become bad during the lifetime of the device. Not desirable especially not for expensive devices.

Frans.


      

[U-Boot] NAND bad environment block handling

[Wolfgang Denk]

Dear Frans Meulenbroeks,

In message <817622.67175.qm@web33604.mail.mud.yahoo.com> you wrote:
> 
> > Time to look into UBI, me thinks.
> 
> For u-boot env settings??

Why not? Once the flash is UBI managed, there are  virtually  no  bad
blocks any more.

> My problem that the designated env blocks can be bad or become bad during the lifetime of the device. Not desirable especially not for expensive devices.

As mentioned before, U-Boot supports bad block handling on NAND, so
just allocate a few spare blocks to the environment area.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Keep your eyes wide open before marriage, half shut afterwards.
                                                 -- Benjamin Franklin

[U-Boot] NAND bad environment block handling

[Schlaegl Manfred jun.]

Am Sonntag, den 11.01.2009, 14:28 +0100 schrieb Wolfgang Denk:
> Dear Manfred,
> 
> In message <1231672254.3130.29.camel@mobil.alm.archives.at> you wrote:
> > 
> > 1. For our systems we assume, that the env-block never gets bad in
> > productive use, because the env-block is never written again. So we
> > reduced the problem to number of initial/factory bad blocks.
> 
> That's a pretty risky setup because  it  is  based  on  an  incorrect
> assumption  - on NAND, block become not only bad when being erased or
> written to, but also after a certain number  of  read  operations  is
> exceeded.  Yes, there is a maximum number of reads per block on NAND!
> People like to ignore that... See for example  the  Micron  Technical
> Note   "Design   and  Use  Considerations  for  NAND  Flash  Memory",
> http://download.micron.com/pdf/technotes/nand/tn2917.pdf
Yes I know. I think my description was a little bit misunderstanding:
Our solution is only a solution for positioning env-block(s) and
handling initial/factory bad blocks. This solution can(should) combined
with redundant env-blocks too, if write/erase-wearout is an issue.

The problem you discribe is a general nand-flash-problem. To keep the
system aware of this you theoretically have to implement a redundant
uboot partition too. It makes no sense to have two env-blocks(except for
erase/write-wearout-issues), but only one u-boot binary in nand-flash.

> 
> Time to look into UBI, me thinks.
Yes I think too. It would make things much easier .. :-)
Perhaps: Uboot + Environment (ev. Kernel) on NOR-Flash and
filesystem-partitions on NAND/UBI.

Best regards,
Manfred

[U-Boot] NAND bad environment block handling

[Derek Ou]

Hi,

I think bad blocks are less likely to happen near the first block, which 
is guaranteed to be good.  It may be possible to find NAND chip that 
guarantee more blocks but I have not encountered one myself.

U-boot has the code to skip bad block when reading environment.  As 
Sylvain said, you just need to reserve enough spare blocks.  Like his 
sample,  if you have 2 spare blocks after the environment block and 
using your 1% probability, the chance of all 3 blocks are bad are 1 out 
of 1,000,000.  In this case, the environment will collide with the next 
partition, which could be your redundant environment or even Linux kernel.

Derek
> ------------------------------
> Hi,
>
> I'm wondering what the best way is to handle bad environment blocks in NAND.
>
> According to the spec of our supplier a delivered component is considered to be OK if less than 2% of the blocks are not bad.
> This means that for our products we need to take into account that worst case 2% of the blocks are bad. But even with 1% bad blocks we have an issue:
> If the bad blocks are distributed randomly we have a chance of 1/100 * 1/100 so 1 out of 10.000 that both U-Boot environment blocks are bad. 
> And actually things could even be worse if the bad blocks are caused by some manufacturing defect at our supplier and both environment blocks happen to be bad in a whole batch of ICs.
>
> Is there a solution for this?
> (obviously I am not considering the situation for a single system, where I just would relocate the environment block; this concerns a production situation).
>
> E.g. it would be nice if U-Boot could read the env from the next non-bad block at/after the env address.
>
> Has someone experience in this area? Ideas? Suggestions?
>
> Thanks alot!
> Frans Meulenbroeks
>   

[U-Boot] NAND bad environment block handling

[David.Kondrad at onqlegrand.com]

Greetings,

I have to agree with Wolfgang.

You should be fine as long as you use multiple blocks per environment range
and use redundant environments.
These are two separate things that both need to be enabled.

Here's the setup that's working good for us.
I tested it by simulating bad blocks with 'nand markbad'.

Davinci DM6441 custom board, 64M small page nand (samsung I believe)

BLOCK
0         Reserved
1-5     TI UBL
2-31   UBoot (redundant copies that new TI UBL supports)
32-35 ENV 1
36-39 ENV 2
...

I didn't list the whole partition table, but we have two copies of every
thing in field update purposes.

Our environment is configured for 16K (1 erase block) with a 64K (4 block)
range.
We use JFFS2 for the rootfs, although as Wolfgang said, UBIFS is much
preferred if your kernel is new enough.

One thing you need to keep in mind is that if you want both copies of your
env. to always match, you need to do:
UBoot> saveenv && saveenv

The environment save does a ping-pong between redundant environments,
presumably for wear leveling.

Regards,
David

DAVID A. KONDRAD
Software Design Engineer
On-Q/Legrand
Telephone (800) 321-2343 x311
www.onqlegrand.com