ubi on MLC nand flash

ubi on MLC nand flash

[Mike Dunn]

Hi everyone,

I recently started to do serious testing of UBI on the diskonchip G4 MLC nand
driver I'm finishing up.  I started with the io_basic ubi test in mtd-utils. 
What I find is that, after a few minutes, enough PEBs are marked as bad to
exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails.  The
reason is that - on this device at least - bit flips seem to be persistent;
i.e., you will get e.g. 1 bit flip every time you read a certain page. 
Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture
test fails because the bit flip reoccurs, and the PEB is marked bad.

I expected that eventually I might have to dig into the "program disturb",
"read-disturb" or "paired pages" MLC issues, but the problem seems more
fundamental.  My general impression is that UBI is too unforgiving for this
device.  The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a
big deal.  I'm new to UBI so this is not a critique or a proposal, I'm just
hoping some experts can offer some advice or opinions.  The obvious remedy is to
set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.

Thanks,
Mike

Re: ubi on MLC nand flash

[Ivan Djelic]

On Sun, Nov 06, 2011 at 03:24:24PM +0000, Mike Dunn wrote:
> Hi everyone,
> 
> I recently started to do serious testing of UBI on the diskonchip G4 MLC nand
> driver I'm finishing up.  I started with the io_basic ubi test in mtd-utils. 
> What I find is that, after a few minutes, enough PEBs are marked as bad to
> exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails.  The
> reason is that - on this device at least - bit flips seem to be persistent;
> i.e., you will get e.g. 1 bit flip every time you read a certain page. 
> Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture
> test fails because the bit flip reoccurs, and the PEB is marked bad.

Hi Mike,
I had the same results on recent (34 nm) SLC devices.
 
> I expected that eventually I might have to dig into the "program disturb",
> "read-disturb" or "paired pages" MLC issues, but the problem seems more
> fundamental.  My general impression is that UBI is too unforgiving for this
> device.  The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a
> big deal.  I'm new to UBI so this is not a critique or a proposal, I'm just
> hoping some experts can offer some advice or opinions.  The obvious remedy is to
> set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.

I discussed the matter with a nand manufacturer a while ago; the information I
could get (for SLC devices, not MLC) can be summarized as follows:

1. A block should be marked bad if a number of bitflips greater than what ecc
is able to correct has been detected after erase/program; or if the operation
failed with a status error

2. If the maximum number of correctable bitflips is reached during a read
operation, data should be relocated to another block, without marking the block
as bad

I could not get definitive information about the handling of persistent
bitflips, apart from the fact that they are expected and should not cause a
block to be marked as bad (as long as the ecc capability is not exceeded).
Most nand datasheets I had in my hands are also vague on the subject; they lack
a precise error handling strategy description for multi-bitflip devices.

Point 2 above seems reasonable as long as bitflips are reversible (i.e.
cancelled by an erase operation); but what if the maximum number of correctable
errors is reached during a read, those errors being caused by persistent
bitflips ? Should the block be considered bad (IMHO it should be scrubbed then
marked bad), or should data be simply relocated ?
When I asked the latter question to a nand manufacturer, his recommendation
was (quoting):
"(...) not to mark the block bad (because the error is correctable), and
to keep a copy of critical data in another location as backup" (!).

I suggest the following strategy:

Upon reading, when errors are detected (and corrected by ecc):
 - if (nb of errors <  ecc capability (*)) then no scrubbing, do nothing
 - if (nb of errors == ecc capability (*)) then
    - scrub block, then torture it and compute nb of persistent bitflips
    - if (nb of persistent errors <  ecc capability (*)) then block is OK
    - if (nb of persistent errors == ecc capability (*)) then mark block as bad
      [because a single additional bitflip (e.g. a read disturb) would cause
      data loss]

(*) In order to improve reliability, thresholds can be used instead of max ecc
capability.

I'm interested to hear opinions from mtd users/nand experts on the subject; I
know that at least a few of us had to implement ecc thresholds recently. And
UBI/mtd should be modified to support this (IIRC Artem was pushing in that
direction a while ago).

BR,
--
Ivan

Re: ubi on MLC nand flash

[Mike Dunn]

Hi and thanks again, Ivan,

On 11/06/2011 09:35 AM, Ivan Djelic wrote:
> On Sun, Nov 06, 2011 at 03:24:24PM +0000, Mike Dunn wrote:
>> Hi everyone,
>>
>> I recently started to do serious testing of UBI on the diskonchip G4 MLC nand
>> driver I'm finishing up.  I started with the io_basic ubi test in mtd-utils. 
>> What I find is that, after a few minutes, enough PEBs are marked as bad to
>> exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails.  The
>> reason is that - on this device at least - bit flips seem to be persistent;
>> i.e., you will get e.g. 1 bit flip every time you read a certain page. 
>> Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture
>> test fails because the bit flip reoccurs, and the PEB is marked bad.
> Hi Mike,
> I had the same results on recent (34 nm) SLC devices.
>  
>> I expected that eventually I might have to dig into the "program disturb",
>> "read-disturb" or "paired pages" MLC issues, but the problem seems more
>> fundamental.  My general impression is that UBI is too unforgiving for this
>> device.  The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a
>> big deal.  I'm new to UBI so this is not a critique or a proposal, I'm just
>> hoping some experts can offer some advice or opinions.  The obvious remedy is to
>> set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.
> I discussed the matter with a nand manufacturer a while ago; the information I
> could get (for SLC devices, not MLC) can be summarized as follows:
>
> 1. A block should be marked bad if a number of bitflips greater than what ecc
> is able to correct has been detected after erase/program; or if the operation
> failed with a status error
>
> 2. If the maximum number of correctable bitflips is reached during a read
> operation, data should be relocated to another block, without marking the block
> as bad
>
> I could not get definitive information about the handling of persistent
> bitflips, apart from the fact that they are expected and should not cause a
> block to be marked as bad (as long as the ecc capability is not exceeded).
> Most nand datasheets I had in my hands are also vague on the subject; they lack
> a precise error handling strategy description for multi-bitflip devices.
>
> Point 2 above seems reasonable as long as bitflips are reversible (i.e.
> cancelled by an erase operation); but what if the maximum number of correctable
> errors is reached during a read, those errors being caused by persistent
> bitflips ? Should the block be considered bad (IMHO it should be scrubbed then
> marked bad), or should data be simply relocated ?
> When I asked the latter question to a nand manufacturer, his recommendation
> was (quoting):
> "(...) not to mark the block bad (because the error is correctable), and
> to keep a copy of critical data in another location as backup" (!).


Assign each block a reliability rating, and data a criticality rating?  UBI and
file I/O would get pretty bureaucratic.    :-) 


> I suggest the following strategy:
>
> Upon reading, when errors are detected (and corrected by ecc):
>  - if (nb of errors <  ecc capability (*)) then no scrubbing, do nothing
>  - if (nb of errors == ecc capability (*)) then
>     - scrub block, then torture it and compute nb of persistent bitflips
>     - if (nb of persistent errors <  ecc capability (*)) then block is OK
>     - if (nb of persistent errors == ecc capability (*)) then mark block as bad
>       [because a single additional bitflip (e.g. a read disturb) would cause
>       data loss]
>
> (*) In order to improve reliability, thresholds can be used instead of max ecc
> capability.


One wrinkle is that the torture test is performed over the entire erase block,
not just the page(s) with the correctible error(s).  So the biflip stats are
cumulative over the entire block, and may not even occur on the same page.  The
current UBI policy for the torture test is that *any* bitflips on *any* page
following the erasure causes the block to be marked bad.

Another complication is that there's currently no way to accurately determine in
the UBI code the number of bitflips the read operation caused.  Currently the
occurrence of bitflips (one or more) is determined by the return code from the
mtd subsystem, which has exclusive access to the device during the read
operation.  Just checking the ecc_stats field in the mtd_info structure could
include errors in read operations performed by other processes.


> I'm interested to hear opinions from mtd users/nand experts on the subject; I
> know that at least a few of us had to implement ecc thresholds recently. And
> UBI/mtd should be modified to support this (IIRC Artem was pushing in that
> direction a while ago).
>
>

Thanks,
Mike

Re: ubi on MLC nand flash

[Artem Bityutskiy]

On Sun, 2011-11-06 at 12:28 -0800, Mike Dunn wrote:
> > I suggest the following strategy:
> >
> > Upon reading, when errors are detected (and corrected by ecc):
> >  - if (nb of errors <  ecc capability (*)) then no scrubbing, do nothing
> >  - if (nb of errors == ecc capability (*)) then
> >     - scrub block, then torture it and compute nb of persistent bitflips
> >     - if (nb of persistent errors <  ecc capability (*)) then block is OK
> >     - if (nb of persistent errors == ecc capability (*)) then mark block as bad
> >       [because a single additional bitflip (e.g. a read disturb) would cause
> >       data loss]
> >
> > (*) In order to improve reliability, thresholds can be used instead of max ecc
> > capability.
> 
> 
> One wrinkle is that the torture test is performed over the entire erase block,
> not just the page(s) with the correctible error(s).  So the biflip stats are
> cumulative over the entire block, and may not even occur on the same page.  The
> current UBI policy for the torture test is that *any* bitflips on *any* page
> following the erasure causes the block to be marked bad.
> 
> Another complication is that there's currently no way to accurately determine in
> the UBI code the number of bitflips the read operation caused.  Currently the
> occurrence of bitflips (one or more) is determined by the return code from the
> mtd subsystem, which has exclusive access to the device during the read
> operation.  Just checking the ecc_stats field in the mtd_info structure could
> include errors in read operations performed by other processes.

What about something like this.

1. MTD knows flash's ECC strength (driver sets it)
2. MTD sets the scrub level = ECC strength by default
3. MTD can expose the scrub level and ECC strength via sysfs and make
   the scrub level sysfs file writable, so the user can vary it between
   1 and ECC strength.
4. MTD just does not report -EUCEAN if the ECC correction order is
   less than the scrub level.

Then you do not need to change UBI at all.

WRT blank pages, I guess MTD can gain some internal smartness as well -
the driver can report to the NAND base that a blank page was read, and
the ECC correction order, then NAND base will make the decision about
reporting -EUCLEAN and setting the buffer to all 0xFFs.

Also, it sounds like this may require re-working the current MTD
interface and turn all these function pointers (mtd->read(), etc) into
normal functions (mtd_read()) which will allow inserting additional
logic at various levels.

WRT ecc_stats  - IMHO, it is useless and rudimentary thing and could be
just killed...

Artem.

Re: ubi on MLC nand flash

[Mike Dunn]

On 11/08/2011 01:45 PM, Artem Bityutskiy wrote:
> On Sun, 2011-11-06 at 12:28 -0800, Mike Dunn wrote:
>>> I suggest the following strategy:
>>>
>>> Upon reading, when errors are detected (and corrected by ecc):
>>>  - if (nb of errors <  ecc capability (*)) then no scrubbing, do nothing
>>>  - if (nb of errors == ecc capability (*)) then
>>>     - scrub block, then torture it and compute nb of persistent bitflips
>>>     - if (nb of persistent errors <  ecc capability (*)) then block is OK
>>>     - if (nb of persistent errors == ecc capability (*)) then mark block as bad
>>>       [because a single additional bitflip (e.g. a read disturb) would cause
>>>       data loss]
>>>
>>> (*) In order to improve reliability, thresholds can be used instead of max ecc
>>> capability.
>>
>> One wrinkle is that the torture test is performed over the entire erase block,
>> not just the page(s) with the correctible error(s).  So the biflip stats are
>> cumulative over the entire block, and may not even occur on the same page.  The
>> current UBI policy for the torture test is that *any* bitflips on *any* page
>> following the erasure causes the block to be marked bad.
>>
>> Another complication is that there's currently no way to accurately determine in
>> the UBI code the number of bitflips the read operation caused.  Currently the
>> occurrence of bitflips (one or more) is determined by the return code from the
>> mtd subsystem, which has exclusive access to the device during the read
>> operation.  Just checking the ecc_stats field in the mtd_info structure could
>> include errors in read operations performed by other processes.
> What about something like this.
>
> 1. MTD knows flash's ECC strength (driver sets it)
> 2. MTD sets the scrub level = ECC strength by default
> 3. MTD can expose the scrub level and ECC strength via sysfs and make
>    the scrub level sysfs file writable, so the user can vary it between
>    1 and ECC strength.
> 4. MTD just does not report -EUCEAN if the ECC correction order is
>    less than the scrub level.
>
> Then you do not need to change UBI at all.


That sounds reasonable, but the changes seem broadly consequential.


> WRT blank pages, I guess MTD can gain some internal smartness as well -
> the driver can report to the NAND base that a blank page was read, and
> the ECC correction order, then NAND base will make the decision about
> reporting -EUCLEAN and setting the buffer to all 0xFFs.


I haven't yet surveyed the other drivers regarding ecc and blank page reading. 
I assumed that ecc was disregarded for blank pages, but probably some drivers
are more thoughtful about it than I originally was.


> Also, it sounds like this may require re-working the current MTD
> interface and turn all these function pointers (mtd->read(), etc) into
> normal functions (mtd_read()) which will allow inserting additional
> logic at various levels.


Oofa.  What have I gotten myself into?  I don't have all those devices on which
to test the changes, and I'd hate to break a driver.  But you're right.  Both
mtd and nand interfaces would have to change to provide a mechanism for
returning an error count (corrected or uncorrected) to some
yet-to-be-implemented mtd infrastructure code.  Drivers that don't use the NAND
interface currently return -EUCLEAN directly to the higher layer (e.g. UBI). 
For drivers using the nand interface, nand_base.c handles it. 


> WRT ecc_stats  - IMHO, it is useless and rudimentary thing and could be
> just killed...


Some userspace mtd-utils for nand currently use it, though.

I'm able to at least look into making these changes if you want to go ahead.  My
motivation is to get a robust ubifs on my diskonchip G4.

Thanks,
Mike

Re: ubi on MLC nand flash

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Tue, 2011-11-08 at 19:04 -0800, Mike Dunn wrote:
> Oofa.  What have I gotten myself into?  I don't have all those devices on which
> to test the changes, and I'd hate to break a driver.

Well, you do not have to test on all devices. If you do small
incremental reviewable changes, CC the right people, and test on nandsim
and mtdram plus your device, this should be good enough. Those who are
interested will review/test. Those who do not care - will fix their
drivers after the breakage. Of course the goal is to not break anything,
but my point is that myriads of divers which is impossible to test
should not prevent the progress :-)

> Some userspace mtd-utils for nand currently use it, though.

Well, not sure if the usage makes much sense with modern flashes.
Anyway, if you do not need to change this - don't touch it :-)

> I'm able to at least look into making these changes if you want to go ahead.  My
> motivation is to get a robust ubifs on my diskonchip G4.

Sounds good, send nice incremental patches :-)

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubi on MLC nand flash

[Mike Dunn]

On 11/09/2011 12:44 AM, Artem Bityutskiy wrote:
>
> Sounds good, send nice incremental patches :-)
>


Which git tree should I use?

Thanks,
Mike

Re: ubi on MLC nand flash

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

On Wed, 2011-11-09 at 05:13 -0800, Mike Dunn wrote:
> On 11/09/2011 12:44 AM, Artem Bityutskiy wrote:
> >
> > Sounds good, send nice incremental patches :-)
> >
> 
> 
> Which git tree should I use?

I guess the l2-mtd-2.6.git tree since it contains the latest MTD work.
Although it is currently based on 3.2-rc1 which may be a bit unstable.
If this does not work, we can create something based on 3.1 release plus
the latest MTD patches for you, I suppose.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: ubi on MLC nand flash

[Artem Bityutskiy]

On Sun, 2011-11-06 at 07:24 -0800, Mike Dunn wrote:
> Hi everyone,
> 
> I recently started to do serious testing of UBI on the diskonchip G4 MLC nand
> driver I'm finishing up.

Sounds promising - we've got another serious mtd community memeber! :-)

>   I started with the io_basic ubi test in mtd-utils.

Makes sense to exclude UBIFS and test UBI directly first indeed.

> What I find is that, after a few minutes, enough PEBs are marked as bad to
> exhaust the reserve PEB pool

I guess you can make it larger, the default 1% is just something which
was good enough for our super-robust OneNAND flash.

Also, for MLC you probably want a smaller WL threshold, I heard that
modern MLCs have ereaseblock liftimes smaller than 10000 erase-cycles.
So the default 4096 might be too big.

> , UBI switches to r/o mode, and the test fails.  The
> reason is that - on this device at least - bit flips seem to be persistent;
> i.e., you will get e.g. 1 bit flip every time you read a certain page. 
> Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture
> test fails because the bit flip reoccurs, and the PEB is marked bad.

A quick hack you can do to go further in your investigations without
being block by this issue is to hack your driver and make it to just not
return -EUCLEAN in case of 1 bit flip or may be even 2. Then you can see
ahead what else happens to UBI.

WRT the real solution - I agree with Ivan - see his e-mail, and I'll
send some comments on that.

> I expected that eventually I might have to dig into the "program disturb",
> "read-disturb" or "paired pages" MLC issues, but the problem seems more
> fundamental.  My general impression is that UBI is too unforgiving for this
> device.  The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a
> big deal.  I'm new to UBI so this is not a critique or a proposal, I'm just
> hoping some experts can offer some advice or opinions.  The obvious remedy is to
> set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.

You are right - UBI is too unforgiving. But this should be fixable, it
just needs a brave knight to do the job :-)

Artem.

Re: ubi on MLC nand flash

[Mike Dunn]

Hi Artem, thanks for having a look.

On 11/08/2011 01:32 PM, Artem Bityutskiy wrote:
> On Sun, 2011-11-06 at 07:24 -0800, Mike Dunn wrote:
>
>> What I find is that, after a few minutes, enough PEBs are marked as bad to
>> exhaust the reserve PEB pool
> I guess you can make it larger, the default 1% is just something which
> was good enough for our super-robust OneNAND flash.


This device only has 512 erase blocks (256K each), and typically fewer usable
for a filesystem, so I hate to expend more for overhead.


> Also, for MLC you probably want a smaller WL threshold, I heard that
> modern MLCs have ereaseblock liftimes smaller than 10000 erase-cycles.
> So the default 4096 might be too big.


Thanks.  Did see the comment in ubi/Kconfig.


>> , UBI switches to r/o mode, and the test fails.  The
>> reason is that - on this device at least - bit flips seem to be persistent;
>> i.e., you will get e.g. 1 bit flip every time you read a certain page. 
>> Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture
>> test fails because the bit flip reoccurs, and the PEB is marked bad.
> A quick hack you can do to go further in your investigations without
> being block by this issue is to hack your driver and make it to just not
> return -EUCLEAN in case of 1 bit flip or may be even 2. Then you can see
> ahead what else happens to UBI.


Already done as a temporary work-around.  But -EUCLEAN is actually returned from
nand_base.c

Mike