CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[David Peverley]

Hi all,

I've noticed that some of the problems we see are exacerbated further
by us having CONFIG_MTD_NAND_VERIFY_WRITE enabled. In the case where
blocks have occasional 1-bit ECC failures (i.e. normally correctable
and not enough to warrant marking as bad) the generic verify routine
will cause nand_write_page() to return failure. I've prototyped a
verify routine that uses an ECC corrected read in our driver and it
seems to do the job correctly.

I think that there's a good argument to address this by changing the
generic verify routine(s) by doing something along the lines of the
following :

static uint8_t readbackbuf[NAND_MAX_PAGESIZE];

static int nand_verify_buf(struct mtd_info *mtd, const uint8_t *buf, int len)
{
    int err;
    struct nand_chip *chip = mtd->priv;

    /* Read back page using ECC read */
    err = chip->ecc.read_page(mtd, this, readbackbuf_ecc);
    if (err < 0) {
        printk(KERN_NOTICE "nand_verify_buf: ERROR: ecc.read_page() failed.\n");
        return -EFAULT;
    }

    if (memcmp(readbackbuf, buf, len) != 0)
        return -EFAULT;

    return 0;
}

which would use the appropriate page read mechanism based on what was
configured as a result of ecc.mode. Does this make sense to you guys?
If so I'll put together a patch to submit. This would also
nand_verify_buf16() becomes obsolete as the bus width differences are
taken care of within the call to read_page().

Of course it adds a little more overhead but will avoid writes failing
and possibly causing some blocks to be marked as bad before their
time...

Cheers,

~Pev

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ricard Wanderlof]

On Tue, 15 Feb 2011, David Peverley wrote:

> I've noticed that some of the problems we see are exacerbated further by 
> us having CONFIG_MTD_NAND_VERIFY_WRITE enabled. In the case where blocks 
> have occasional 1-bit ECC failures (i.e. normally correctable and not 
> enough to warrant marking as bad) the generic verify routine will cause 
> nand_write_page() to return failure. I've prototyped a verify routine 
> that uses an ECC corrected read in our driver and it seems to do the job 
> correctly.

I may be wrong here, but as I understand, normally, bit read errors occur 
after a while (often called 'read disturb' in the literature) as the 
contents of certain bit cells leak away (either over time, or by 
accessing), first causing random data, and then after a while settling 
down into a permanently inverted state, until the data is rewritten.

If this is the case, then a verify performed just after a write (which I'm 
assuming is when it is performed) should yield correct data, unless a 
given bit cell is in a really bad condition, in which case the block 
should probably not be used anyway, as there as not been enough time for 
the bit to decay for whatever reason.

Of course, if there are bit cells which have other failure modes, your 
idea might help, but again, if a bit is in such a bad state as to not 
being able to retain data between write and verify, the block should 
probably be discarded (marked as bad) anyway.

One rationale would be that if you have one more or less permanently bad 
bit, although it can be handled by ECC, as soon the contents of another 
bit cell decays, the ECC won't be effective, so in practice, the ECC 
strength is severely reduced for the page (or sub-page more likely, as ECC 
usually operates over less-than-page sizes in modern flashes) in question.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[David Peverley]

Hi Ricard,

Thats a good analysis ; I've dug around as a result and found an
interesting tech note from Micron at :
    http://download.micron.com/pdf/technotes/nand/tn2917.pdf

They classify NAND failures into various types ; Permanent Failures
and Temporary Failures, with the temporary failures being split into
"Program Disturb", "Read disturb", "Over-programming" and "Data loss"
- I won't paraphrase further here as the Micron document describes it
better than I ever could! :-)

>From the description of read disturb, it occurs due to many reads
(hundreds of thousands or millions) prior to an erase. Currently my
testing is using nandtestc.c and mtd_stresstest.ko - the former tests
one cycle before re-programming and the latter is random but not
expected to be more than tens of reads before a re-programme becomes
statistically likely. Potentially program disturb sounds like it
_could_ be the behaviour I observe but it's not clear.

My general take on this is that only the permanent type failures i.e.
those involving permanently stuck bits, require marking as bad blocks.
The recovery recommended for the other scenarios is always to erase
and re-programme. This potentially opens up a whole can of worms... My
interpretation of this is that if we verify a write and we've had a
(correctable and non-permanent) single bit error the Right Thing To Do
would be to erase and re-programme the block, probably with a very
small retry limit. We could argue that it's the responsibility of the
file-system to do this but programatically I think nand_write_page()
is best placed to be able to do this.

Certainly the verify failures we see here with a raw read are
occasional (and not consistently the same blocks) and hence not
indicative of stuck bits and generally after the block is re-written
the read is correct. What do you reckon?

Cheers,

~Pev

On 15 February 2011 13:02, Ricard Wanderlof <ricard.wanderlof@axis.com> wrote:
>
> On Tue, 15 Feb 2011, David Peverley wrote:
>
>> I've noticed that some of the problems we see are exacerbated further by
>> us having CONFIG_MTD_NAND_VERIFY_WRITE enabled. In the case where blocks
>> have occasional 1-bit ECC failures (i.e. normally correctable and not enough
>> to warrant marking as bad) the generic verify routine will cause
>> nand_write_page() to return failure. I've prototyped a verify routine that
>> uses an ECC corrected read in our driver and it seems to do the job
>> correctly.
>
> I may be wrong here, but as I understand, normally, bit read errors occur
> after a while (often called 'read disturb' in the literature) as the
> contents of certain bit cells leak away (either over time, or by accessing),
> first causing random data, and then after a while settling down into a
> permanently inverted state, until the data is rewritten.
>
> If this is the case, then a verify performed just after a write (which I'm
> assuming is when it is performed) should yield correct data, unless a given
> bit cell is in a really bad condition, in which case the block should
> probably not be used anyway, as there as not been enough time for the bit to
> decay for whatever reason.
>
> Of course, if there are bit cells which have other failure modes, your idea
> might help, but again, if a bit is in such a bad state as to not being able
> to retain data between write and verify, the block should probably be
> discarded (marked as bad) anyway.
>
> One rationale would be that if you have one more or less permanently bad
> bit, although it can be handled by ECC, as soon the contents of another bit
> cell decays, the ECC won't be effective, so in practice, the ECC strength is
> severely reduced for the page (or sub-page more likely, as ECC usually
> operates over less-than-page sizes in modern flashes) in question.
>
> /Ricard
> --
> Ricard Wolf Wanderlöf                           ricardw(at)axis.com
> Axis Communications AB, Lund, Sweden            www.axis.com
> Phone +46 46 272 2016                           Fax +46 46 13 61 30
>

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ricard Wanderlof]

On Tue, 15 Feb 2011, David Peverley wrote:

> Thats a good analysis ; I've dug around as a result and found an
> interesting tech note from Micron at :
>    http://download.micron.com/pdf/technotes/nand/tn2917.pdf

Yes, that's a good (and one of the few) notes on failure mechanisms.

> They classify NAND failures into various types ; Permanent Failures
> and Temporary Failures, with the temporary failures being split into
> "Program Disturb", "Read disturb", "Over-programming" and "Data loss"
> ...
> From the description of read disturb, it occurs due to many reads
> (hundreds of thousands or millions) prior to an erase. Currently my
> testing is using nandtestc.c and mtd_stresstest.ko - the former tests
> one cycle before re-programming and the latter is random but not
> expected to be more than tens of reads before a re-programme becomes
> statistically likely.

I agree, it doesn't sound like that is your problem.

> Potentially program disturb sounds like it _could_ be the behaviour I 
> observe but it's not clear.

It seems to fit in with the description, i.e. you get bits programmed that 
were not intended to be programmed. It seems to get worse when not 
programming whole pages at once (partial page programming).

> My general take on this is that only the permanent type failures i.e. 
> those involving permanently stuck bits, require marking as bad blocks. 
> The recovery recommended for the other scenarios is always to erase and 
> re-programme. This potentially opens up a whole can of worms... My 
> interpretation of this is that if we verify a write and we've had a 
> (correctable and non-permanent) single bit error the Right Thing To Do 
> would be to erase and re-programme the block, probably with a very small 
> retry limit. We could argue that it's the responsibility of the 
> file-system to do this but programatically I think nand_write_page() is 
> best placed to be able to do this.

Either the file system, or some flash management layer such as UBI should 
take care of this, but I know that jffs2 doesn't at any rate. I'd agree it 
makes sense for the lower level to to the best of its capability guarantee 
that the data written actually does get written on the flash.

I think that the application note in some respects simplifies matters a 
bit. If you have a block that is wearing out, due to a large number of 
erase/write cycles, it will exhibit several failure modes at an increasing 
rate, in particular long-term data loss. At some point one could argue 
that the block is essentially worn out, and mark it as bad, even though an 
erase/write/verify cycle actually might be successful. I don't think that 
is what is happening in your case though.

> Certainly the verify failures we see here with a raw read are
> occasional (and not consistently the same blocks) and hence not
> indicative of stuck bits and generally after the block is re-written
> the read is correct. What do you reckon?

It would seem that if you have only occasional faults in arbitrary blocks 
it wouldn't be a wear problem; if the blame is with the flash I would 
agree it fits in with the 'Program Disturb' description. I must admit I've 
not come across this type of error myself, but that could be because of 
limited experience or that it occurs extremely infrequently in the types 
of flash that I've been exposed to so I've never noticed it.

I don't know if anyone else here on this list has any experiences to 
share. Frankly, if I saw errors of that type I would start looking for 
hardware problems, or some sort of hardware or software induced contention 
on the flash chip bus. Not that that would necessarily be the right 
approach, but I've seen errors of that type occurring as the result of 
out-of-spec level shifters between the MCU and flash chip, or incorrectly 
set up bus timing towards the flash.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[David Peverley]

Hi Ricard,

I did some more Googling and found a couple more interesting articles
; I don't know if you've read them (I assume you probably have!) but
thought I'd repost for anyone else interested as this seems to be a
really fascinating and not often discussed topic! :
  http://www.broadbandreports.com/r0/download/1507743~59e7b9dda2c0e0a0f7ff119a7611c641/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf

I found the following also really interesting, especially for the
analysis of lots of devices with a plot of numbers of initial bad
blocks marked which I'd always wondered about! :
  http://trs-new.jpl.nasa.gov/dspace/bitstream/2014/40761/1/08-07.pdf

> Either the file system, or some flash management layer such as UBI should
> take care of this, but I know that jffs2 doesn't at any rate. I'd agree it
> makes sense for the lower level to to the best of its capability guarantee
> that the data written actually does get written on the flash.
Well, we're using YAFFS2 - I haven't looked into how it deals with
these scenarios though... I'll see if I can figure out how it behaves.

> I think that the application note in some respects simplifies matters a bit.
> If you have a block that is wearing out, due to a large number of
> erase/write cycles, it will exhibit several failure modes at an increasing
> rate, in particular long-term data loss. At some point one could argue that
> the block is essentially worn out, and mark it as bad, even though an
> erase/write/verify cycle actually might be successful. I don't think that is
> what is happening in your case though.
That's all to do with what one considers a "Bad Block" - I'd agree
that the repeated failures can show that there might be an issue but
all the literature I've read today state that only permanent failures
are regarded as showing a bad block and these are reported via the
flashes status read. In fact I found a Micron Appnote AN1819

> I don't know if anyone else here on this list has any experiences to share.
> Frankly, if I saw errors of that type I would start looking for hardware
> problems, or some sort of hardware or software induced contention on the
> flash chip bus. Not that that would necessarily be the right approach, but
> I've seen errors of that type occurring as the result of out-of-spec level
> shifters between the MCU and flash chip, or incorrectly set up bus timing
> towards the flash.
We're looking into these possibilities as well - However as is often
the case, such problems provoke testing of less used code paths so
it's quite a good thing to look at the right thing to do in this event
in conjunction with fixing the root cause of the problem...

Thanks again!

~Pev

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ricard Wanderlof]

On Tue, 15 Feb 2011, David Peverley wrote:

> I did some more Googling and found a couple more interesting articles
> ; I don't know if you've read them (I assume you probably have!) but
> thought I'd repost for anyone else interested as this seems to be a
> really fascinating and not often discussed topic! :
>  http://www.broadbandreports.com/r0/download/1507743~59e7b9dda2c0e0a0f7ff119a7611c641/flash_mem_summit_jcooke_inconvenient_truths_nand.pdf
>
> I found the following also really interesting, especially for the
> analysis of lots of devices with a plot of numbers of initial bad
> blocks marked which I'd always wondered about! :
>  http://trs-new.jpl.nasa.gov/dspace/bitstream/2014/40761/1/08-07.pdf

Thanks for the links. I have actually seen the first of these, but not the 
second (which however in parts seems to draw heavily from the first one).

A couple of years ago I actually did some tests on a few 256 Mb SLC NAND 
chips in order to better understand the failure modes. These were 
specified as withstanding a maximum of 100 000 erase/program cycles. 
Indeed, in one portion of the test, after 1.7 million erase/write cycles, 
repetetive reading the same block would result in uncorrectable ECC 
failures (i.e. two bit errors per 256 bytes) after 1.6 million read 
cycles. This is, with some margin, in tune to Cooke's rule of thumb that 
a maximum of 1 million read cycles should be done before rewriting the 
block (the block in question started exhibiting correctable (i.e. 1-bit) 
errors after a mere 40000 read cycles, but then again I was way over the 
spec in terms of erase/write cycles).

One part of Nasa's analysis confused me. They attempt to erase factory 
marked bad blocks and include them in the rest of the tests in order to 
compare the characteristics of the known bad blocks with the rest of the 
memory array.

However, data sheets often mention that one should not attempt to write or 
erase bad blocks (e.g. the Samsung K9F1G08X0B data sheet simply says 'Do 
not erase or program factory-marked bad blocks.'), and I've read 
application notes explaining that bad blocks may have failures that can 
cause unreliable operation of the rest of the chip if it is attempt to use 
them. I.e. bad blocks do not necessarily just contain bad bit cells, 
indeed there could be any other failure in the die causing irregular 
operation when for instance internal erase voltages are applied to the 
block.

Therefore, erasing bad blocks and including them in the test is basically 
violating the device specification to start with.

The fact that bad blocks can not be replicated by simple testing is also 
well documented by flash manufacturers, either in data sheets or 
application notes. I remember reading that in order to determine whether a 
block is bad, a number of tests are run at the factory, in much more 
severe conditions that the average use case. I don't know if this just 
involves testing at abnormal voltages and temperatures, or if the die is 
also examined visually (or some other non-electric way) for errors prior 
to being packaged, or if undocumented chip access modes are used during 
the tests.

Indeed, in ST's application note AN1819 which you refer too (as sourced 
from Micron, I think this is simply because Micron bought out Numonyx 
which bought out ST's flash division, or however the exact economic 
transactions went), mentions that "The invalid [i.e. bad] blocks are 
detected at the factory during the testing process which involves severe 
environmental conditions and program/erase cycles as well as proprietary 
test modes. The failures that affect invalid blocks may not all be 
recognized if methods different from those implemented in the factory are 
used.".

So as far as I'm concerned it is well documented that it is not possible 
to determine which blocks were marked bad at the factory from the user's 
point of view.

Indeed, erasing or writing bad blocks could put the integrity of the whole 
chip in question.

I second their conclusion however that most often bad blocks after erasure 
seem to function just as well as good blocks. From time to time I've had 
flash programmers screw up and mark a large portion of blocks as bad. 
Erasing the whole chip, including factory-marked bad blocks, yields a 
workable chip again, seemingly without any adverse affects at least in a 
lab environment. I would not try and sell a unit whose factory-marked 
flash bad blocks had been erased however, but it seems perfectly fine for 
everyday use as a development platform.

I have yet to come across a chip with hard bad block errors, on the other 
hand I can't say that I've come across more than a handful of chips who 
have had their initial bad blocks erased.

>> If you have a block that is wearing out, due to a large number of
>> erase/write cycles, it will exhibit several failure modes at an increasing
>> rate, in particular long-term data loss. At some point one could argue that
>> the block is essentially worn out, and mark it as bad, even though an
>> erase/write/verify cycle actually might be successful. I don't think that is
>> what is happening in your case though.
> That's all to do with what one considers a "Bad Block" - I'd agree
> that the repeated failures can show that there might be an issue but
> all the literature I've read today state that only permanent failures
> are regarded as showing a bad block and these are reported via the
> flashes status read. In fact I found a Micron Appnote AN1819

In the torture testing I mentioned above on the 256 Mb chips, one part of 
the test involved erase/write/read-cycling four blocks over and over again 
until an erase or write failure was reported by the chip. In this 
particular case, 3.7 million cycles were performed before mtd reported an 
'I/O error' (probably due to a chip timeout) on erase.

However, after this, I performed a complete reflash of the chip, which 
didn't result in any errors reported at all. A couple of more erase/write 
attempts showed that sometimes an error during write would be reported, 
sometimes not, never during erase (as had initially been the case).

The tortured block endured a mere 550 reads before the ECC 'correctable' 
counter started increasing, and a mere 10000 reads before the ECC 'failed' 
(2-bit errors) counter started increasing.

While this is a bit of an artificial test, being well over the specified 
endurance specficiation, it does show a couple of things.

When an erase or write error is reported by the flash chip, the block in 
question may be well over its specified erase/write endurance cycle 
specification. So just using the chip's own failed erase or write 
indication is not really sufficient or reliable. Indeed, as shown, after 
3.7 million erase/write cycles, some writes still performed flawlessly, 
but read performance was very lacking. Most likely read performance would 
have been just as bad after say 3.5 million erase/write cycles, before any 
write or erase errors had been reported by the chip.

Which all goes to say that as Cooke notes in his document above, it is 
necessary for the software to keep track of the number of erase cycles, 
and not just rely in the erase/write status that the chip itself reports.

> We're looking into these possibilities as well - However as is often
> the case, such problems provoke testing of less used code paths so
> it's quite a good thing to look at the right thing to do in this event
> in conjunction with fixing the root cause of the problem...

I agree.

Sorry for the long-winded reply. I'm not trying to prove I'm correct, but 
I do want to share my experiences and value input from others, as there 
doesn't seem to be a lot of hard information out there on these issues. 
Besides, there seems to be a lot of misconceptions which I think can be 
cleared up by discussion.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Thu, 2011-02-17 at 11:04 +0100, Ricard Wanderlof wrote:
> Which all goes to say that as Cooke notes in his document above, it is 
> necessary for the software to keep track of the number of erase cycles, 
> and not just rely in the erase/write status that the chip itself reports.

Yeah, in UBI we do keep have erase counters, but we do not actually use
them to make decisions about whether to mark a block as good or bad.
Probably we should.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ricard Wanderlof]

On Fri, 25 Feb 2011, Artem Bityutskiy wrote:
> 
> > On Thu, 2011-02-17 at 11:04 +0100, Ricard Wanderlof wrote:
> > Which all goes to say that as Cooke notes in his document above, it is 
> > necessary for the software to keep track of the number of erase cycles, 
> > and not just rely in the erase/write status that the chip itself reports.
> 
> Yeah, in UBI we do keep have erase counters, but we do not actually use
> them to make decisions about whether to mark a block as good or bad.
> Probably we should.

It's a difficult call. How bad is bad? If the spec says 100 000 cycles, 
should we mark a block bad after that time, or could we in a certain case 
go to 200 000 cycles and still get reasonable performance? It would all 
depend on the application and the also specs of the actual chip in 
question, which is something that cannot be read from the chip so it would 
have to be configured.

I think the best way is for UBI to provide only wear levelling, in order 
to use the flash optimally, then it's up to the system designer to design 
the system so that the maxium erase cycle spec is not exceeded for the 
life of the system.

Hm, perhaps there could be a tuning option, first to enable bad block 
marking when the erase counters reach a certain value, and secondly a 
parameter specifying the number of cycles. Most users would probably not 
bother about this, but it would be there for those who want to make sure 
that blocks are not used that potentially could be out of spec.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Fri, 2011-02-25 at 10:09 +0100, Ricard Wanderlof wrote:
> On Fri, 25 Feb 2011, Artem Bityutskiy wrote:
> > 
> > > On Thu, 2011-02-17 at 11:04 +0100, Ricard Wanderlof wrote:
> > > Which all goes to say that as Cooke notes in his document above, it is 
> > > necessary for the software to keep track of the number of erase cycles, 
> > > and not just rely in the erase/write status that the chip itself reports.
> > 
> > Yeah, in UBI we do keep have erase counters, but we do not actually use
> > them to make decisions about whether to mark a block as good or bad.
> > Probably we should.
> 
> It's a difficult call. How bad is bad? If the spec says 100 000 cycles, 
> should we mark a block bad after that time, or could we in a certain case 
> go to 200 000 cycles and still get reasonable performance? It would all 
> depend on the application and the also specs of the actual chip in 
> question, which is something that cannot be read from the chip so it would 
> have to be configured.
> 
> I think the best way is for UBI to provide only wear levelling, in order 
> to use the flash optimally, then it's up to the system designer to design 
> the system so that the maxium erase cycle spec is not exceeded for the 
> life of the system.
> 
> Hm, perhaps there could be a tuning option, first to enable bad block 
> marking when the erase counters reach a certain value, and secondly a 
> parameter specifying the number of cycles. Most users would probably not 
> bother about this, but it would be there for those who want to make sure 
> that blocks are not used that potentially could be out of spec.

Yes, something like this. I am not going to do anything about this now,
just wanted to let potential UBI users know about this good idea.

Currently the mechanism to mark a block is bad is the torture function
failure: we write a pattern, read it back, compare, and do this several
times with different patterns. In case of any error in any step, or if
we read back something we did not write, or even if we get a bit-flip
when we read back the data, we bark the eraseblock as bad. Otherwise it
is returned to the pull of free eraseblocks.

See torture_peb() in drivers/mtd/ubi/io.c

This procedure is not ideal, and could be improved:

a) we could store amount of times the eraseblock was tortured. Since we
torture only if there was a write error, too many torture session would
indicate that the eraseblock is unstable.
b) we could take into account the erase count somehow.

But yes, the threshold would probably set up by the system designer at
the end.

Thanks!

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ivan Djelic]

On Fri, Feb 25, 2011 at 10:29:22AM +0000, Artem Bityutskiy wrote:
(...)
> Currently the mechanism to mark a block is bad is the torture function
> failure: we write a pattern, read it back, compare, and do this several
> times with different patterns. In case of any error in any step, or if
> we read back something we did not write, or even if we get a bit-flip
> when we read back the data, we bark the eraseblock as bad. Otherwise it
> is returned to the pull of free eraseblocks.
> 
> See torture_peb() in drivers/mtd/ubi/io.c
> 
> This procedure is not ideal, and could be improved:
> 
> a) we could store amount of times the eraseblock was tortured. Since we
> torture only if there was a write error, too many torture session would
> indicate that the eraseblock is unstable.
> b) we could take into account the erase count somehow.
> 
> But yes, the threshold would probably set up by the system designer at
> the end.

The fact that a bitflip detected during torture is enough to decide that a
block is bad causes problems on some 4-bit ecc devices we are using. If we
stick to this policy, we end up with a _lot_ of blocks being marked as bad
(i.e. way too many).

Our NAND manufacturer tells us that, as long as a block erase operation
completes without a failure reported by the device, it should not be classified
as bad, even if it has bitflips (which sounds risky at best).

Right now, we implement a bitflip threshold, below which we correct ecc errors
without reporting them. When the bitflip threshold is reached, we report the
amount of corrected errors, triggering block scrubbing, etc.
This is not ideal, but it prevents UBI from torturing and marking too many
blocks as bad.
Regards,

Ivan

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Fri, 2011-02-25 at 12:36 +0100, Ivan Djelic wrote:
> On Fri, Feb 25, 2011 at 10:29:22AM +0000, Artem Bityutskiy wrote:
> (...)
> > Currently the mechanism to mark a block is bad is the torture function
> > failure: we write a pattern, read it back, compare, and do this several
> > times with different patterns. In case of any error in any step, or if
> > we read back something we did not write, or even if we get a bit-flip
> > when we read back the data, we bark the eraseblock as bad. Otherwise it
> > is returned to the pull of free eraseblocks.
> > 
> > See torture_peb() in drivers/mtd/ubi/io.c
> > 
> > This procedure is not ideal, and could be improved:
> > 
> > a) we could store amount of times the eraseblock was tortured. Since we
> > torture only if there was a write error, too many torture session would
> > indicate that the eraseblock is unstable.
> > b) we could take into account the erase count somehow.
> > 
> > But yes, the threshold would probably set up by the system designer at
> > the end.
> 
> The fact that a bitflip detected during torture is enough to decide that a
> block is bad causes problems on some 4-bit ecc devices we are using. If we
> stick to this policy, we end up with a _lot_ of blocks being marked as bad
> (i.e. way too many).

I see. May be in your case 1 bit errors are completely harmless, but 2
and 3 are not?

> Our NAND manufacturer tells us that, as long as a block erase operation
> completes without a failure reported by the device, it should not be classified
> as bad, even if it has bitflips (which sounds risky at best).

For any amount of flipped bits per page? Sounds a bit scary.

> Right now, we implement a bitflip threshold, below which we correct ecc errors
> without reporting them. When the bitflip threshold is reached, we report the
> amount of corrected errors, triggering block scrubbing, etc.
> This is not ideal, but it prevents UBI from torturing and marking too many
> blocks as bad.

Hmm... Working around UBI behavior does not sound like a the best
solution.

How about changing the MTD interface a little and teach it to:

1. Report the bit-flip level (or you name it properly) - the amount of
bits flipped in this NAND page (or sub-page). If we read more than one
NAND page at one go, and several pages had bit-flips of different level,
report the maximum.

2. Make it possible for drivers to set the "bit-flip tolerance
threshold" (invent a better name please), which is lowest the bit-flip
level which should be considered harmful. E.g., in your case, the
threshold could be 2.

3. Make UBI only react on bit-flips with order higher or equivalend to
the threshold. In your case then, UBI would ignore all level 1 bit-flips
and react only to level 2, 3, and 4 bit-flips.

Does this sound sensible?

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[David Peverley]

Hi All,

> How about changing the MTD interface a little and teach it to:
> 1. Report the bit-flip level (or you name it properly) - the amount of
> bits flipped in this NAND page (or sub-page). If we read more than one
> NAND page at one go, and several pages had bit-flips of different level,
> report the maximum.
This is important to allow the file-system to make an informed
decision based on what happened within MTD. i.e. it would be able to
ascertain whether a read corrected N bit errors and thus be able to
re-program a block as required. I think is the only way to accomplish
handling this at the FS level? However, given that the count comes
from a specific read operation, I think it would need to be
implemented as part of the read call e.g. passing in something like
"unsigned *bit_errors"? This would mean changing something fundamental
or providing a new API to read with this extra parameter so I'm not
sure how this could be accomplished 'nicely'...

Something else to consider - what about bit-flips during writes? These
are currently only spotted if you enable write verify (and also not
dealt with or even notified beyond a generic errcode) and ought to be
handled / notified in a similar manner to the read but it's a lot
harder to decide how to split this functionality between the MTD and
FS layers as they're both closely involved. Not everyone wants to
enable the additional overhead of verifys. Would it be reasonable just
to leave it to the ECC check on the next read to that block instead?

In many ways it would be a lot easier (and involve only a single
implementation) to provide a config option where if a read or write in
MTD fails with a (correctable) bit flip or flips that the MTD itself
takes it on itself to erase the block and re-write the data cleanly. I
like this from the simplicity perspective and the fact it would be
transparent. However its contrary to the concept of this level of
management being performed exclusively by the FS layer. Of course you
still need some sort of stats collecting to help the FS if it wants to
use these stats to assist in deciding when to retire blocks...

~Pev

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Fri, 2011-02-25 at 12:59 +0000, David Peverley wrote:
> Hi All,
> 
> > How about changing the MTD interface a little and teach it to:
> > 1. Report the bit-flip level (or you name it properly) - the amount of
> > bits flipped in this NAND page (or sub-page). If we read more than one
> > NAND page at one go, and several pages had bit-flips of different level,
> > report the maximum.
> This is important to allow the file-system to make an informed
> decision based on what happened within MTD. i.e. it would be able to
> ascertain whether a read corrected N bit errors and thus be able to
> re-program a block as required. I think is the only way to accomplish
> handling this at the FS level? However, given that the count comes
> from a specific read operation, I think it would need to be
> implemented as part of the read call e.g. passing in something like
> "unsigned *bit_errors"? This would mean changing something fundamental
> or providing a new API to read with this extra parameter so I'm not
> sure how this could be accomplished 'nicely'...

I do not think it is an issue - changing in-kernel API is relatively
easy and straight-forward task.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ivan Djelic]

On Fri, Feb 25, 2011 at 12:59:01PM +0000, David Peverley wrote:
(...)
> Something else to consider - what about bit-flips during writes? These
> are currently only spotted if you enable write verify (and also not
> dealt with or even notified beyond a generic errcode) and ought to be
> handled / notified in a similar manner to the read but it's a lot
> harder to decide how to split this functionality between the MTD and
> FS layers as they're both closely involved. Not everyone wants to
> enable the additional overhead of verifys. Would it be reasonable just
> to leave it to the ECC check on the next read to that block instead?

It is reasonable only if you can guarantee that each page you program belongs to
a properly erased block, and has never been programmed before.

Power failure during erase or programming can produce seemingly erased pages,
but, in reality, unstable pages showing a random (i.e. large), changing amount
of bitflips. At least on recent (<= 50 nm technology) devices.

As of today, the above safety condition is not guaranteed by mtd+ubi+ubifs in
the presence of power failures; in which case enabling write verify is necessary
to avoid data corruption.

> In many ways it would be a lot easier (and involve only a single
> implementation) to provide a config option where if a read or write in
> MTD fails with a (correctable) bit flip or flips that the MTD itself
> takes it on itself to erase the block and re-write the data cleanly.

In order to do that in a safe, atomic way, you need to move data from one block
to another; and to do it in a transparent way, you need some kind of block
management layer, hence UBI...

BR,

Ivan

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ivan Djelic]

On Fri, Feb 25, 2011 at 12:12:10PM +0000, Artem Bityutskiy wrote:
> On Fri, 2011-02-25 at 12:36 +0100, Ivan Djelic wrote:
(...)
> > 
> > The fact that a bitflip detected during torture is enough to decide that a
> > block is bad causes problems on some 4-bit ecc devices we are using. If we
> > stick to this policy, we end up with a _lot_ of blocks being marked as bad
> > (i.e. way too many).
> 
> I see. May be in your case 1 bit errors are completely harmless, but 2
> and 3 are not?

When a NAND device requires 4-bit ecc or more, you do see a lot of 1-bit errors
(compared to previous NAND devices). They are not "completely harmless" because
you are still supposed to relocate data in some other block and erase the block
(those bitflips are reversible errors), in order to avoid error accumulation
and stay below the specified ecc requirement. But they probably should not be
considered an indication that the block has gone bad.

> > Our NAND manufacturer tells us that, as long as a block erase operation
> > completes without a failure reported by the device, it should not be classified
> > as bad, even if it has bitflips (which sounds risky at best).
> 
> For any amount of flipped bits per page? Sounds a bit scary.

I agree. Our NAND manufacturer even told us that a single permanent 1-bit
failure in a block is not enough for marking this block as bad on 4-bit ecc NAND
devices. I still think there should be a specified amount of errors above which
the block should be considered bad. Maybe only permanent bit failures should
be considered.

Just for information, in our case: 1-bit and 2-bit errors are not reported,
3-bit and above are reported. And we are able to correct up to 8 errors (while
the device only requires 4-bit correction), so we have some kind of safety
margin.

> > Right now, we implement a bitflip threshold, below which we correct ecc errors
> > without reporting them. When the bitflip threshold is reached, we report the
> > amount of corrected errors, triggering block scrubbing, etc.
> > This is not ideal, but it prevents UBI from torturing and marking too many
> > blocks as bad.
> 
> Hmm... Working around UBI behavior does not sound like a the best
> solution.

Agreed.

> How about changing the MTD interface a little and teach it to:
> 
> 1. Report the bit-flip level (or you name it properly) - the amount of
> bits flipped in this NAND page (or sub-page). If we read more than one
> NAND page at one go, and several pages had bit-flips of different level,
> report the maximum.

Yes, we do need the maximum error count per subpage. Today we only have a
cumulative count.

> 2. Make it possible for drivers to set the "bit-flip tolerance
> threshold" (invent a better name please), which is lowest the bit-flip
> level which should be considered harmful. E.g., in your case, the
> threshold could be 2.

This kind of threshold is NAND-device specific, ideally it could be derived
from ONFI + manufacturer information, in a driver-independent way. Ideally...

> 3. Make UBI only react on bit-flips with order higher or equivalend to
> the threshold. In your case then, UBI would ignore all level 1 bit-flips
> and react only to level 2, 3, and 4 bit-flips.
 
Yes, this makes sense. When you say "react", do you also mean not doing any
scrubbing when the error count is below the threshold ?
In future devices, 1-bit errors will become very common and we'll probably need
to ignore them to avoid scrubbing blocks all the time.

BR,

Ivan

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Fri, 2011-02-25 at 15:44 +0100, Ivan Djelic wrote:
> On Fri, Feb 25, 2011 at 12:12:10PM +0000, Artem Bityutskiy wrote:
> > On Fri, 2011-02-25 at 12:36 +0100, Ivan Djelic wrote:
> (...)
> > > 
> > > The fact that a bitflip detected during torture is enough to decide that a
> > > block is bad causes problems on some 4-bit ecc devices we are using. If we
> > > stick to this policy, we end up with a _lot_ of blocks being marked as bad
> > > (i.e. way too many).
> > 
> > I see. May be in your case 1 bit errors are completely harmless, but 2
> > and 3 are not?
> 
> When a NAND device requires 4-bit ecc or more, you do see a lot of 1-bit errors
> (compared to previous NAND devices). They are not "completely harmless" because
> you are still supposed to relocate data in some other block and erase the block
> (those bitflips are reversible errors), in order to avoid error accumulation
> and stay below the specified ecc requirement. But they probably should not be
> considered an indication that the block has gone bad.

So basically, this means that UBI need to distinguish between "gentle
bit-flips" and "evil bit-flips". This could be done, again, but teaching
MTD to return the bit-flip level and provide us some thresholds, which
could either be set automatically from OFNI data or heuristics or could
be provided by the user via some MTD sysfs files.

This should not be huge amount of job.


> > > Our NAND manufacturer tells us that, as long as a block erase operation
> > > completes without a failure reported by the device, it should not be classified
> > > as bad, even if it has bitflips (which sounds risky at best).
> > 
> > For any amount of flipped bits per page? Sounds a bit scary.
> 
> I agree. Our NAND manufacturer even told us that a single permanent 1-bit
> failure in a block is not enough for marking this block as bad on 4-bit ecc NAND
> devices. I still think there should be a specified amount of errors above which
> the block should be considered bad. Maybe only permanent bit failures should
> be considered.

Yes, this sounds logical and gives system designers flexibility.

> Yes, this makes sense. When you say "react", do you also mean not doing any
> scrubbing when the error count is below the threshold ?

Yes. There may be separate thresholds for different things: one for
scrubbing, one for marking bad. Feel free to send patches :-)

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

On Fri, 2011-02-25 at 12:36 +0100, Ivan Djelic wrote:
> Right now, we implement a bitflip threshold, below which we correct ecc errors
> without reporting them. When the bitflip threshold is reached, we report the
> amount of corrected errors, triggering block scrubbing, etc.
> This is not ideal, but it prevents UBI from torturing and marking too many
> blocks as bad.

Hmm, what exactly the threshold you are implementing mean?

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Ivan Djelic]

On Fri, Feb 25, 2011 at 12:22:19PM +0000, Artem Bityutskiy wrote:
> On Fri, 2011-02-25 at 12:36 +0100, Ivan Djelic wrote:
> > Right now, we implement a bitflip threshold, below which we correct ecc errors
> > without reporting them. When the bitflip threshold is reached, we report the
> > amount of corrected errors, triggering block scrubbing, etc.
> > This is not ideal, but it prevents UBI from torturing and marking too many
> > blocks as bad.
> 
> Hmm, what exactly the threshold you are implementing mean?

When errors are detected in a subpage, we correct them. If the error count is
less than or equal to a threshold (in our case 2), we clear it (i.e. we forget
how many errors we just corrected). We accumulate those error counts into the
global page error count which we return (from function nand.ecc.correct()).
It's just a hack, waiting for a better solution.

Ivan

Re: CONFIG_MTD_NAND_VERIFY_WRITE with Software ECC

[Artem Bityutskiy]

Hi,

On Tue, 2011-02-15 at 14:00 +0000, David Peverley wrote:
> From the description of read disturb, it occurs due to many reads
> (hundreds of thousands or millions) prior to an erase. Currently my
> testing is using nandtestc.c and mtd_stresstest.ko - the former tests
> one cycle before re-programming and the latter is random but not
> expected to be more than tens of reads before a re-programme becomes
> statistically likely. Potentially program disturb sounds like it
> _could_ be the behaviour I observe but it's not clear.

If you verify the page just after you have programmed it, then program
disturb is out of the picture. AFAIK, program disturb is about
"disturbing" other NAND pages while programming.

> My general take on this is that only the permanent type failures i.e.
> those involving permanently stuck bits, require marking as bad blocks.
> The recovery recommended for the other scenarios is always to erase
> and re-programme. This potentially opens up a whole can of worms... My
> interpretation of this is that if we verify a write and we've had a
> (correctable and non-permanent) single bit error the Right Thing To Do
> would be to erase and re-programme the block, probably with a very
> small retry limit. We could argue that it's the responsibility of the
> file-system to do this but programatically I think nand_write_page()
> is best placed to be able to do this.

Yeah, UBI does this for example. If we program an eraseblock, and we get
and error while writing a NAND page, we try to recover:

1. We pick another eraseblock.
2. We move the data from the faulty eraseblock to the new one.
3. We erase and torture the faulty eraseblock.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)