Unstable bits and JFFS2

Unstable bits and JFFS2

[Matej Kupljen]

To all,

during my experiments with the UBIFS crashing, I believe that I am experiencing
the "unstable bit" issue.

Would another FS, like JFFS2 work on such device, or I'd have the problems on
such device with all file systems?

Thank you and Best Regards,
Matej

Re: Unstable bits and JFFS2

[Brian Norris]

Hi Matej,

On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
> during my experiments with the UBIFS crashing, I believe that I am experiencing
> the "unstable bit" issue.
>
> Would another FS, like JFFS2 work on such device, or I'd have the problems on
> such device with all file systems?

I doubt it. JFFS2 is not supported much anymore, and it is not
designed for some of the problems with modern NAND flash.

Have you read the information here?

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits

I'm not much of an export on UBIFS or unstable bits, but I think that
there are some unsolved problems. However, before ruling this an
unstable bits problem, it's a good idea to rule out any other possible
issues.

Brian

Re: Unstable bits and JFFS2

[Matej Kupljen]

Brian,

> On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
>> during my experiments with the UBIFS crashing, I believe that I am experiencing
>> the "unstable bit" issue.
>>
>> Would another FS, like JFFS2 work on such device, or I'd have the problems on
>> such device with all file systems?
>
> I doubt it. JFFS2 is not supported much anymore, and it is not
> designed for some of the problems with modern NAND flash.

Aha, I see.
Thank you for the information.

> Have you read the information here?
>
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits

Yes, I have. Artem already pointed me to that FAQ.

> I'm not much of an export on UBIFS or unstable bits, but I think that
> there are some unsolved problems. However, before ruling this an
> unstable bits problem, it's a good idea to rule out any other possible
> issues.

Yes, I'll check the MTD driver.

At the moment I am running again integrity check on my UBIFS for
almost 24 hours
now, but until now no error:
# ./integck -n 0 -v -e /media/card

The only difference from previous run, where I saw one bit flip is
that previously
I run the test with power failure option set to ON. Now it is off.

Artem, you said that this unstable bits only happen during power cuts,
is this right?
Would those appear also on simulated power cuts, the ones that can
integck produce?

As for the ECC correction of the empty space, nothing is mentioned in
the manual.
Maybe I should ask the manufacturer of the chip (NXP LPC3152)??
It uses HW error correction included in the FLASH controller, with:
"The error correction code used is Reed-Solomon over GF(2^9). The
primitive polynomial
g(x) over GF(2) is:
g(x) = x9 + x4 +1"

Thank you and Best Regards,
Matej

RE: Unstable bits and JFFS2

[Atlant Schmidt]

Matej:

> Artem, you said that this unstable bits only happen
> during power cuts, is this right? Would those appear
> also on simulated power cuts, the ones that integck
> can produce?

  (Note: This isn't Artem replying.)

  Unstable bits can happen anytime, but *REAL* power-cuts
  while writing are certainly a great way to produce them;
  after all, if you've only transferred half the electrons
  to the floating gate when power goes away, well, that
  Flash cell is now a roll-of-the-quantum-dice each time
  you read it.

  Simulated power cuts (that just stop the software
  processing at arbitrary and random points) can't
  produce this effect.

  But any time a read-disturb or write-disturb takes place,
  there's some probability that a Flash cell will be left
  with a "near-threshold" charge on the floating gate, so
  unstable bits are a fact of life that must be faced by
  any software that drives NAND Flash memory chips. This
  is, of course, especially true of MLC chips and even
  more-true for TLC chips (with three bits per cell).

                           Atlant

-----Original Message-----
From: linux-mtd-bounces@lists.infradead.org [mailto:linux-mtd-bounces@lists.infradead.org] On Behalf Of Matej Kupljen
Sent: Wednesday, April 04, 2012 05:17
To: Brian Norris
Cc: linux-mtd
Subject: Re: Unstable bits and JFFS2

Brian,

> On Wed, Mar 28, 2012 at 4:16 AM, Matej Kupljen <matej.kupljen@gmail.com> wrote:
>> during my experiments with the UBIFS crashing, I believe that I am experiencing
>> the "unstable bit" issue.
>>
>> Would another FS, like JFFS2 work on such device, or I'd have the problems on
>> such device with all file systems?
>
> I doubt it. JFFS2 is not supported much anymore, and it is not
> designed for some of the problems with modern NAND flash.

Aha, I see.
Thank you for the information.

> Have you read the information here?
>
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_unstable_bits

Yes, I have. Artem already pointed me to that FAQ.

> I'm not much of an export on UBIFS or unstable bits, but I think that
> there are some unsolved problems. However, before ruling this an
> unstable bits problem, it's a good idea to rule out any other possible
> issues.

Yes, I'll check the MTD driver.

At the moment I am running again integrity check on my UBIFS for
almost 24 hours
now, but until now no error:
# ./integck -n 0 -v -e /media/card

The only difference from previous run, where I saw one bit flip is
that previously
I run the test with power failure option set to ON. Now it is off.

Artem, you said that this unstable bits only happen during power cuts,
is this right?
Would those appear also on simulated power cuts, the ones that can
integck produce?

As for the ECC correction of the empty space, nothing is mentioned in
the manual.
Maybe I should ask the manufacturer of the chip (NXP LPC3152)??
It uses HW error correction included in the FLASH controller, with:
"The error correction code used is Reed-Solomon over GF(2^9). The
primitive polynomial
g(x) over GF(2) is:
g(x) = x9 + x4 +1"

Thank you and Best Regards,
Matej

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/


 Click https://www.mailcontrol.com/sr/kqBJlvK+JBjTndxI!oX7Ulvwl1GqFAPrim4OaE2i4BWjrlQi2a8dR8ju1tDr2izUVJQZBBFsBrWtoBNtFUgrMg==  to report this email as spam.

This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

Re: Unstable bits and JFFS2

[Matej Kupljen]

To all,

>> Artem, you said that this unstable bits only happen
>> during power cuts, is this right? Would those appear
>> also on simulated power cuts, the ones that integck
>> can produce?

I've been trying to find the issue with my device over last week and for now,
I didn't find anything.

Here is what I've done:
Please NOTE, that I have used THE SAME device for testing, which showed
the problems in the past.
1) I've prepared SD card and booted the device from it. Then I run the
integchk without
    simulating power cuts. The test run form more then 96 hours (four
days and nights)
    and nothing happened. I mean, no errors were reported.
2) I made a simple "power cut simulator", which cuts the power to the
device. The device
    was booted from SD card, then the UBIFS partition on NAND Flash was mounted.
    The test file was removed, and then a new file was created with
simple dd from
    /dev/zero to the test file on UBIFS.
    The test run for 72 hours and again, nothing happened.
3) Now I am running the test, with power cut simulator, bit this time
the device is booted
    from UBIFS. It has been running for more then 12h at the moment,
and for now it
    is still working as expected.

The only real thing from previous tests is that now, I have formatted
the UBIFS from
the device, when it was booted from SD card and Linux was up and running.
Previous tests were done (when I've noticed the problems) only when U-Boot was
running and the ubinized image was flashed to Flash trough U-Boot.

Could this be the problem?
Maybe different timings or something in U-Boot compared to Linux?

Thank you and Best Regards,
Matej Kupljen

RE: Unstable bits and JFFS2

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

On Wed, 2012-04-04 at 06:54 -0400, Atlant Schmidt wrote:
> Matej:
> 
> > Artem, you said that this unstable bits only happen
> > during power cuts, is this right? Would those appear
> > also on simulated power cuts, the ones that integck
> > can produce?
> 
>   (Note: This isn't Artem replying.)
> 
>   Unstable bits can happen anytime, but *REAL* power-cuts
>   while writing are certainly a great way to produce them;
>   after all, if you've only transferred half the electrons
>   to the floating gate when power goes away, well, that
>   Flash cell is now a roll-of-the-quantum-dice each time
>   you read it.
> 
>   Simulated power cuts (that just stop the software
>   processing at arbitrary and random points) can't
>   produce this effect.
> 
>   But any time a read-disturb or write-disturb takes place,
>   there's some probability that a Flash cell will be left
>   with a "near-threshold" charge on the floating gate, so
>   unstable bits are a fact of life that must be faced by
>   any software that drives NAND Flash memory chips. This
>   is, of course, especially true of MLC chips and even
>   more-true for TLC chips (with three bits per cell).

Yeah, thanks for correcting. Yeah, read/write-disturb may make bits to
become unstable, but we assume this is a slow process which will
gradually make more and more bits flip and ECC will take care of that.
So I think Matej can exclude this.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

RE: Unstable bits and JFFS2

[Atlant Schmidt]

Artem:

> Yeah, thanks for correcting. Yeah, read/write-disturb may make bits
> to become unstable, but we assume this is a slow process which will
> gradually make more and more bits flip and ECC will take care of
> that. So I think Matej can exclude this.

  Unfortunately, ECC can only fix those PEBs that
  are actually read. If one of the PEBs in your
  filesystem is being used entirely to contain
  obscure data that only gets read once in a
  blue moon (i.e., very rarely), then multiple
  read-/write-disturbs can hit accumulate in
  that PEB and when it is finally read, it may
  already contains too many errors to be
  corrected by the ECC.

  We've seen this exact scenario occur with the
  current UBI/UBIfs and will be implementing a
  userland "scrubbing" system to ensure that
  every PEB gets read at least once in a while
  (e.g., weekly or whatever interval seems right).

                              Atlant

-----Original Message-----
From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
Sent: Friday, April 13, 2012 12:28
To: Atlant Schmidt
Cc: 'Matej Kupljen'; Brian Norris; linux-mtd
Subject: RE: Unstable bits and JFFS2

On Wed, 2012-04-04 at 06:54 -0400, Atlant Schmidt wrote:
> Matej:
>
> > Artem, you said that this unstable bits only happen
> > during power cuts, is this right? Would those appear
> > also on simulated power cuts, the ones that integck
> > can produce?
>
>   (Note: This isn't Artem replying.)
>
>   Unstable bits can happen anytime, but *REAL* power-cuts
>   while writing are certainly a great way to produce them;
>   after all, if you've only transferred half the electrons
>   to the floating gate when power goes away, well, that
>   Flash cell is now a roll-of-the-quantum-dice each time
>   you read it.
>
>   Simulated power cuts (that just stop the software
>   processing at arbitrary and random points) can't
>   produce this effect.
>
>   But any time a read-disturb or write-disturb takes place,
>   there's some probability that a Flash cell will be left
>   with a "near-threshold" charge on the floating gate, so
>   unstable bits are a fact of life that must be faced by
>   any software that drives NAND Flash memory chips. This
>   is, of course, especially true of MLC chips and even
>   more-true for TLC chips (with three bits per cell).

Yeah, thanks for correcting. Yeah, read/write-disturb may make bits to
become unstable, but we assume this is a slow process which will
gradually make more and more bits flip and ECC will take care of that.
So I think Matej can exclude this.

--
Best Regards,
Artem Bityutskiy

This e-mail and the information, including any attachments, it contains are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

RE: Unstable bits and JFFS2

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

On Fri, 2012-04-13 at 12:40 -0400, Atlant Schmidt wrote:
> Artem:
> 
> > Yeah, thanks for correcting. Yeah, read/write-disturb may make bits
> > to become unstable, but we assume this is a slow process which will
> > gradually make more and more bits flip and ECC will take care of
> > that. So I think Matej can exclude this.
> 
>   Unfortunately, ECC can only fix those PEBs that
>   are actually read. If one of the PEBs in your
>   filesystem is being used entirely to contain
>   obscure data that only gets read once in a
>   blue moon (i.e., very rarely), then multiple
>   read-/write-disturbs can hit accumulate in
>   that PEB and when it is finally read, it may
>   already contains too many errors to be
>   corrected by the ECC.
> 
>   We've seen this exact scenario occur with the
>   current UBI/UBIfs and will be implementing a
>   userland "scrubbing" system to ensure that
>   every PEB gets read at least once in a while
>   (e.g., weekly or whatever interval seems right).

This is right, people should have a user-space app which periodically
reads all /dev/ubiX_Y. Care to send a patch against mtd-www with some
explantion of this issue and extend the unstable bits issue?

But in Matej's case I do not believe he is testing long enough to hit
this scenario. But everything is possible of course.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

RE: Unstable bits and JFFS2

[Ricard Wanderlof]

On Fri, 13 Apr 2012, Artem Bityutskiy wrote:

> On Fri, 2012-04-13 at 12:40 -0400, Atlant Schmidt wrote:
>>
>>   Unfortunately, ECC can only fix those PEBs that
>>   are actually read. If one of the PEBs in your
>>   filesystem is being used entirely to contain
>>   obscure data that only gets read once in a
>>   blue moon (i.e., very rarely), then multiple
>>   read-/write-disturbs can hit accumulate in
>>   that PEB and when it is finally read, it may
>>   already contains too many errors to be
>>   corrected by the ECC.
>>
>>   We've seen this exact scenario occur with the
>>   current UBI/UBIfs and will be implementing a
>>   userland "scrubbing" system to ensure that
>>   every PEB gets read at least once in a while
>>   (e.g., weekly or whatever interval seems right).
>
> This is right, people should have a user-space app which periodically
> reads all /dev/ubiX_Y. Care to send a patch against mtd-www with some
> explantion of this issue and extend the unstable bits issue?

Would it not be preferable to have this functionality within UBI itself, 
so that, for instance governed by some sysfs parameter, UBI reads through 
all PEBs at some interval, triggering scrubbing of any blocks that require 
too much ECC for comfort?

To me it seems something that userspace should not be concerned with, 
given the rest of the features that UBI deals with so that userspace does 
not have to.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: Unstable bits and JFFS2

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On Sat, 2012-04-14 at 06:21 +0200, Ricard Wanderlof wrote:
> To me it seems something that userspace should not be concerned with, 
> given the rest of the features that UBI deals with so that userspace does 
> not have to.

Yes, may probably you are right. Now we just need the brave knight who'd
send the patch.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Unstable bits and JFFS2

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]

Hi,

> > > Artem, you said that this unstable bits only happen
> > > during power cuts, is this right? Would those appear
> > > also on simulated power cuts, the ones that integck
> > > can produce?

A bit of hijacking this thread, yet it fits somehow to the idea I got
yesterday:

There is the Open Project Proposal again, where the CEWG funds topics which
improve embedded linux in general [1]. I wonder if it makes sense to hand in a
proposal starting to tackle the 'unstable bits' issue. I could imagine task 1
mentioned in the wiki page (1. Improve the existing power cut emulation
infrastructure in UBIFS and start emulating unstable bits. Start with emulating
only one type of unstable bits, e.g., type 1.) could be a good candidate for
such a proposal. Since I am not deeply involved with this topic:

1) Is this task still needed?
2) Is it a good task for a 2-3 week period (I'd think so)
3) Are there people around up for the task?

I'd be willing to write the proposal, if nobody else volunteers. (yet, given my
constraints, I am not the right person to actually work on it). Even if the
proposal doesn't get contracted, the topic itself might get some attention
since it will be discussed in the CEWG.

What do you think?

Regards,

   Wolfram

[1] http://elinux.org/CEWG_Open_Project_Proposal_2012

-- 
Pengutronix e.K.                           | Wolfram Sang                |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Unstable bits and JFFS2

[Artem Bityutskiy]

On Fri, 2012-04-13 at 20:36 +0200, Wolfram Sang wrote:
> Hi,
> 
> > > > Artem, you said that this unstable bits only happen
> > > > during power cuts, is this right? Would those appear
> > > > also on simulated power cuts, the ones that integck
> > > > can produce?
> 
> A bit of hijacking this thread, yet it fits somehow to the idea I got
> yesterday:
> 
> There is the Open Project Proposal again, where the CEWG funds topics which
> improve embedded linux in general [1]. I wonder if it makes sense to hand in a
> proposal starting to tackle the 'unstable bits' issue. I could imagine task 1
> mentioned in the wiki page (1. Improve the existing power cut emulation
> infrastructure in UBIFS and start emulating unstable bits. Start with emulating
> only one type of unstable bits, e.g., type 1.) could be a good candidate for
> such a proposal. Since I am not deeply involved with this topic:
> 
> 1) Is this task still needed?
> 2) Is it a good task for a 2-3 week period (I'd think so)
> 3) Are there people around up for the task?
> 
> I'd be willing to write the proposal, if nobody else volunteers. (yet, given my
> constraints, I am not the right person to actually work on it). Even if the
> proposal doesn't get contracted, the topic itself might get some attention
> since it will be discussed in the CEWG.

Yes, definitely a hot topic now, I am swamped with lots of things and
never found time for writing proposals like this. Please, go ahead,
would be very appreciated. I might help reviewing it and amending.