Does UBI LEB-level access interlock happily with UBIfs access?

Does UBI LEB-level access interlock happily with UBIfs access?

[Atlant Schmidt]

Folks:

  We use the ordinary MTD/UBI/UBIfs stack on our
  Embedded Linux system.

  For the purposes of scrubbing-out single bit errors,
  I'd like to read through all of the LEBs stored in the
  UBI device and whenever the ECC information indicates
  that any correctable errors occurred, I'd like to
  *RE-WRITE* that LEB (thereby forcing it to be scrubbed).
  (Note: I might do this page-by-page rather than LEB-
  by-LEB.)

  But I would expect that because I'd have a hard
  (impossible?) time doing an atomic read/re-write of a
  LEB (or page), the UBIfs and my scrubber would interact
  badly with my scrubber eventually corrupting the UBIfs
  file system. Is there any easy way to interlock these
  accesses (from the UBIfs and from my UBI-level scrubber)?
  A way to temporarily suspend activity from the UBIfs?

  One kludge that might work is that I'm operating in a
  real-time environment. If I made my scrubbing requests
  from a very high priority (higher than the "System"
  tasks that run around Priority 50), could I be sure
  my read + rewrite scrubbing requests would at least
  enter the UBI's work queue immediately adjacent to
  each other (and without UBIfs requests intermingled)?

  Alternatively, I could probably dismount the UBIfs
  before doing scrubbing, but I'd rather not have to
  do that.
                          Atlant


This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Artem Bityutskiy]

On Tue, 2014-09-16 at 10:04 -0400, Atlant Schmidt wrote:
> Folks:
> 
>   We use the ordinary MTD/UBI/UBIfs stack on our
>   Embedded Linux system.
> 
>   For the purposes of scrubbing-out single bit errors,
>   I'd like to read through all of the LEBs stored in the
>   UBI device and whenever the ECC information indicates
>   that any correctable errors occurred, I'd like to
>   *RE-WRITE* that LEB (thereby forcing it to be scrubbed).
>   (Note: I might do this page-by-page rather than LEB-
>   by-LEB.)

Well, you may do something like

dd if=/dev/ubiX_Y of=/dev/null

For all volumes, and this will make UBI real all pages from all volumes
and whenever there was a bit-flip, schedule the LEB for scrubbing.

The volume table LEBs wont'd be read, though. To cover even that, you
one could introduce a special ioctl.

In case of fastmap, this ioctl would cover all the fatmap special LEBs
too.

>   But I would expect that because I'd have a hard
>   (impossible?) time doing an atomic read/re-write of a
>   LEB (or page), the UBIfs and my scrubber would interact
>   badly with my scrubber eventually corrupting the UBIfs
>   file system. Is there any easy way to interlock these
>   accesses (from the UBIfs and from my UBI-level scrubber)?
>   A way to temporarily suspend activity from the UBIfs?
> 
>   One kludge that might work is that I'm operating in a
>   real-time environment. If I made my scrubbing requests
>   from a very high priority (higher than the "System"
>   tasks that run around Priority 50), could I be sure
>   my read + rewrite scrubbing requests would at least
>   enter the UBI's work queue immediately adjacent to
>   each other (and without UBIfs requests intermingled)?
> 
>   Alternatively, I could probably dismount the UBIfs
>   before doing scrubbing, but I'd rather not have to
>   do that.

You do not need to do unmount anything with the above technique. If,
say, UBI scheduled an LEB 0:1 (volume 0, LEB 1) for scrubbing, and UBIFS
(which has volume 0 mounted) writes to LEB 1, UBI will block the writer
until the scrubbing finishes, so you do not have to worry.

HTH.

-- 
Best Regards,
Artem Bityutskiy

RE: Does UBI LEB-level access interlock happily with UBIfs access?

[Atlant Schmidt]

Artem:

  Thanks, but the key was in this line of my question:

> For the purposes of scrubbing-out *SINGLE BIT* errors,


  Single bit errors are below the threshold set by either
  current UBI software or newer NAND Flash chips that
  contain on-board ECC and set a "Rewrite recommended"
  Status Register bit.

  That's why I proposed doing the UBI-level LEB rewrites
  myself, any time even a single bit-flip was reported as
  being corrected.

  For software ECC done in the UBI code, I guess one
  strategy (as we discussed a few months back) is to
  modify the UBI code so it schedules a block for
  scrubbing even if a single-bit correctable error
  occurs.

  For ONFI Flash with hardware ECC, I'm not sure there
  is a solution; I don't see where the ONFI Flash reports
  "correctable error but not at the rewrite threshold";
  a read seems to either:

    o Succeed (with no or few corrections;
               no special report generated)

    o Succeed (with "Rewrite" being recommended)

    o Fail


  Maybe I'll ask Micron's tech-support folks if there's
  more info available than is made obvious in the
  datasheet.

                      Atlant

-----Original Message-----
From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
Sent: Friday, September 19, 2014 11:27 AM
To: Atlant Schmidt
Cc: linux-mtd@lists.infradead.org
Subject: Re: Does UBI LEB-level access interlock happily with UBIfs access?

On Tue, 2014-09-16 at 10:04 -0400, Atlant Schmidt wrote:
> Folks:
>
>   We use the ordinary MTD/UBI/UBIfs stack on our
>   Embedded Linux system.
>
>   For the purposes of scrubbing-out single bit errors,
>   I'd like to read through all of the LEBs stored in the
>   UBI device and whenever the ECC information indicates
>   that any correctable errors occurred, I'd like to
>   *RE-WRITE* that LEB (thereby forcing it to be scrubbed).
>   (Note: I might do this page-by-page rather than LEB-
>   by-LEB.)

Well, you may do something like

dd if=/dev/ubiX_Y of=/dev/null

For all volumes, and this will make UBI real all pages from all volumes
and whenever there was a bit-flip, schedule the LEB for scrubbing.

The volume table LEBs wont'd be read, though. To cover even that, you
one could introduce a special ioctl.

In case of fastmap, this ioctl would cover all the fatmap special LEBs
too.

>   But I would expect that because I'd have a hard
>   (impossible?) time doing an atomic read/re-write of a
>   LEB (or page), the UBIfs and my scrubber would interact
>   badly with my scrubber eventually corrupting the UBIfs
>   file system. Is there any easy way to interlock these
>   accesses (from the UBIfs and from my UBI-level scrubber)?
>   A way to temporarily suspend activity from the UBIfs?
>
>   One kludge that might work is that I'm operating in a
>   real-time environment. If I made my scrubbing requests
>   from a very high priority (higher than the "System"
>   tasks that run around Priority 50), could I be sure
>   my read + rewrite scrubbing requests would at least
>   enter the UBI's work queue immediately adjacent to
>   each other (and without UBIfs requests intermingled)?
>
>   Alternatively, I could probably dismount the UBIfs
>   before doing scrubbing, but I'd rather not have to
>   do that.

You do not need to do unmount anything with the above technique. If,
say, UBI scheduled an LEB 0:1 (volume 0, LEB 1) for scrubbing, and UBIFS
(which has volume 0 mounted) writes to LEB 1, UBI will block the writer
until the scrubbing finishes, so you do not have to worry.

HTH.

--
Best Regards,
Artem Bityutskiy


This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Brian Norris]

On Tue, Sep 16, 2014 at 10:04:29AM -0400, Atlant Schmidt wrote:
> This e-mail and the information, including any attachments it
> contains, are intended to be a confidential communication only to the
> person or entity to whom it is addressed and may contain information
> that is privileged. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited.
> If you have received this communication in error, please immediately
> notify the sender and destroy the original message.

Can you please consider dropping this notice from your email? It's
effectively useless when you're sending mail to a publicly-accessible
(and indefinitely archived) mailing list.

Thanks,
Brian

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Artem Bityutskiy]

On Fri, 2014-09-19 at 12:58 -0400, Atlant Schmidt wrote:
> Artem:
> 
>   Thanks, but the key was in this line of my question:
> 
> > For the purposes of scrubbing-out *SINGLE BIT* errors,
> 
> 
>   Single bit errors are below the threshold set by either
>   current UBI software or newer NAND Flash chips that
>   contain on-board ECC and set a "Rewrite recommended"
>   Status Register bit.

This is MTD hiding these bit-flips from UBI. The idea is that depending
on the flash/controller, you may want to scrub or ignore bit-flips lower
than certain level.

Indeed, for some NANDs 1-bit flips happen on nearly every read,
scrubbing them all is not feasible.

Sow what you say sounds like you want to lower the bit-flip handling
threshold.

To do this, you either need to amend your driver to have the default
threshold that you need, or change the
/sys/class/mtd/mtdX/bitflip_threshold value (do not remember for sure,
but I believe it is writable).

Please, refer to the documentation here:

Documentation/ABI/testing/sysfs-class-mtd


>   That's why I proposed doing the UBI-level LEB rewrites
>   myself, any time even a single bit-flip was reported as
>   being corrected.

Well, if you feel comfortable with this, go ahead, but without knowing
the particulars of your systems, this sounds like asking for troubles.

Indeed, changing something underneath the live volume manager (UBI) and
file-system (UBIFS) is error-prone at the very least.

Asking UBI to do this sounds a lot better.

But again, I do not know the specific of the system you are designing.

>   For software ECC done in the UBI code, I guess one
>   strategy (as we discussed a few months back) is to
>   modify the UBI code so it schedules a block for
>   scrubbing even if a single-bit correctable error
>   occurs.

All you need to do is to lower 'bitflip_threshold' instead.

But note, this will be "passive" scrubbing, meaning that UBI will scrub
only when there is a read operation. But the read operation may be
extremely rare for certain LEBs, and the data may bit-rot due to various
"radiation" effects (doing I/O on the neighbor PEBs).

So I suggested you to just read all volumes periodically from the
user-space, may be from a background cron task.

But as I pointed, this will not force re-read of the volume table LEBs.
To address this, you'd need to do some additional, not very difficult
work.

-- 
Best Regards,
Artem Bityutskiy

RE: Does UBI LEB-level access interlock happily with UBIfs access?

[Atlant Schmidt]

Artem:

  Thanks -- that's all consistent with my current thinking.

                            Atlant

-----Original Message-----
From: Artem Bityutskiy [mailto:dedekind1@gmail.com]
Sent: Friday, September 19, 2014 1:14 PM
To: Atlant Schmidt
Cc: linux-mtd@lists.infradead.org
Subject: Re: Does UBI LEB-level access interlock happily with UBIfs access?

On Fri, 2014-09-19 at 12:58 -0400, Atlant Schmidt wrote:
> Artem:
>
>   Thanks, but the key was in this line of my question:
>
> > For the purposes of scrubbing-out *SINGLE BIT* errors,
>
>
>   Single bit errors are below the threshold set by either
>   current UBI software or newer NAND Flash chips that
>   contain on-board ECC and set a "Rewrite recommended"
>   Status Register bit.

This is MTD hiding these bit-flips from UBI. The idea is that depending
on the flash/controller, you may want to scrub or ignore bit-flips lower
than certain level.

Indeed, for some NANDs 1-bit flips happen on nearly every read,
scrubbing them all is not feasible.

Sow what you say sounds like you want to lower the bit-flip handling
threshold.

To do this, you either need to amend your driver to have the default
threshold that you need, or change the
/sys/class/mtd/mtdX/bitflip_threshold value (do not remember for sure,
but I believe it is writable).

Please, refer to the documentation here:

Documentation/ABI/testing/sysfs-class-mtd


>   That's why I proposed doing the UBI-level LEB rewrites
>   myself, any time even a single bit-flip was reported as
>   being corrected.

Well, if you feel comfortable with this, go ahead, but without knowing
the particulars of your systems, this sounds like asking for troubles.

Indeed, changing something underneath the live volume manager (UBI) and
file-system (UBIFS) is error-prone at the very least.

Asking UBI to do this sounds a lot better.

But again, I do not know the specific of the system you are designing.

>   For software ECC done in the UBI code, I guess one
>   strategy (as we discussed a few months back) is to
>   modify the UBI code so it schedules a block for
>   scrubbing even if a single-bit correctable error
>   occurs.

All you need to do is to lower 'bitflip_threshold' instead.

But note, this will be "passive" scrubbing, meaning that UBI will scrub
only when there is a read operation. But the read operation may be
extremely rare for certain LEBs, and the data may bit-rot due to various
"radiation" effects (doing I/O on the neighbor PEBs).

So I suggested you to just read all volumes periodically from the
user-space, may be from a background cron task.

But as I pointed, this will not force re-read of the volume table LEBs.
To address this, you'd need to do some additional, not very difficult
work.

--
Best Regards,
Artem Bityutskiy


This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Artem Bityutskiy]

On Fri, 2014-09-19 at 13:17 -0400, Atlant Schmidt wrote:
> But as I pointed, this will not force re-read of the volume table LEBs.
> To address this, you'd need to do some additional, not very difficult
> work.

Oh, and this won't make UBI re-read the EC and VID headers, so they may
bit-rot too.

So indeed it sounds like UBI needs a separate interface for this kind of
"scrub all bit-flips" issues. I do not think it is hard to do - all the
mechanisms are already implemented, so this would mostly be about
inventing good API.


-- 
Best Regards,
Artem Bityutskiy

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Richard Weinberger]

On Fri, Sep 19, 2014 at 7:24 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> On Fri, 2014-09-19 at 13:17 -0400, Atlant Schmidt wrote:
>> But as I pointed, this will not force re-read of the volume table LEBs.
>> To address this, you'd need to do some additional, not very difficult
>> work.
>
> Oh, and this won't make UBI re-read the EC and VID headers, so they may
> bit-rot too.
>
> So indeed it sounds like UBI needs a separate interface for this kind of
> "scrub all bit-flips" issues. I do not think it is hard to do - all the
> mechanisms are already implemented, so this would mostly be about
> inventing good API.

We could implement a trivial knob to trigger such a check in kernel.
I.e. you trigger the check via an ioctl() or whatever and UBI schedules
such a read-check for every PEB into the UBI background thread.

I'd volunteer to implement this.

-- 
Thanks,
//richard

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Ricard Wanderlof]

On Sat, 20 Sep 2014, Richard Weinberger wrote:

> On Fri, Sep 19, 2014 at 7:24 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
>> On Fri, 2014-09-19 at 13:17 -0400, Atlant Schmidt wrote:
>>> But as I pointed, this will not force re-read of the volume table LEBs.
>>> To address this, you'd need to do some additional, not very difficult
>>> work.
>>
>> Oh, and this won't make UBI re-read the EC and VID headers, so they may
>> bit-rot too.
>>
>> So indeed it sounds like UBI needs a separate interface for this kind of
>> "scrub all bit-flips" issues. I do not think it is hard to do - all the
>> mechanisms are already implemented, so this would mostly be about
>> inventing good API.
>
> We could implement a trivial knob to trigger such a check in kernel.
> I.e. you trigger the check via an ioctl() or whatever and UBI schedules
> such a read-check for every PEB into the UBI background thread.

During the scanning operation that takes place when a partition is 
attached, doesn't this also trigger a check of all headers, as all the 
data needs to be read as part of the ubiattach process?

Not that that would be a practical solution for many systems where it is 
not practical to detach and re-attach, for instance the partition where 
the root volume is located, as that would in practice require a reboot.

> I'd volunteer to implement this.

I think it would be good if such a forced re-read could be set to happen 
automatically at a specified interval, say by default once a day.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Does UBI LEB-level access interlock happily with UBIfs access?

[Richard Weinberger]

Am 22.09.2014 10:34, schrieb Ricard Wanderlof:
> 
> On Sat, 20 Sep 2014, Richard Weinberger wrote:
> 
>> On Fri, Sep 19, 2014 at 7:24 PM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
>>> On Fri, 2014-09-19 at 13:17 -0400, Atlant Schmidt wrote:
>>>> But as I pointed, this will not force re-read of the volume table LEBs.
>>>> To address this, you'd need to do some additional, not very difficult
>>>> work.
>>>
>>> Oh, and this won't make UBI re-read the EC and VID headers, so they may
>>> bit-rot too.
>>>
>>> So indeed it sounds like UBI needs a separate interface for this kind of
>>> "scrub all bit-flips" issues. I do not think it is hard to do - all the
>>> mechanisms are already implemented, so this would mostly be about
>>> inventing good API.
>>
>> We could implement a trivial knob to trigger such a check in kernel.
>> I.e. you trigger the check via an ioctl() or whatever and UBI schedules
>> such a read-check for every PEB into the UBI background thread.
> 
> During the scanning operation that takes place when a partition is attached, doesn't this also trigger a check of all headers, as all the data needs to be read as part of the
> ubiattach process?

Only headers will be read.
And with fastmap enabled only very few of these headers are read.

> Not that that would be a practical solution for many systems where it is not practical to detach and re-attach, for instance the partition where the root volume is located, as that
> would in practice require a reboot.
> 
>> I'd volunteer to implement this.
> 
> I think it would be good if such a forced re-read could be set to happen automatically at a specified interval, say by default once a day.

cron can trigger that easily.

Thanks,
//richard