[U-Boot] UBI fixable bit-flip issue

[Richard Weinberger <richard@nod.at>]

Mark,

Am Donnerstag, 12. Juli 2018, 07:22:13 CEST schrieb Heiko Schocher:
> Hello Mark,
> 
> added Richard Weinberger to cc...
> 
> Am 12.07.2018 um 02:28 schrieb Mark Spieth:
> > Hi
> > 
> > In the process of investigating a boot failure on one of our devices, the
> > 
> > UBI: fixable bit-flip detected at PEB
> > 
> > message was seen with the following behaviour during kernel load in u-boot.
> > 
> > Read [2285568] bytes
> > UBI: fixable bit-flip detected at PEB 415
> > UBI: schedule PEB 415 for scrubbing
> > UBI: fixable bit-flip detected at PEB 415
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: schedule PEB 419 for scrubbing
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: schedule PEB 420 for scrubbing
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > UBI: fixable bit-flip detected at PEB 420
> > UBI: fixable bit-flip detected at PEB 419
> > 
> > This repeats until reset.

Do you see the same symptom also on Linux?
We need to be very sure that it is actually a UBI problem.

> > This fix is not a root cause fix though. Investigating further led to the following root cause 
> > solution. The following is AFAICT.
> > 
> > When the scrubber chooses a PEB to move the from the free balanced tree. This tree is sorted by EC 
> > (erase count) and then by PEB number.
> > 
> > The find_wl_entry call uses a max parameter of WL_FREE_MAX_DIFF which is 8192 in this config. So the 
> > find_wl_entry function will find a PEB that is better in error count that the current PEB EC. This

error count? You mean erase count?
 
> > can easily cause it to find the PEB that was just moved from if it is the lowest numbered PEB in the 
> > free tree. Waiting for EC to go above 8192 would take a long time and cause premature aging of the 
> > flash PEBs in question.
> > 
> > The easy solution is to change the max parameter to this call to 0 so it finds a PEB with a smaller 
> > EC than the one being replaced. This means it wont use the previously discarded PEB as its first 
> > choice.

For scrubbing this might be a good idea, but not for regular wear-leveling.

See comment in UBI:
/*
 * When a physical eraseblock is moved, the WL sub-system has to pick the target
 * physical eraseblock to move to. The simplest way would be just to pick the
 * one with the highest erase counter. But in certain workloads this could lead
 * to an unlimited wear of one or few physical eraseblock. Indeed, imagine a
 * situation when the picked physical eraseblock is constantly erased after the
 * data is written to it. So, we have a constant which limits the highest erase
 * counter of the free physical eraseblock to pick. Namely, the WL sub-system
 * does not pick eraseblocks with erase counter greater than the lowest erase
 * counter plus %WL_FREE_MAX_DIFF.
 */
#define WL_FREE_MAX_DIFF (2*UBI_WL_THRESHOLD)

So we could change the logic such that for regular wear-leveling we keep using WL_FREE_MAX_DIFF,
but for scrubbing (which is 1:1 wear-leveling but the source PEB is showing bit-flips) we use
a lower value. IMHO WL_FREE_MAX_DIFF/2 would be a good choice.
I'm not sure whether 0 is too extreme and might cause other distortions.

Mark, can you please file a patch and send it to linux-mtd mailing list?
Such a change needs to go through Linux and then to u-boot.
But first we need to think about and discuss it in detail.
 
>   I am not sure if it is so easy ...
>
> > This fix was implemented and fixable bit-flip errors no longer hang/freeze the boot process! UBI 
> > erase and reformat was used between re-tests to get consistent results.
> > 
> > Adding the above 75% correctable bitflip threshold is also a good thing as less movement will ensue 
> > when the FLASH is new, but as the flash ages, the root cause will once again be invoked causing 
> > un-recoverable boot failures.
> > 
> > Note this fault is also in the latest kernel drivers for UBI and may also exist in other wear 
> > leveling implementations. The kernel driver issue may be at fault for android devices locking 
> > up/freezing sporadically during FLASH read when scrubbing due to a relatively full flash and 
> > correctable errors causing ping pong PEB moves.
> > 
> > The question is, is my root cause solution sound or have I missed something?
> 
> I have to think about, before I write nonsene, but may Richard has
> here a deeper insight.

Please see my comments. :)

Thanks,
//richard