Get error -74 (ECC error) during ubiattach

Get error -74 (ECC error) during ubiattach

[t kevin]

Hi all,

I get below error message everytime when my box boots up.
It appears there is a ECC error. But such errors should be taken care
by UBI/UBIFS itself, right?
Is it because the background thread "ubi_bgt2d" has not started yet so
no one could do the data scrubbing during ubiattach stage?
Is it possible to do something like a fsck.ubifs to fix this issue?

The system can boot up correctly but this message is kind of terrifying.

Thanks a lot
Kevin

UBI: attaching mtd10 to ubi2
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI warning: ubi_io_read: error -74 (ECC error) while reading 512
bytes from PEB 404:2048, read only 512 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 512
bytes from PEB 404:2048, read only 512 bytes, retry
UBI warning: ubi_io_read: error -74 (ECC error) while reading 512
bytes from PEB 404:2048, read only 512 bytes, retry
UBI error: ubi_io_read: error -74 (ECC error) while reading 512 bytes
from PEB 404:2048, read 512 bytes
[<c0138160>] (unwind_backtrace+0x0/0xf4) from [<c02f43dc>]
(ubi_io_read+0x134/0x314)
[<c02f43dc>] (ubi_io_read+0x134/0x314) from [<c02f49cc>]
(ubi_io_read_vid_hdr+0x48/0x214)
[<c02f49cc>] (ubi_io_read_vid_hdr+0x48/0x214) from [<c02f8d48>]
(ubi_attach+0x1f0/0x1464)
[<c02f8d48>] (ubi_attach+0x1f0/0x1464) from [<c02ef284>]
(ubi_attach_mtd_dev+0x770/0xd04)
[<c02ef284>] (ubi_attach_mtd_dev+0x770/0xd04) from [<c02f081c>]
(ctrl_cdev_ioctl+0xc8/0x174)
[<c02f081c>] (ctrl_cdev_ioctl+0xc8/0x174) from [<c01cfd14>]
(do_vfs_ioctl+0x3e0/0x560)
[<c01cfd14>] (do_vfs_ioctl+0x3e0/0x560) from [<c01cfec8>] (sys_ioctl+0x34/0x64)
[<c01cfec8>] (sys_ioctl+0x34/0x64) from [<c0131aa0>] (ret_fast_syscall+0x0/0x30)
UBI: max. sequence number:       1599
UBI: attached mtd10 to ubi2
UBI: MTD device name:            "data"
UBI: MTD device size:            240 MiB
UBI: number of good PEBs:        1920
UBI: number of bad PEBs:         0
UBI: number of corrupted PEBs:   0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 1920
UBI: number of PEBs reserved for bad PEB handling: 76
UBI: max/mean erase counter: 7/1
UBI: image sequence number:  1940927810
UBI: background thread "ubi_bgt2d" started, PID 575

Re: Get error -74 (ECC error) during ubiattach

[Richard Weinberger]

On Fri, Dec 19, 2014 at 1:24 PM, t kevin <kevint324@gmail.com> wrote:
> Hi all,
>
> I get below error message everytime when my box boots up.
> It appears there is a ECC error. But such errors should be taken care
> by UBI/UBIFS itself, right?

No. If the MTD reports an uncorrectable ECC error UBI can do nothing.

> Is it because the background thread "ubi_bgt2d" has not started yet so
> no one could do the data scrubbing during ubiattach stage?

UBI does scrubbing if it faces bit-flips (correctable ECC errors).

> Is it possible to do something like a fsck.ubifs to fix this issue?

No. PEB 404 is bad.
Is it always the same block?

I tend to remove the dump_stack() in this error path. It is a bit overkill.

-- 
Thanks,
//richard

Re: Get error -74 (ECC error) during ubiattach

[Tanya Brokhman]

On 12/19/2014 2:24 PM, t kevin wrote:
> Hi all,
>
> I get below error message everytime when my box boots up.
> It appears there is a ECC error. But such errors should be taken care
> by UBI/UBIFS itself, right?
> Is it because the background thread "ubi_bgt2d" has not started yet so
> no one could do the data scrubbing during ubiattach stage?
> Is it possible to do something like a fsck.ubifs to fix this issue?
>
> The system can boot up correctly but this message is kind of terrifying.

Hi Kevin

Is it always the same block/offset? If so then Richard is right, the PEB 
probably got bad.
Is this first boot up after flashing the image? I encountered similar 
behavior on first boot up after flashing the image. What helped me is to 
generate the image with -F flag, or use a UBI-aware flasher (ubiformat 
for example) to flash it.


Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Get error -74 (ECC error) during ubiattach

[t kevin]

Hi Richard,

Please see inline comments.

2014-12-19 21:49 GMT+08:00 Richard Weinberger <richard.weinberger@gmail.com>:
> On Fri, Dec 19, 2014 at 1:24 PM, t kevin <kevint324@gmail.com> wrote:
>> Hi all,
>>
>> I get below error message everytime when my box boots up.
>> It appears there is a ECC error. But such errors should be taken care
>> by UBI/UBIFS itself, right?
>
> No. If the MTD reports an uncorrectable ECC error UBI can do nothing.
>

So for uncorrectable ECC error ( e.g., bad block) the only way out is
to do a fresh ubiformat?
For device like harddisk you get bad blocks all the time, if it is not
in the critical sector things can be fixed as much as possible by fsck
so I was looking for something like fsck.ubifs.

>> Is it because the background thread "ubi_bgt2d" has not started yet so
>> no one could do the data scrubbing during ubiattach stage?
>
> UBI does scrubbing if it faces bit-flips (correctable ECC errors).
>
>> Is it possible to do something like a fsck.ubifs to fix this issue?
>
> No. PEB 404 is bad.
> Is it always the same block?

Yes, it is always the same block. What got me confused is below
message telling bad PEBs is still 0.
UBI: number of good PEBs:        1920
UBI: number of bad PEBs:         0
UBI: number of corrupted PEBs:   0


>
> I tend to remove the dump_stack() in this error path. It is a bit overkill.
>
> --
> Thanks,
> //richard

Thanks a lot
Kevin

Re: Get error -74 (ECC error) during ubiattach

[t kevin]

Hi Tanya,

Thanks for the suggestion.
I create the UBIFS by ubiformat+ubimkvol. It was fine at the very
beginning and that error message pops out after a few erase cycles (<
10).
I'm wondering what can I do to fix the problem instead of erase
everything to make it go away.

Thanks
Kevin

2014-12-19 23:31 GMT+08:00 Tanya Brokhman <tlinder@codeaurora.org>:
> On 12/19/2014 2:24 PM, t kevin wrote:
>>
>> Hi all,
>>
>> I get below error message everytime when my box boots up.
>> It appears there is a ECC error. But such errors should be taken care
>> by UBI/UBIFS itself, right?
>> Is it because the background thread "ubi_bgt2d" has not started yet so
>> no one could do the data scrubbing during ubiattach stage?
>> Is it possible to do something like a fsck.ubifs to fix this issue?
>>
>> The system can boot up correctly but this message is kind of terrifying.
>
>
> Hi Kevin
>
> Is it always the same block/offset? If so then Richard is right, the PEB
> probably got bad.
> Is this first boot up after flashing the image? I encountered similar
> behavior on first boot up after flashing the image. What helped me is to
> generate the image with -F flag, or use a UBI-aware flasher (ubiformat for
> example) to flash it.
>
>
> Thanks,
> Tanya Brokhman
> --
> Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project

Re: Get error -74 (ECC error) during ubiattach

[Richard Weinberger]

Hi!

Am 20.12.2014 um 04:43 schrieb t kevin:
> So for uncorrectable ECC error ( e.g., bad block) the only way out is
> to do a fresh ubiformat?
> For device like harddisk you get bad blocks all the time, if it is not
> in the critical sector things can be fixed as much as possible by fsck
> so I was looking for something like fsck.ubifs.

An uncorrectable ECC error means that more bits flipped than your ECC
algorithm can fix.

All you can do is trying to erase the block and hoping that it will
work again.

>>> Is it because the background thread "ubi_bgt2d" has not started yet so
>>> no one could do the data scrubbing during ubiattach stage?
>>
>> UBI does scrubbing if it faces bit-flips (correctable ECC errors).
>>
>>> Is it possible to do something like a fsck.ubifs to fix this issue?
>>
>> No. PEB 404 is bad.
>> Is it always the same block?
> 
> Yes, it is always the same block. What got me confused is below
> message telling bad PEBs is still 0.
> UBI: number of good PEBs:        1920
> UBI: number of bad PEBs:         0
> UBI: number of corrupted PEBs:   0

I agree that this is a bit strange.
UBI treats a block as bad if it has a bad block marker or it detects
so while scrubbing.

But it can also happen that you get a block with bad ECC if a powercut happened
while erasing it.
This is why UBI does not automatically mark it as bad while scanning.
Such a block will be erased.

Thanks,
//richard

Re: Get error -74 (ECC error) during ubiattach

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]

Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
> Hi!
> 
> Am 20.12.2014 um 04:43 schrieb t kevin:
>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>> to do a fresh ubiformat?
>> For device like harddisk you get bad blocks all the time, if it is not
>> in the critical sector things can be fixed as much as possible by fsck
>> so I was looking for something like fsck.ubifs.
> 
> An uncorrectable ECC error means that more bits flipped than your ECC
> algorithm can fix.

At same time uncorrectable ECC error mean, the page was erase and there
is no ECC sum. Something should be written to create ECC.

> All you can do is trying to erase the block and hoping that it will
> work again.
> 
>>>> Is it because the background thread "ubi_bgt2d" has not started yet so
>>>> no one could do the data scrubbing during ubiattach stage?
>>>
>>> UBI does scrubbing if it faces bit-flips (correctable ECC errors).
>>>
>>>> Is it possible to do something like a fsck.ubifs to fix this issue?
>>>
>>> No. PEB 404 is bad.
>>> Is it always the same block?
>>
>> Yes, it is always the same block. What got me confused is below
>> message telling bad PEBs is still 0.
>> UBI: number of good PEBs:        1920
>> UBI: number of bad PEBs:         0
>> UBI: number of corrupted PEBs:   0
> 
> I agree that this is a bit strange.
> UBI treats a block as bad if it has a bad block marker or it detects
> so while scrubbing.
> 
> But it can also happen that you get a block with bad ECC if a powercut happened
> while erasing it.
> This is why UBI does not automatically mark it as bad while scanning.
> Such a block will be erased.
> 
> Thanks,
> //richard
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 


-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Get error -74 (ECC error) during ubiattach

[Richard Weinberger]

Am 20.12.2014 um 11:00 schrieb Oleksij Rempel:
> Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
>> Hi!
>>
>> Am 20.12.2014 um 04:43 schrieb t kevin:
>>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>>> to do a fresh ubiformat?
>>> For device like harddisk you get bad blocks all the time, if it is not
>>> in the critical sector things can be fixed as much as possible by fsck
>>> so I was looking for something like fsck.ubifs.
>>
>> An uncorrectable ECC error means that more bits flipped than your ECC
>> algorithm can fix.
> 
> At same time uncorrectable ECC error mean, the page was erase and there
> is no ECC sum. Something should be written to create ECC.

Which would be a driver bug and needs fixing.

Thanks,
//richard

Re: Get error -74 (ECC error) during ubiattach

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

Am 20.12.2014 um 11:02 schrieb Richard Weinberger:
> Am 20.12.2014 um 11:00 schrieb Oleksij Rempel:
>> Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
>>> Hi!
>>>
>>> Am 20.12.2014 um 04:43 schrieb t kevin:
>>>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>>>> to do a fresh ubiformat?
>>>> For device like harddisk you get bad blocks all the time, if it is not
>>>> in the critical sector things can be fixed as much as possible by fsck
>>>> so I was looking for something like fsck.ubifs.
>>>
>>> An uncorrectable ECC error means that more bits flipped than your ECC
>>> algorithm can fix.
>>
>> At same time uncorrectable ECC error mean, the page was erase and there
>> is no ECC sum. Something should be written to create ECC.
> 
> Which would be a driver bug and needs fixing.

No at all. If driver get request to erase page, it should do it and
nothing more.
There are use cases where erase means erase, and not erase + write ecc.
It is not a driver bug!

-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Get error -74 (ECC error) during ubiattach

[Richard Weinberger]

Am 20.12.2014 um 11:05 schrieb Oleksij Rempel:
> Am 20.12.2014 um 11:02 schrieb Richard Weinberger:
>> Am 20.12.2014 um 11:00 schrieb Oleksij Rempel:
>>> Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
>>>> Hi!
>>>>
>>>> Am 20.12.2014 um 04:43 schrieb t kevin:
>>>>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>>>>> to do a fresh ubiformat?
>>>>> For device like harddisk you get bad blocks all the time, if it is not
>>>>> in the critical sector things can be fixed as much as possible by fsck
>>>>> so I was looking for something like fsck.ubifs.
>>>>
>>>> An uncorrectable ECC error means that more bits flipped than your ECC
>>>> algorithm can fix.
>>>
>>> At same time uncorrectable ECC error mean, the page was erase and there
>>> is no ECC sum. Something should be written to create ECC.
>>
>> Which would be a driver bug and needs fixing.
> 
> No at all. If driver get request to erase page, it should do it and
> nothing more.
> There are use cases where erase means erase, and not erase + write ecc.

Raw mode is a different thing.

> It is not a driver bug!

If users on top of MTD read from an erased block it has to return 0xFF bytes and
not an ECC error.
IOW if mtd_erase() -> mtd_read() causes an ECC error with your driver it needs
fixing.

Thanks,
//richard

Re: Get error -74 (ECC error) during ubiattach

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 1692 bytes --]

Am 20.12.2014 um 11:12 schrieb Richard Weinberger:
> Am 20.12.2014 um 11:05 schrieb Oleksij Rempel:
>> Am 20.12.2014 um 11:02 schrieb Richard Weinberger:
>>> Am 20.12.2014 um 11:00 schrieb Oleksij Rempel:
>>>> Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
>>>>> Hi!
>>>>>
>>>>> Am 20.12.2014 um 04:43 schrieb t kevin:
>>>>>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>>>>>> to do a fresh ubiformat?
>>>>>> For device like harddisk you get bad blocks all the time, if it is not
>>>>>> in the critical sector things can be fixed as much as possible by fsck
>>>>>> so I was looking for something like fsck.ubifs.
>>>>>
>>>>> An uncorrectable ECC error means that more bits flipped than your ECC
>>>>> algorithm can fix.
>>>>
>>>> At same time uncorrectable ECC error mean, the page was erase and there
>>>> is no ECC sum. Something should be written to create ECC.
>>>
>>> Which would be a driver bug and needs fixing.
>>
>> No at all. If driver get request to erase page, it should do it and
>> nothing more.
>> There are use cases where erase means erase, and not erase + write ecc.
> 
> Raw mode is a different thing.

There is no raw erase mode in this api.

>> It is not a driver bug!
> 
> If users on top of MTD read from an erased block it has to return 0xFF bytes and
> not an ECC error.
> IOW if mtd_erase() -> mtd_read() causes an ECC error with your driver it needs
> fixing.


check out this code:
https://github.com/olerem/linux-2.6/blob/uparm_9260-2014.12.17.1/drivers/mtd/nand/nand_base.c#L2686

This commands passed to flash chip. There is nothing about erase and
write ecc.

-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: Get error -74 (ECC error) during ubiattach

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 2293 bytes --]

Am 20.12.2014 um 11:16 schrieb Oleksij Rempel:
> Am 20.12.2014 um 11:12 schrieb Richard Weinberger:
>> Am 20.12.2014 um 11:05 schrieb Oleksij Rempel:
>>> Am 20.12.2014 um 11:02 schrieb Richard Weinberger:
>>>> Am 20.12.2014 um 11:00 schrieb Oleksij Rempel:
>>>>> Am 20.12.2014 um 10:27 schrieb Richard Weinberger:
>>>>>> Hi!
>>>>>>
>>>>>> Am 20.12.2014 um 04:43 schrieb t kevin:
>>>>>>> So for uncorrectable ECC error ( e.g., bad block) the only way out is
>>>>>>> to do a fresh ubiformat?
>>>>>>> For device like harddisk you get bad blocks all the time, if it is not
>>>>>>> in the critical sector things can be fixed as much as possible by fsck
>>>>>>> so I was looking for something like fsck.ubifs.
>>>>>>
>>>>>> An uncorrectable ECC error means that more bits flipped than your ECC
>>>>>> algorithm can fix.
>>>>>
>>>>> At same time uncorrectable ECC error mean, the page was erase and there
>>>>> is no ECC sum. Something should be written to create ECC.
>>>>
>>>> Which would be a driver bug and needs fixing.
>>>
>>> No at all. If driver get request to erase page, it should do it and
>>> nothing more.
>>> There are use cases where erase means erase, and not erase + write ecc.
>>
>> Raw mode is a different thing.
> 
> There is no raw erase mode in this api.
> 
>>> It is not a driver bug!
>>
>> If users on top of MTD read from an erased block it has to return 0xFF bytes and
>> not an ECC error.
>> IOW if mtd_erase() -> mtd_read() causes an ECC error with your driver it needs
>> fixing.
> 
> 
> check out this code:
> https://github.com/olerem/linux-2.6/blob/uparm_9260-2014.12.17.1/drivers/mtd/nand/nand_base.c#L2686
> 
> This commands passed to flash chip. There is nothing about erase and
> write ecc.

After long IRC discussion, i will agree that it is hw issue:
not all nand controllers use ECC algorithm with ECC=0xFF on empty page.
In this case ECC of page written full of 0xff != ECC of erased page. For
this case, on ECC error, OOB should be read and tested if (OOB=all(0xff).

May be it is more then one controller with this issue, and this case
should be handled by nand_base. If it is only single controller, it can
be solved in driver.
Other way to solve it is to use SW_ECC mode.


-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]