Re: ubifs_recover_leb() failure

Re: ubifs_recover_leb() failure

[Richard Weinberger]

Hi!

Am 25.06.2015 um 15:40 schrieb Brokhman, Tanya:
> Hi Richard et. al,
> 
> We recently encountered the following error messages:
> 
> UBIFS-0 error (pid 1): ubifs_recover_leb: corrupt empty space LEB 159:32768, corruption starts at 3810
> 
> UBIFS-0 error (pid 1): ubifs_scanned_corruption: corruption at LEB 159:36578
> 
> UBIFS-0 error (pid 1): ubifs_scanned_corruption: first 8192 bytes from LEB 159:36578
> 
> 00000000: fffffffe ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
> 
> 00000020: ffffffff ffffffff ffffffff ffffffff ffffffff fffffffd ffffffff ffffffff  ................................
> 
> 
> The test scenario is simple: power cut while writing.
> We run this in a loop and the issue is not easily reproducible but the failure is constant.
> 
> We're working with a 3.10 based kernel, fastmap enabled. I saw that you released a lot of fixes in fastmap (~46).
> I picked up these below 2 as they seemed to resolve a similar issue:
> 1. UBI: Fastmap: Fix race in ubi_eba_atomic_leb_change() - http://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/commit/?id=36a87e44f642966442fd0d23f2ec536851e00236
> 2. UBI: Fastmap: Fix races in ubi_wl_get_peb() - http://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/commit/?id=d59f21bebe9d0fda34027ff1afda4f2b0d5f1869
> 
> Unfortunately, even with these fixes, the issue is still reproducible.
> Picking up all 46 fixes is a bit risky since I will be picking them out of context, without other related ubi patches.
> I was wondering if you encountered similar error message and was hoping that you could advise me on a shorter list of patches to pick up.

Please apply all patches. They depend on each other and fix a lot of nasty issues.

Currently I have a similar corruption to analyze on my desk. Maybe you're hitting the same one.
If you can trigger the issue using a fully patched kernel, I'd love to get a copy of the corrupted image.

Thanks,
//richard

Re: ubifs_recover_leb() failure

[Brokhman, Tanya]

Hi Richard

On 6/25/2015 4:45 PM, Richard Weinberger wrote:
> Hi!
>
> Am 25.06.2015 um 15:40 schrieb Brokhman, Tanya:
>> Hi Richard et. al,
>>
>> We recently encountered the following error messages:
>>
>> UBIFS-0 error (pid 1): ubifs_recover_leb: corrupt empty space LEB 159:32768, corruption starts at 3810
>>
>> UBIFS-0 error (pid 1): ubifs_scanned_corruption: corruption at LEB 159:36578
>>
>> UBIFS-0 error (pid 1): ubifs_scanned_corruption: first 8192 bytes from LEB 159:36578
>>
>> 00000000: fffffffe ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff  ................................
>>
>> 00000020: ffffffff ffffffff ffffffff ffffffff ffffffff fffffffd ffffffff ffffffff  ................................
>>
>>
>> The test scenario is simple: power cut while writing.
>> We run this in a loop and the issue is not easily reproducible but the failure is constant.
>>
>> We're working with a 3.10 based kernel, fastmap enabled. I saw that you released a lot of fixes in fastmap (~46).
>> I picked up these below 2 as they seemed to resolve a similar issue:
>> 1. UBI: Fastmap: Fix race in ubi_eba_atomic_leb_change() - http://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/commit/?id=36a87e44f642966442fd0d23f2ec536851e00236
>> 2. UBI: Fastmap: Fix races in ubi_wl_get_peb() - http://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/commit/?id=d59f21bebe9d0fda34027ff1afda4f2b0d5f1869
>>
>> Unfortunately, even with these fixes, the issue is still reproducible.
>> Picking up all 46 fixes is a bit risky since I will be picking them out of context, without other related ubi patches.
>> I was wondering if you encountered similar error message and was hoping that you could advise me on a shorter list of patches to pick up.
> Please apply all patches. They depend on each other and fix a lot of nasty issues.
>
> Currently I have a similar corruption to analyze on my desk. Maybe you're hitting the same one.
> If you can trigger the issue using a fully patched kernel, I'd love to get a copy of the corrupted image.

Thank you for your inputs!
Unfortunately I'm bound by our legal team from sharing any images.
Applying all of the FM patches didn't go easily, so we did a different 
test - tried to disable fastmap and reproduce the issue. We're hitting 
bitflips now but the original issue is not reproduced and the build is 
significantly more stable now.
Hope this helps you with your issue.

Thanks
-Tanya

-- 
QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Re: ubifs_recover_leb() failure

[Richard Weinberger]

Tanya,

Am 30.06.2015 um 13:48 schrieb Brokhman, Tanya:
>> Currently I have a similar corruption to analyze on my desk. Maybe you're hitting the same one.
>> If you can trigger the issue using a fully patched kernel, I'd love to get a copy of the corrupted image.
> 
> Thank you for your inputs!
> Unfortunately I'm bound by our legal team from sharing any images.

Very sad. :-(

> Applying all of the FM patches didn't go easily, so we did a different test - tried to disable fastmap and reproduce the issue. We're hitting bitflips now but the original issue is
> not reproduced and the build is significantly more stable now.
> Hope this helps you with your issue.

Currently it looks more like a BSP issue. But investigation is still ongoing.

Thanks,
//richard