Re: btrfs crash on armv7

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2893 bytes --]



On 2020/11/26 下午2:15, Qu Wenruo wrote:
> 
> 
> On 2020/11/25 下午11:28, Joe Hermaszewski wrote:
>> Hi,
>>
>> I have a arm32 machine with four drives with a btrfs fs spanning then in RAID1.
>> The filesystem has started behaving badly recently and I'm writing to:
>>
>> - Solicit advice on how best to get the system back to a stable state
>> - Report a potential bug
>>
>> ## What happened:
>>
>> A couple of days ago I could no longer ssh into it, and on the serial
>> connection there were heaps of messages (and new ones appearing with great
>> frequency) along the lines of: `parent transid verify failed on blah... wanted
>> x got y`.
>>
>> Although I don't have a record of the precise messages I do remember that there
>> was a difference of `15` between x and y.
> 
> This normally can be a sign of unreliable HDD, which lies on FLUSH,
> killing metadata COW.
> 
> But, your btrfs check doesn't report the same problem, thus I'm confused.
> 
> Would you please try to run a "btrfs check --readonly /dev/sda1" with
> the fs unmounted?
> 
> And, would you provide the full dmesg of that mismatch?
> The reason for the exact number is, I'm suspecting hardware memory
> corruption.
> 
>>
>> I power-cycled system and started a scrub after it rebooted, this was
>> interrupted quite promptly by several more errors in btrfs, and the disk
>> remounted RO.
>>
>> Every now and then in the kernel log I get messages like:
>>
>> `parent transid verify failed on blah... wanted x got y`
> 
> Not showing up in the gist dmesg though.
> 
>>
>> ## Important info
>>
>> The dev stats are all zero.
>>
>> Here are the outputs of some btrfs commands, dmesg and the kernel log from the
>> previous two boots: https://gist.github.com/b1beab134403c5047e2efbceb98985f9
>>
>> The "cut here" portion of the kernel log is as follows
>>
>> ```
>> [  409.158097] ------------[ cut here ]------------
>> [  409.158205] WARNING: CPU: 1 PID: 217 at fs/btrfs/disk-io.c:531
>> btree_csum_one_bio+0x208/0x248 [btrfs]
> 
> The line number shows, the tree bytenr doesn't match with the one in memory.
> This is really too rare to be seen, especially when we have no other
> error reported from btrfs (at least in the gist)
> 
> Since there is no other problems showing up in the gist, it means it
> could be a bit flip, considering the magic generation gap you mention is
> 15, I'm more suspicious about memory bit flip.
> 
> If you can provide the full parent transid mismatch error message, it
> may help to determine the possibility of hardware memory corruption.

Also if you can access the device, especially sda, please also try to
attach it to a known working system, and check if "btrfs check" reports
any error. (That file hole error can be ignored, as it won't cause any
problem).

Thanks,
Qu

> 
> Thanks,
> Qu



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]