parent transid verify failed + level verify failed

parent transid verify failed + level verify failed

[Martin Steigerwald]

Hi!

Have a great Sunday and absolutely feel free to reply at a later time :)

After having used linux-6.7-rc1 almost rc2, git commit
791c8ab095f71327899023223940dd52257a4173 in order to test out BCacheFS,
back on 6.6.1 after two attempts of non working hibernation and a strange
experience with GRUB presenting its command prompt instead of a boot menu¹,
I now got this funny stuff:

[ 1849.408572] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901
[ 1849.408943] BTRFS error (device dm-1: state EA): level verify failed on logical 1132724224 mirror 1 wanted 0 found 1
[ 1849.414073] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901
[ 1849.446144] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.448792] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.449515] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.451647] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.454860] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.455527] BTRFS error (device dm-1: state EA): parent transid verify failed on 2a006853-bec0-4026-94aa-8ade2e0c3674logical 1204469760 mirror 1 wanted 538478 found 538904
[ 1849.456286] BTRFS error (device dm-1: state EA): parent transid verify failed on logical 1204469760 mirror 1 wanted 538478 found 538904

Noticed as BTRFS for / went read only.

Kernel 6.6.1 on Devuan Ceres on ThinkPad T14 AMD Gen 1 32 GiB RAM with
2 TB Samsung 980 Pro NVME SSD.

I cannot do a scrub cause despite

WARNING: failed to write the progress status file: Read-only file system. Status recording disabled
scrub started on /, fsid […] (pid=31693)

scrub does not seem to do anything.

Will now boot into GRML and recover from backup, also checking the other
BTRFS filesystems.

While focus will be to get to a full working state as quickly as possible,
I intend to do a full dd copy for forensic analysis.



[1] Described in more detail at the end of:

Re: Questions related to BCacheFS

https://lore.kernel.org/linux-bcachefs/2273246.iZASKD2KPV@lichtvoll.de/T/#m3fc22fb92f1f2d160f2b3a2387dc1f10d067fb97

Best,
-- 
Martin

Re: parent transid verify failed + level verify failed

[Martin Steigerwald]

Martin Steigerwald - 19.11.23, 12:34:44 CET:
> After having used linux-6.7-rc1 almost rc2, git commit
> 791c8ab095f71327899023223940dd52257a4173 in order to test out BCacheFS,
> back on 6.6.1 after two attempts of non working hibernation and a
> strange experience with GRUB presenting its command prompt instead of a
> boot menu¹, I now got this funny stuff:
> 
> [ 1849.408572] BTRFS error (device dm-1: state EA): parent transid
> verify failed on logical 1178386432 mirror 1 wanted 538478 found 538901

Finally recovered.

/ was not mountable anymore after booting into GRML. Made image with dd 
and restored from backup onto a newly created BTRFS.

/home also had errors. Did not take a chance. Scrubbed it, updated a 
backup with good old rsync, recreated filesystem and restored from updated 
backup.

Another filesystem with larger files that only sees user space writes when I 
write something to it, has been okay.

I have the following diagnostic data:

1) For /: dd image on external SSD with BTRFS, thus I can snapshot and/or 
ref-link copy it around and run various diagnostic stuff on it, btrfs check 
output (no repair), scrub was okay.

2) For /home: btrfs check output (no repair), scrub was okay.

3) I think I have some dmesg output from what caused the root filesystem 
going read only, but I think that is the output I already posted.

SMART status of 2TB Samsung 980 Pro disk okay. I do not assume that there 
is anything wrong with the device.

On what caused the corruption? No idea. Could be something wrong with that 
6.7-almost-rc kernel, could be some corruption that happened on failed 
hibernation attempts, could have to do something with BCacheFS albeit I 
doubt it. Anyway, I am back on 6.6.1 for now and will not touch 6.7 kernel 
again on this laptop before I am confident that it is is safe.

About diagnostic data: Willing to post more after the weekend. Just let me 
know what you need. Of course I am not willing to upload complete dd image 
somewhere, even not with just the operating system filesystem, but I can 
run stuff on it and provide output. Otherwise you can have any of the btrfs 
check output.

Best,
-- 
Martin

Re: parent transid verify failed + level verify failed

[Martin Steigerwald]

Martin Steigerwald - 19.11.23, 17:41:17 CET:
> / was not mountable anymore after booting into GRML. Made image with dd
> and restored from backup onto a newly created BTRFS.

I at least have a possible explanation.

Yesterday I experienced 6.7-almost-rc2 not hibernating correctly, but 
instead hanging with black screen. I rebooted into 6.6.1. Last thing I did 
was remove 6.7-almost-rc2 and hibernate.

In the morning I was greeted by GRUB command line prompt for a reason I do 
not understand yet either. I recovered from this with two GRML sessions. 
For that at least I unlocked LUKS and mounted / and /boot in GRML. If my 
memory is correct after I fixed booting up in these two GRML sessions, 
kernel 6.6.1 resumed from hibernation. That does make some session as in 
the GRML session I did not touch the swap volume. So I bet the hibernation 
image was not invalidated. However that would explain the corruption on / 
filesystem. Cause it was mounted in between and then the kernel resumed 
from a hibernation image with outdated in-memory data structures for 
BTRFS.

If my memory and analysis is correct this would easily explain the 
corruption on /. Still not sure what happened to /home. I think I did not 
touch it on attempting to repair GRUB. However I am not 100% sure about 
that.

Would be interesting to know whether diagnostic data would fit that 
explanation.

In case that is correct, then I think it would be good that in case I ever 
need to fix up GRUB again I also activate swap volume within GRML in the 
hope to invalidate the hibernation image. But I am not completely sure 
whether activating swap would be enough for that.

> /home also had errors. Did not take a chance. Scrubbed it, updated a
> backup with good old rsync, recreated filesystem and restored from
> updated backup.

Best,
-- 
Martin