parent transid verify failed on 604602368 wanted 840641 found 840639

parent transid verify failed on 604602368 wanted 840641 found 840639

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

Hi!

Yesterday I upgraded from Linux 5.1.1 (built with GCC 8.3.0) to Linux 5.1.4
(built with GCC 9.1.0).  The next boot was extremely slow and the desktop
environment (KDE Plasma) never really started, but got kind of stuck in the
startup screen.  So I switched to a VT and pressed ctrl+alt+del.  The next
boot stopped early with following message:

[T445] BTRFS: device label <...> devid 1 transid 840641 /dev/bcache0
[T599] BTRFS info (device bcache0): disk space caching is enabled
[T599] BTRFS info (device bcache0): has skinny extents
[T599] BTRFS error (device bcache0): parent transid verify failed on 604602368
wanted 840641 found 840639
[T599] BTRFS error (device bcache0): open_ctree failed

How can I recover from this?

The filesystem should have several snapshots (created by snapper [1], on every
boot and hourly).  Will they be of any help recovering my data?

Best regards,
Dennis

[1]: http://snapper.io/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: parent transid verify failed on 604602368 wanted 840641 found 840639

[Chris Murphy]

On Mon, May 27, 2019 at 3:33 PM Dennis Schridde <devurandom@gmx.net> wrote:
>
> Hi!
>
> Yesterday I upgraded from Linux 5.1.1 (built with GCC 8.3.0) to Linux 5.1.4
> (built with GCC 9.1.0).  The next boot was extremely slow and the desktop
> environment (KDE Plasma) never really started, but got kind of stuck in the
> startup screen.  So I switched to a VT and pressed ctrl+alt+del.  The next
> boot stopped early with following message:
>
> [T445] BTRFS: device label <...> devid 1 transid 840641 /dev/bcache0
> [T599] BTRFS info (device bcache0): disk space caching is enabled
> [T599] BTRFS info (device bcache0): has skinny extents
> [T599] BTRFS error (device bcache0): parent transid verify failed on 604602368
> wanted 840641 found 840639
> [T599] BTRFS error (device bcache0): open_ctree failed
>
> How can I recover from this?
>
> The filesystem should have several snapshots (created by snapper [1], on every
> boot and hourly).  Will they be of any help recovering my data?
>
> Best regards,
> Dennis
>
> [1]: http://snapper.io/

What happens if you revert to 5.1.1? That error suggests the super
wants a newer transid than what exist on the filesystem, which
suggests file system metadata was dropped. It's not certain from this
information what caused that: device, or some layer in between like
bcache, or Btrfs.

-- 
Chris Murphy

Re: parent transid verify failed on 604602368 wanted 840641 found 840639

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 2181 bytes --]

On Dienstag, 28. Mai 2019 06:31:00 CEST Chris Murphy wrote:
> On Mon, May 27, 2019 at 3:33 PM Dennis Schridde <devurandom@gmx.net> wrote:
> > Yesterday I upgraded from Linux 5.1.1 (built with GCC 8.3.0) to Linux
> > 5.1.4
> > (built with GCC 9.1.0).  The next boot was extremely slow and the desktop
> > environment (KDE Plasma) never really started, but got kind of stuck in
> > the
> > startup screen.  So I switched to a VT and pressed ctrl+alt+del.  The next
> > boot stopped early with following message:
> >
> > [T445] BTRFS: device label <...> devid 1 transid 840641 /dev/bcache0
> > [T599] BTRFS info (device bcache0): disk space caching is enabled
> > [T599] BTRFS info (device bcache0): has skinny extents
> > [T599] BTRFS error (device bcache0): parent transid verify failed on
> > 604602368 wanted 840641 found 840639
> > [T599] BTRFS error (device bcache0): open_ctree failed
> >
> > How can I recover from this?
> >
> > The filesystem should have several snapshots (created by snapper [1], on
> > every boot and hourly).  Will they be of any help recovering my data?
> >
> > Best regards,
> > Dennis
> >
> > [1]: http://snapper.io/
>
> What happens if you revert to 5.1.1? That error suggests the super
> wants a newer transid than what exist on the filesystem, which
> suggests file system metadata was dropped. It's not certain from this
> information what caused that: device, or some layer in between like
> bcache, or Btrfs.

The basic issue appears to be the same with Linux 5.1.1:

[T512] BTRFS: device label <...> devid 1 transid 840641 /dev/bcache0
[T587] BTRFS info (device bcache0): disk space caching is enabled
[T587] BTRFS info (device bcache0): has skinny extents
[T587] BTRFS error (device bcache0): parent transid verify failed on 368672768
wanted 840529 found 840072
[T587] BTRFS error (device bcache0): failed to read block groups: -5
[T587] BTRFS error (device bcache0): open_ctree failed

I think on subsequent boots Linux 5.1.4 also pointed out other transids than
"wanted 840641 found 840639", but I am not sure and I am hesitant to keep
rebooting in case that means loosing more data.

--Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: parent transid verify failed on 604602368 wanted 840641 found 840639

[Szalma László]

Dear Dennis,

I experienced the same, the problem is with bcache + gcc9. Immediately 
remove the cache device from the bcache, as it prevents more damages to 
your filesystem. After it downgrade gcc to 8 or avoid using bcache (with 
cache device) until the problem is solved by upstream.

Please see here for more information: 
https://bugzilla.kernel.org/show_bug.cgi?id=203573

This is a very serious issue I think, but in my case I could save all my 
files after reboot without bcache cache device (except 1 insignificant 
one), but I guess you might need backups (I hope you have).

Good luck,
László Szalma

ps: i use gentoo, not fedora by the way, so it is general issue with 
gcc, i read reports with gcc9 + 4.xxx kernels too.

2019. 05. 27. 23:33 keltezéssel, Dennis Schridde írta:
> Hi!
>
> Yesterday I upgraded from Linux 5.1.1 (built with GCC 8.3.0) to Linux 5.1.4
> (built with GCC 9.1.0).  The next boot was extremely slow and the desktop
> environment (KDE Plasma) never really started, but got kind of stuck in the
> startup screen.  So I switched to a VT and pressed ctrl+alt+del.  The next
> boot stopped early with following message:
>
> [T445] BTRFS: device label <...> devid 1 transid 840641 /dev/bcache0
> [T599] BTRFS info (device bcache0): disk space caching is enabled
> [T599] BTRFS info (device bcache0): has skinny extents
> [T599] BTRFS error (device bcache0): parent transid verify failed on 604602368
> wanted 840641 found 840639
> [T599] BTRFS error (device bcache0): open_ctree failed
>
> How can I recover from this?
>
> The filesystem should have several snapshots (created by snapper [1], on every
> boot and hourly).  Will they be of any help recovering my data?
>
> Best regards,
> Dennis
>
> [1]: http://snapper.io/

Re: parent transid verify failed on 604602368 wanted 840641 found 840639

[Dennis Schridde]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

Dear Szalma!

Thank you for the information.

On Dienstag, 28. Mai 2019 08:40:40 CEST Szalma László wrote:
> I experienced the same, the problem is with bcache + gcc9. Immediately
> remove the cache device from the bcache, as it prevents more damages to
> your filesystem. After it downgrade gcc to 8 or avoid using bcache (with
> cache device) until the problem is solved by upstream.
>
> Please see here for more information:
> https://bugzilla.kernel.org/show_bug.cgi?id=203573
> 
> This is a very serious issue I think, but in my case I could save all my
> files after reboot without bcache cache device (except 1 insignificant
> one), but I guess you might need backups (I hope you have).

I did the following so far:

readlink /sys/fs/bcache/*/cache0/set | sed 's,.*/,,' > /sys/block/bcache0/
bcache/detach

How should I proceed from here on the path that you took to recover your 
system?

--Dennis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: parent transid verify failed on 604602368 wanted 840641 found 840639

[Szalma László]

Dear Dennis,

Without the bcache cache device, I was able to mount the file system ( 
root ) for read-only. I recreated it on the same LVM volume, and 
rsync-ed the files. The files I lost (i/o error because checksum error) 
was not important. Your case might be different. But without the cache, 
additional corruption stopped happening, so I think it is safe to 
recover your files with the gcc9 + 5.1 kernel (on bcache0) - if cache 
device is not present. If you cannot mount the fs, you can try the 
standard btrfs recovery methods (mount read only) or btrfsck (please be 
aware you can make things worse with btrfsck, so if the data is 
important backup the device as byte stream for new chance)

I want to mention I had another corruption (btrfs checksum error), but 
it was simply not there after reboot without bcacha cache device. This 
means the corruption that time was not written back to the backing 
device, so I haven't lost data that time. I think I was lucky.

László Szalma

2019. 05. 28. 14:36 keltezéssel, Dennis Schridde írta:
> Dear Szalma!
>
> Thank you for the information.
>
> On Dienstag, 28. Mai 2019 08:40:40 CEST Szalma László wrote:
>> I experienced the same, the problem is with bcache + gcc9. Immediately
>> remove the cache device from the bcache, as it prevents more damages to
>> your filesystem. After it downgrade gcc to 8 or avoid using bcache (with
>> cache device) until the problem is solved by upstream.
>>
>> Please see here for more information:
>> https://bugzilla.kernel.org/show_bug.cgi?id=203573
>>
>> This is a very serious issue I think, but in my case I could save all my
>> files after reboot without bcache cache device (except 1 insignificant
>> one), but I guess you might need backups (I hope you have).
> I did the following so far:
>
> readlink /sys/fs/bcache/*/cache0/set | sed 's,.*/,,' > /sys/block/bcache0/
> bcache/detach
>
> How should I proceed from here on the path that you took to recover your
> system?
>
> --Dennis