btrfs scrub results in kernel oops does not proceed and cannot be canceled

btrfs scrub results in kernel oops does not proceed and cannot be canceled

[Sergio Callegari]

Hi,

yesterday my laptop (kernel 6.10.13) froze coming out of hibernation. 
After that it will not boot anymore, saying that the root (that is on 
btrfs) cannot be mounted. I am dropped to an emergency shell and if I 
try to manually mount from there, I get level verify errors.

Tried to boot a live iso (with kernel 6.11.5) and to see what might be 
going on.

Managed to mount with -o rescue=ibadroots,ro getting transid errors.

- As soon as I start the scrub the kernel oopses.
- The scrub does not seem to progress (calling it with status).
- The scrub cannot be canceled.

The kernel oops appears scary. Even if my filesystem is corrupted the 
way in which these tools break rather than erroring out in a nicer way 
is not very helpful.

The plain btrfs check report a level error on one root and I think I do 
not have backup roots.

- is there a way to find out if the problem affects all subvolumes or a 
single one?
- is there a way to find out what can be trusted attempting a data 
recovery with the mount based on -o rescue=ibadroots,ro?

Is there any way to find out if my nvme (the hardware device) can be 
trusted to be used again (namely if the problem originated from the 
hardware of from an error in the kernel)? The fact that the problem 
appeared when using hibernation makes me thing that maybe the nvme is 
not at fault and that it was something else.

Any clue on what could be tried on my side? Why does the kernel end up 
in an oops?

Thanks,

Sergio

Re: btrfs scrub results in kernel oops does not proceed and cannot be canceled

[Qu Wenruo]

在 2024/11/15 04:08, Sergio Callegari 写道:
> Hi,
>
> yesterday my laptop (kernel 6.10.13) froze coming out of hibernation.

Not sure why but hibernation/suspension doesn't see btrfs friendly.

> After that it will not boot anymore, saying that the root (that is on
> btrfs) cannot be mounted. I am dropped to an emergency shell and if I
> try to manually mount from there, I get level verify errors.
>
> Tried to boot a live iso (with kernel 6.11.5) and to see what might be
> going on.
>
> Managed to mount with -o rescue=ibadroots,ro getting transid errors.
>
> - As soon as I start the scrub the kernel oopses.

Dmesg please, for the regular mount (without rescue options) and the
scrub oops (with rescue options).

And I'm guess it's an extent/csum tree corruption, that caused some
btrfs_root_node() failure.

This patch should solve it:

https://lore.kernel.org/linux-btrfs/20241025045553.2012160-1-lizhi.xu@windriver.com/

> - The scrub does not seem to progress (calling it with status).
> - The scrub cannot be canceled.
>
> The kernel oops appears scary. Even if my filesystem is corrupted the
> way in which these tools break rather than erroring out in a nicer way
> is not very helpful.
>
> The plain btrfs check report a level error on one root and I think I do
> not have backup roots.
>
> - is there a way to find out if the problem affects all subvolumes or a
> single one?

So far it looks like the corruption is in extent or csum tree.
Not something you can easily fix.

> - is there a way to find out what can be trusted attempting a data
> recovery with the mount based on -o rescue=ibadroots,ro?

Just ibadroots will still verify datacsum when possible, unless the
whole csum tree is corrupted.

>
> Is there any way to find out if my nvme (the hardware device) can be
> trusted to be used again (namely if the problem originated from the
> hardware of from an error in the kernel)? The fact that the problem
> appeared when using hibernation makes me thing that maybe the nvme is
> not at fault and that it was something else.

Yes, I do not think it's the driver.
When hibernation and suspension is involved, things can get tricky.
(From ACPI bugs to btrfs bugs)

Thus my personal experience is, just do not utilize them.

Thanks,
Qu
>
> Any clue on what could be tried on my side? Why does the kernel end up
> in an oops?
>
> Thanks,
>
> Sergio
>
>

Re: btrfs scrub results in kernel oops does not proceed and cannot be canceled

[Sergio Callegari]

Thanks for the answer!

On 14/11/2024 21:20, Qu Wenruo wrote:
> 
> Dmesg please, for the regular mount (without rescue options) and the
> scrub oops (with rescue options).

Thought I had saved it, but in the restlessness of trying to put the 
machine back in a condition to let me finish work, I ended up messing 
it. My bad.

> 
> And I'm guess it's an extent/csum tree corruption, that caused some
> btrfs_root_node() failure.
> 
> This patch should solve it:
> 
> https://lore.kernel.org/linux-btrfs/20241025045553.2012160-1- 
> lizhi.xu@windriver.com/
> 
Is it already in some stable kernel? Was it a regression or something 
that was just recently discovered?

>> - The scrub does not seem to progress (calling it with status).
>> - The scrub cannot be canceled.

Mouting read only with the `rescue=` option let me get some access to 
the filesystem content and helped me recover some of the data. Other 
data was recovered from a backup that, unfortunately, was not very recent.

What seems strange is that if the filesystem can be mounted it cannot be 
scrubbed.

I have a questions, if you are so kind to answer:

- mounting in rescue mode I have apparently lost all the files at the 
top of my subvolumes (some disappeared altogether, other appeared with 
zero length). The stuff inside the directories that survived seems to be 
there. Can I trust these files to be correct? Are there checksums in 
place, so I would have seen errors if they were not?

> 
> Yes, I do not think it's the driver.
> When hibernation and suspension is involved, things can get tricky.
> (From ACPI bugs to btrfs bugs)

My swap (for hibernation) was not a file in btrfs, but a separate 
partition. Can this make any difference?

> Thus my personal experience is, just do not utilize them.

Hard not to use hibernation with laptops that only sleep to hidle, so 
that the battery charge ends up while sleeping and you get an unclean 
shutdown whenever you leave the laptop alone for more than 12 hours.

And to switch off the laptop is a pain with full disk encryption and 
grub taking ages to unlock the encrypted device...

I'll try to back up more often...

Thanks
Sergio

Re: btrfs scrub results in kernel oops does not proceed and cannot be canceled

[Qu Wenruo]

在 2024/11/18 08:19, Sergio Callegari 写道:
> Thanks for the answer!
> 
> On 14/11/2024 21:20, Qu Wenruo wrote:
>>
>> Dmesg please, for the regular mount (without rescue options) and the
>> scrub oops (with rescue options).
> 
> Thought I had saved it, but in the restlessness of trying to put the 
> machine back in a condition to let me finish work, I ended up messing 
> it. My bad.
> 
>>
>> And I'm guess it's an extent/csum tree corruption, that caused some
>> btrfs_root_node() failure.
>>
>> This patch should solve it:
>>
>> https://lore.kernel.org/linux-btrfs/20241025045553.2012160-1- 
>> lizhi.xu@windriver.com/
>>
> Is it already in some stable kernel? Was it a regression or something 
> that was just recently discovered?

It's something introduced with the rescue=ibadroots option.
With that option, we allow some tree roots to be corrupted, but not 
every call site can handle it (scrub is one example).

It's not yet in upstream, but I believe we will push it for an rc 
release and backport it.

> 
>>> - The scrub does not seem to progress (calling it with status).
>>> - The scrub cannot be canceled.
> 
> Mouting read only with the `rescue=` option let me get some access to 
> the filesystem content and helped me recover some of the data. Other 
> data was recovered from a backup that, unfortunately, was not very recent.
> 
> What seems strange is that if the filesystem can be mounted it cannot be 
> scrubbed.

> 
> I have a questions, if you are so kind to answer:
> 
> - mounting in rescue mode I have apparently lost all the files at the 
> top of my subvolumes (some disappeared altogether, other appeared with 
> zero length). The stuff inside the directories that survived seems to be 
> there. Can I trust these files to be correct? Are there checksums in 
> place, so I would have seen errors if they were not?

If your csum tree is not corrupted, ibadroots still allow csum verification.

This can only be determined from your dmesg.

And if csum verification is in place, you do not need scrub, but regular 
read is enough to know if it's fine.
(Bad one will cause the read to return -EIO).

> 
>>
>> Yes, I do not think it's the driver.
>> When hibernation and suspension is involved, things can get tricky.
>> (From ACPI bugs to btrfs bugs)
> 
> My swap (for hibernation) was not a file in btrfs, but a separate 
> partition. Can this make any difference?

I do not think it's the swap partition/file causing problems.

Mind to share if you have hit some other problems related to hibernation 
or suspension?

I'm not an expert but I guess we have to take this issue more seriously.

> 
>> Thus my personal experience is, just do not utilize them.
> 
> Hard not to use hibernation with laptops that only sleep to hidle, so 
> that the battery charge ends up while sleeping and you get an unclean 
> shutdown whenever you leave the laptop alone for more than 12 hours.
> 
> And to switch off the laptop is a pain with full disk encryption and 
> grub taking ages to unlock the encrypted device...

I regularly shutdown my laptop and accept the cost of LUKS unlock.
But in my experience, the LUKS unlock only takes seconds...

But I'm using systemd-boot with an plaintext ESP partition (with 
kernel/initramfs in it), so the experience may differ.

Thanks,
Qu

> 
> I'll try to back up more often...
> 
> Thanks
> Sergio
> 
> 

Re: btrfs scrub results in kernel oops does not proceed and cannot be canceled

[Andrei Borzenkov]

On Mon, Nov 18, 2024 at 9:48 AM Qu Wenruo <wqu@suse.com> wrote:
> >
> > And to switch off the laptop is a pain with full disk encryption and
> > grub taking ages to unlock the encrypted device...
>
> I regularly shutdown my laptop and accept the cost of LUKS unlock.
> But in my experience, the LUKS unlock only takes seconds...
>

grub is compiled for the most basic x86 CPU (i386) and does not have
any handcrafted assembler code to compute digests, so the time is
spent in PBKDF. The only way to make it faster is to reduce the number
of PBKDF iterations.

> But I'm using systemd-boot with an plaintext ESP partition (with
> kernel/initramfs in it), so the experience may differ.
>

Yes. systemd-boot does not touch LUKS at all, it is unlocked after the
full kernel is booted using all available optimizations.