Re: I/O blocked after booting

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2024/6/14 18:02, Massimo B. 写道:
> On Fri, 2024-03-29 at 06:53 +1030, Qu Wenruo wrote:
>> I mean you should not do any fstrim/discard to see if everything works
>> fine first.
>>
>> This is to make sure the problem is really from the trim/discard part.
>
> I'm still facing this issue, sometimes after boot and sometimes also after
> longer uptimes...
>
> Meanwhile I switched from mount option discard=async to nodiscard.
> I also disabled my weekly cronjob doing fstrim -av.
>
> btrfsmaintenance currently looks like this:
>
> # grep PERIOD /etc/default/btrfsmaintenance
> BTRFS_DEFRAG_PERIOD="none"
> BTRFS_BALANCE_PERIOD="none"
> BTRFS_SCRUB_PERIOD="monthly"
> BTRFS_TRIM_PERIOD="none"
>
> So there should be no trim or discard happen anymore in the background.
> I'm going to also disable the SCRUB now, though I doubt it's related to that.

For the possibility of causing hanging IO, my personal list is:

- Qgroup
- Balance
- Discard
- Scrub

So if you hit such situation again, mind to take a periodical `ps -aux`
or `top` (using CPU usage sorting) output?

Qgroup should show one btrfs-commit-transaction kthread taking all the CPU.
And can be easily verified using `btrfs qgroup show -prce` to verify.

Balance should show some `btrfs balance` call (as long as there is no
tool directly calling the balance ioctl).
Discard is much harder to detect though.
Scrub is similar to balance.

>
> But I still run into blocked IO sometimes. It's weird, some commands and apps I
> still can run, some others are blocked, maybe due to cached IO. I'm able to
> continue doing builds and system updates by the package manager (Gentoo, emerge)
> in the background but at the same time it fails to start or stop some
> applications. I can't even shutdown the window manager or the system properly.
> All data, root and home are on the same btrfs on luks on NVMe or SSD, just
> different subvolumes.

Another solution is, to disable the discard support for LUKS.
Since it is not enabled by default, you can disable it easily.

Thanks,
Qu

>
> In that situation (kernel 6.6.13, rebooting into 6.6.30) there is nothing to see
> in dmesg. Looking at the syslog, the last line always is from last night
> midnight before the hard reboot via SysRq is done:
>
> Jun 14 00:00:00 [fcron] Job 'run-parts /etc/cron.daily' started for user systab (pid 29389)
> Jun 14 07:51:49 [kernel] Linux version 6.6.30-gentoo (root@gentoo-m) (gcc (Gentoo 13.2.1_p20240210 p14) 13.2.1 20240210, GNU ld (Gentoo 2.42 p3) 2.42.0) #1 SMP PREEMPT_DYNAMIC Fri Jun 14 07:43:15 CEST 2024
>
> In cron.daily there is nothing special:
> # ls /etc/cron.daily/
> 1metalog_flush*  disabled/  logrotate*  makewhatis*  man-db*  mlocate*  rkhunter*  systemd-tmpfiles-clean*
>
> 1metalog_flush only does...
> kill -USR1 $metalog_pid	# Set metalog to sync...
> kill -USR2 $metalog_pid	# Set metalog to async...
> ... in order to flush the metalog cached logs.
>
> Again after the shutdown doesn't do anything I start using the SysRq series R E
> I S U B and again after the E (SIGTERM to all processes) I see some errors about
> trim popping on the virtual console followed by a call trace:
>
>
> BTRFS warning (device dm-0): failed to trim 361 block group(s), last error -512
> BTRFS warning (device dm-0): failed to trim 1 block device(s), last error -512
> ----------------------------[ cut here ]----------------------------------
> WARNING: CPU: 0:PID: 4783 at 0xffffffffffffffa025fed3
> Modules linked in: af_alg md5 ....
> Hardware name:...
> RIP:...
> Code:...
> ...
> Call Trace:
> <TASK>
> ...
> </TASK>
> ---[ end trace 000000000000 ]---
> acpid: exiting
> BTRFS info (device dm-2): last umount of filesystem 1d5.....
> INIT: Switching to runlevel: 6
> ...
>
>
> Could it be that something else could still trigger a trim in the background?
> If trim is really the cause of failure, then I wonder that I have that issue on
> all my machines with different hardwares, NVMe but also SATA SSDs. Just the
> Gentoo Linux installation is almost identical.
>
> Best regards,
> Massimo
>