Re: rm hanging, v6.1.35

[Chris Dunlop <chris@onthe.net.au>]

On Tue, Jul 11, 2023 at 10:13:31AM +1000, Chris Dunlop wrote:
> On Tue, Jul 11, 2023 at 07:53:54AM +1000, Chris Dunlop wrote:
>> Hi,
>>
>> This box is newly booted into linux v6.1.35 (2 days ago), it was 
>> previously running v5.15.118 without any problems (other than that 
>> fixed by "5e672cd69f0a xfs: non-blocking inodegc pushes", the reason 
>> for the upgrade).
>>
>> I have rm operations on two files that have been stuck for in excess 
>> of 22 hours and 18 hours respectively:
> ...
>> ...subsequent to starting writing all this down I have another two 
>> sets of rms stuck, again on unremarkable files, and on two more 
>> separate filesystems.
>>
>> ...oh. And an 'ls' on those files is hanging. The reboot has become 
>> more urgent.
>
> FYI, it's not 'ls' that's hanging, it's bash, because I used a 
> wildcard on the command line. The bash stack:
>
> $ cat /proc/24779/stack
> [<0>] iterate_dir+0x3e/0x180
> [<0>] __x64_sys_getdents64+0x71/0x100
> [<0>] do_syscall_64+0x34/0x80
> [<0>] entry_SYSCALL_64_after_hwframe+0x46/0xb0
>
> 'lsof' shows me it's trying to read one of the directories holding the 
> file that one of the newer hanging "rm"s is trying to remove.

Ugh. It wasn't just the "rm"s and bash hanging (and as it turns out, 
xfs_logprint), they were just obvious because that's what I was looking 
at.  It turns out there was a whole lot more hanging.

Full sysrq-w output at:

https://file.io/tg7F5OqIWo1B

Sorry if there's more I could be looking at, but I've gotta reboot this 
thing NOW...

Cheers,

Chris