xfs: log recovery hang fixes

Dave Chinner
2022-03-07T05:33:01Z

Hi folks,

Willy reported generic/530 had started hanging on his test machines
and I've tried to reproduce the problem he reported. While I haven't
reproduced the exact hang he's been having, I've found a couple of
others while running g/530 in a tight loop on a couple of test
machines.

The first two patches are a result of a hang documented in patch 1.
The change to run the log worker earlier is defensive, but serves to
break generic log space deadlocks during intent and unlinked inode
recovery as it does at normal runtime. This doesn't fix the problem,
just adds a layer of protection that means stuff that gets stuck on
pinned buffers, push hangs, etc only stays hung up for 30s at most.

The second patch fixes the hang that results from delwri buffer
pushing racing with modifications that pin the buffer (i.e.
transaction commit) and then require access to it again soon after.
The buffer is locked by delwri submission that is waiting for it to
be unpinned, but the processes that might be able to trigger an
unpin are blocked waiting for the buffer lock itself. This happens
during log recovery when processing unlinked inodes that hit the
same inode cluster buffer.

The third patch is for log recovery hangs I've been seeing that
occur after unlinked inode recovery has completed and the AIL is
being pushed out. The trigger may be unique to the highly modified
kernel I was running (and mitigated to a 30s delay to log recovery
completion in g/530 by the first patch in the series), but I have
occasionally seen period hangs in xfs_ail_push_all_sync() in the
past where the AIL has not been fully emptied but it is sleeping
without making progress. Hence I think the problem is a real one,
just I don't have a way of reproducing it reliably an unmodified
kernel.

Willy, can you see if these patches fix the problem you are seeing?
If not, I still think they stand alone as necessary fixes, but I'll
have to keep digging to find out why you are seeing hangs in g/530.

Cheers,

Dave.