Re: xfs: log recovery hang fixes

[Dave Chinner <david@fromorbit.com>]

On Mon, Mar 07, 2022 at 05:43:18PM +0000, Matthew Wilcox wrote:
> On Mon, Mar 07, 2022 at 04:32:49PM +1100, Dave Chinner wrote:
> > Willy reported generic/530 had started hanging on his test machines
> > and I've tried to reproduce the problem he reported. While I haven't
> > reproduced the exact hang he's been having, I've found a couple of
> > others while running g/530 in a tight loop on a couple of test
> > machines.
> [...]
> > 
> > Willy, can you see if these patches fix the problem you are seeing?
> > If not, I still think they stand alone as necessary fixes, but I'll
> > have to keep digging to find out why you are seeing hangs in g/530.
> 
> I no longer see hangs, but I do see an interesting pattern in runtime
> of g/530.  I was seeing hangs after only a few minutes of running g/530,
> and I was using 15 minutes of success to say "git bisect good".  Now at 45
> minutes of runtime with no hangs.  Specifically, I'm testing 0020a190cf3e
> ("xfs: AIL needs asynchronous CIL forcing"), plus these three patches.
> If you're interested, I can see which of these three patches actually
> fixes my hang.  I should also test these three patches on top of current
> 5.17-rc, but I wanted to check they were backportable to current stable
> first.
> 
> Of the 120 times g/530 has run, I see 30 occurrences of the test taking
> 32-35 seconds.  I see one occurrence of the test taking 63 seconds.
> Usually it takes 2-3s.  This smacks to me of a 30s timeout expiring.
> Let me know if you want me to try to track down which one it is.

That'll be the log worker triggering a log force after 30s, and that
gets it unstuck. So you're still seeing the problem, only now the
watchdog kicks everything back into life.

Can you run a trace for me that captures one of those 30-60s runs
so I can see what might be happening? Something like:

# trace-cmd record -e xlog\* -e xfs_ail\* -e xfs_log\* -e xfs_inodegc\* -e printk ./check generic/530

I don't need all the XFS tracepoints - I'm mainly interested in log
and AIL interactions and what is stuck on them and when...

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com