Re: Hang / zombie process from Xarray page-fault conversion (bisected)

[Dave Chinner <david@fromorbit.com>]

On Mon, Mar 11, 2019 at 08:35:05PM -0700, Dan Williams wrote:
> On Mon, Mar 11, 2019 at 8:10 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Thu, Mar 07, 2019 at 10:16:17PM -0800, Dan Williams wrote:
> > > Hi Willy,
> > >
> > > We're seeing a case where RocksDB hangs and becomes defunct when
> > > trying to kill the process. v4.19 succeeds and v4.20 fails. Robert was
> > > able to bisect this to commit b15cd800682f "dax: Convert page fault
> > > handlers to XArray".
> > >
> > > I see some direct usage of xa_index and wonder if there are some more
> > > pmd fixups to do?
> > >
> > > Other thoughts?
> >
> > I don't see why killing a process would have much to do with PMD
> > misalignment.  The symptoms (hanging on a signal) smell much more like
> > leaving a locked entry in the tree.  Is this easy to reproduce?  Can you
> > get /proc/$pid/stack for a hung task?
> 
> It's fairly easy to reproduce, I'll see if I can package up all the
> dependencies into something that fails in a VM.
> 
> It's limited to xfs, no failure on ext4 to date.
> 
> The hung process appears to be:
> 
>      kworker/53:1-xfs-sync/pmem0

That's completely internal to XFS. Every 30s the work is triggered
and it either does a log flush (if the fs is active) or it syncs the
superblock to clean the log and idle the filesystem. It has nothing
to do with user processes, and I don't see why killing a process has
any effect on what it does...

> ...and then the rest of the database processes grind to a halt from there.
> 
> Robert was kind enough to capture /proc/$pid/stack, but nothing interesting:
> 
> [<0>] worker_thread+0xb2/0x380
> [<0>] kthread+0x112/0x130
> [<0>] ret_from_fork+0x1f/0x40
> [<0>] 0xffffffffffffffff

Much more useful would be:

# echo w > /proc/sysrq-trigger

And post the entire output of dmesg.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com