Re: xfs trace in 4.4.2 / also in 4.3.3 WARNING fs/xfs/xfs_aops.c:1232 xfs_vm_releasepage

From: Brian Foster <bfoster@redhat.com>
To: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
Cc: Dave Chinner <david@fromorbit.com>,
	linux-fsdevel@vger.kernel.org,
	"xfs-masters@oss.sgi.com" <xfs-masters@oss.sgi.com>,
	"xfs@oss.sgi.com" <xfs@oss.sgi.com>
Subject: Re: xfs trace in 4.4.2 / also in 4.3.3 WARNING fs/xfs/xfs_aops.c:1232 xfs_vm_releasepage
Date: Wed, 23 Mar 2016 10:07:36 -0400	[thread overview]
Message-ID: <20160323140736.GD43073@bfoster.bfoster> (raw)
In-Reply-To: <56F299E3.4020703@profihost.ag>

On Wed, Mar 23, 2016 at 02:28:03PM +0100, Stefan Priebe - Profihost AG wrote:
> sorry new one the last one got mangled. Comments inside.
> 
> Am 05.03.2016 um 23:48 schrieb Dave Chinner:
> > On Fri, Mar 04, 2016 at 04:03:42PM -0500, Brian Foster wrote:
> >> On Fri, Mar 04, 2016 at 09:02:06PM +0100, Stefan Priebe wrote:
> >>> Am 04.03.2016 um 20:13 schrieb Brian Foster:
> >>>> On Fri, Mar 04, 2016 at 07:47:16PM +0100, Stefan Priebe wrote:
> >>>>> Am 20.02.2016 um 19:02 schrieb Stefan Priebe - Profihost AG:
> >>>>>>
> >>>>>>> Am 20.02.2016 um 15:45 schrieb Brian Foster <bfoster@redhat.com>:
> >>>>>>>
> >>>>>>>> On Sat, Feb 20, 2016 at 09:02:28AM +0100, Stefan Priebe wrote:
...
> 
> This has happened again on 8 different hosts in the last 24 hours
> running 4.4.6.
> 
> All of those are KVM / Qemu hosts and are doing NO I/O except the normal
> OS stuff as the VMs have remote storage. So no database, no rsync on
> those hosts - just the OS doing nearly nothing.
> 
> All those show:
> [153360.287040] WARNING: CPU: 0 PID: 109 at fs/xfs/xfs_aops.c:1234
> xfs_vm_releasepage+0xe2/0xf0()
> 

Ok, well at this point the warning isn't telling us anything beyond
you're reproducing the problem. We can't really make progress without
more information. We don't necessarily know what application or
operations caused this by the time it occurs, but perhaps knowing what
file is affected could give us a hint.

We have the xfs_releasepage tracepoint, but that's unconditional and so
might generate a lot of noise by default. Could you enable the
xfs_releasepage tracepoint and hunt for instances where delalloc != 0?
E.g., we could leave a long running 'trace-cmd record -e
"xfs:xfs_releasepage" <cmd>' command on several boxes and wait for the
problem to occur. Alternatively (and maybe easier), run 'trace-cmd start
-e "xfs:xfs_releasepage"' and leave something like 'cat
/sys/kernel/debug/tracing/trace_pipe | grep -v "delalloc 0" >
~/trace.out' running to capture instances.

If we can get a tracepoint hit, it will include the inode number and
something like 'find / -inum <ino>' can point us at the file.

Brian

> Stefan
> 
> > 
> > -Dave.
> > 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs