Re: XFS: possible memory allocation deadlock in kmem_alloc on glusterfs setup

[Dave Chinner <david@fromorbit.com>]

On Mon, Dec 05, 2016 at 07:51:45AM -0800, Cyril Peponnet wrote:
> I had the issue again but I don’t have more output in dmesg or
> journalctl even with the echo 11 > /proc/sys/fs/xfs/error_level
> set.

Which means your kernel does not have this commit:

commit 847f9f6875fb02b576035e3dc31f5e647b7617a7
Author: Eric Sandeen <sandeen@redhat.com>
Date:   Mon Oct 12 16:04:45 2015 +1100

    xfs: more info from kmem deadlocks and high-level error msgs
    
    In an effort to get more useful out of "possible memory
    allocation deadlock" messages, print the size of the
    requested allocation, and dump the stack if the xfs error
    level is tuned high.
    
    The stack dump is implemented in define_xfs_printk_level()
    for error levels >= LOGLEVEL_ERR, partly because it
    seems generically useful, and also because kmem.c has
    no knowledge of xfs error level tunables or other such bits,
    it's very kmem-specific.
    
    Signed-off-by: Eric Sandeen <sandeen@redhat.com>
    Reviewed-by: Dave Chinner <dchinner@redhat.com>
    Signed-off-by: Dave Chinner <david@fromorbit.com>

> Is there another location where I should look at ?

Nope, there's nothing in your kernel we can use to identify the
source of memory allocations. I'm pretty sure that RH have used
systemtap scripts to pull this information from these kernels for
RHEL customers - we've added additional debug help here to avoid
that need, but your kernel doesn't have that code....

Essentially, best guess is that it's file fragmentation causing
problems with extent list allocation. Finding out why that one
snapshot is fragmenting so much and mitigating it is probably the
only thing you can do right now (i.e. extent size hints). Long term
is to get gluster to do the mitigation for VM images automatically.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com