From: Dave Chinner <david@fromorbit.com>
To: Cyril Peponnet <cyril.peponnet@nuagenetworks.net>
Cc: linux-xfs@vger.kernel.org
Subject: Re: XFS: possible memory allocation deadlock in kmem_alloc on glusterfs setup
Date: Mon, 5 Dec 2016 10:50:59 +1100 [thread overview]
Message-ID: <20161204235059.GO31101@dastard> (raw)
In-Reply-To: <D3BB642A-1CBB-47CB-89AA-E99837228F35@nuagenetworks.net>
On Sun, Dec 04, 2016 at 03:24:50PM -0800, Cyril Peponnet wrote:
> > On Dec 4, 2016, at 2:46 PM, Dave Chinner <david@fromorbit.com>
> > Which used LVM snapshots to take snapshots of the entire brick.
> > I don't see any LVM in your config, so I'm not sure what
> > snapshot implementation you are using here. What are you using
> > to take the snapshots of your VM image files? Are you actually
> > using the qemu qcow2 snapshot functionality rather than anything
> > native to gluster?
> >
>
> Yes sorry it was not clear enough, qemu-img snapshots no native
> snapshots.
Ok, so that's a fragmentation problem in it's own right. both
internal qcow2 fragmentation and file fragmentation.
> > Also, can you attach the 'xfs_bmap -vp' output of some of these
> > image files and their snapshots?
>
> A snapshot:
> https://gist.github.com/CyrilPeponnet/8108c74b9e8fd1d9edbf239b2872378d
> (let me know if you need more basically there is around 600 live
> snapshots sitting here).
1200 extents, mostly small, almost entirely adjacent. Typical qcow2
file fragmentation pattern. That's not going to cause your memory
allocation problems - can you find one that has hundreds of
thousands of extents?
> >
> > 56GB of cached file data. If you're getting high order
> > allocation failures (which I suspect is the problem) then this
> > is a memory fragmentation problem more than anything.
> >
> >> ----------------------------------------------------------------
> >> DG/VD TYPE State Access Consist Cache Cac sCC Size Name
> >> ----------------------------------------------------------------
> >> 0/0 RAID0 Optl RW Yes RAWBC - ON 7.275 TB scratch
> >> ----------------------------------------------------------------
> >>
> >> Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially
> >> Degraded|dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read
> >> Write|HD=Hidden|B=Blocked|Consist=Consistent| R=Read Ahead
> >> Always|NR=No Read Ahead|WB=WriteBack| AWB=Always
> >> WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
> >
> > IIRC, AWB means that if the cache goes into degraded/offline
> > mode, you’re vulnerable to corruption/loss on power
> > failure…
>
> Yes we have BBU + redundant PSU to address that.
BBU fails, data center loses power, corruption/data loss still
occurs. Not my problem, though.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2016-12-04 23:51 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-12-03 19:08 XFS: possible memory allocation deadlock in kmem_alloc on glusterfs setup Cyril Peponnet
2016-12-04 21:49 ` Dave Chinner
2016-12-04 22:07 ` Cyril Peponnet
2016-12-04 22:46 ` Dave Chinner
2016-12-04 23:24 ` Cyril Peponnet
2016-12-04 23:50 ` Dave Chinner [this message]
2016-12-05 1:14 ` Cyril Peponnet
2016-12-05 1:22 ` Dave Chinner
2016-12-05 1:48 ` Cyril Peponnet
[not found] ` <C07DD929-5600-4934-A6B0-C0A7D83D7247@nuagenetworks.net>
2016-12-05 7:46 ` Dave Chinner
2016-12-05 15:51 ` Cyril Peponnet
2016-12-05 21:45 ` Dave Chinner
2016-12-06 17:54 ` Cyril Peponnet
2016-12-07 6:16 ` Dave Chinner
[not found] ` <473936408.4772.1481091425441@itfw6.prod.google.com>
[not found] ` <8176484246282250577@unknownmsgid>
2016-12-07 19:44 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161204235059.GO31101@dastard \
--to=david@fromorbit.com \
--cc=cyril.peponnet@nuagenetworks.net \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).