linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Qu Wenruo <quwenruo@cn.fujitsu.com>
To: Marc MERLIN <marc@merlins.org>, "Andrew E. Mileski" <andrewm@isoar.ca>
Cc: <linux-btrfs@vger.kernel.org>
Subject: Re: btrfs is  related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?
Date: Mon, 14 Jul 2014 09:24:11 +0800	[thread overview]
Message-ID: <53C3313B.1080500@cn.fujitsu.com> (raw)
In-Reply-To: <20140713142918.GH10641@merlins.org>


-------- Original Message --------
Subject: Re: btrfs is related to OOM death problems on my 8GB server 
with both 3.15.1 and 3.14?
From: Marc MERLIN <marc@merlins.org>
To: Andrew E. Mileski <andrewm@isoar.ca>
Date: 2014年07月13日 22:29
> On Sun, Jul 06, 2014 at 07:58:15AM -0700, Marc MERLIN wrote:
>>> As an update, after 1.7 days of scrubbing, the system has started
>>> getting sluggish, I'm getting synchronization problems/crashes in some of
>>> my tools that talk to serial ports (likely due to mini deadlocks in the
>>> kernel), and I'm now getting a few btrfs hangs.
>> Predictably, it died yesterday afternoon after going into memory death
>> (it was answering pings, but userspace was dead, and even sysrq-o did
>> not respond, I had to power cycle the power outlet).
>>
>> This happened just before my 3rd scrub finished, so I'm now 2 out of 2:
>> running scrub on my 3 filesystems kills the system half way through the
>> 3rd scrub.
> Ok, I now changed the subject line to confirm that btrfs is to blame.
>
> I had my system booted 6 days and it held steady at 6.4GB free.
> I mounted 2 of my 4 btrfs filesystems (one by one) and waited a few
> days, and no problems with RAM going down.
>
> Then I mounted my 3rd btrfs filesystem, the one that holds many files
> that has rsync backups running, and I lost 1.4GB of RAM overnight.
> I've just umounted one of its mountpoints, the one where all the backups
> happen while leaving its main pool mounted and will see if RAM keeps
> going down or not (i.e. whether the memory leak is due to rsync activity
> or some other background btrfs process).
>
> But generally, is there a tool to locate which kernel function allocated
> all that RAM that seems to get allocated and forgotten?
This can be done by kernel memleak detection.
Location:
-> Kernel hacking
     -> Memory Debugging
         -> Kernel memory leak detector

Then you can check /sys/kernel/debug/memleak to see which function call 
caused the problem.

Thanks,
Qu
>
> Is /proc/slabinfo supposed to show anything useful?
>
> This is the filesystem in question:
> gargamel:~# btrfs fi df /mnt/btrfs_pool2/
> Data, single: total=3.34TiB, used=3.32TiB
> System, DUP: total=8.00MiB, used=400.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=77.50GiB, used=59.87GiB
> Metadata, single: total=8.00MiB, used=0.00
>
>
> Thanks,
> Marc


  parent reply	other threads:[~2014-07-14  1:24 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-07-04  1:19 Is btrfs related to OOM death problems on my 8GB server with both 3.15.1 and 3.14? Marc MERLIN
2014-07-04  4:33 ` Russell Coker
2014-07-04  6:04   ` Marc MERLIN
2014-07-04  6:23   ` Satoru Takeuchi
2014-07-04 14:24     ` Marc MERLIN
2014-07-04 14:45       ` Russell Coker
2014-07-04 15:07         ` Marc MERLIN
2014-07-04 22:13       ` Duncan
2014-07-05 13:47 ` Andrew E. Mileski
2014-07-05 14:43   ` Marc MERLIN
2014-07-05 15:17     ` Andrew E. Mileski
2014-07-06 14:58     ` Marc MERLIN
2014-07-13 14:29       ` btrfs is " Marc MERLIN
2014-07-13 15:37         ` Marc MERLIN
2014-07-13 15:45           ` btrfs quotas " Marc MERLIN
2014-07-14  1:36             ` Qu Wenruo
2014-07-14  2:43               ` Marc MERLIN
2014-07-14  1:24         ` Qu Wenruo [this message]
2014-07-16  0:36           ` btrfs is " Jérôme Poulin
2014-07-16 15:55           ` Marc MERLIN
2014-07-17  2:22             ` Marc MERLIN
2014-07-16  0:45       ` Is btrfs " Jérôme Poulin
2014-07-05 14:27 ` Andrew E. Mileski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=53C3313B.1080500@cn.fujitsu.com \
    --to=quwenruo@cn.fujitsu.com \
    --cc=andrewm@isoar.ca \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=marc@merlins.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).