From: Qu Wenruo <quwenruo@cn.fujitsu.com>
To: Marc MERLIN <marc@merlins.org>, "Andrew E. Mileski" <andrewm@isoar.ca>
Cc: <linux-btrfs@vger.kernel.org>
Subject: Re: btrfs is related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?
Date: Mon, 14 Jul 2014 09:24:11 +0800 [thread overview]
Message-ID: <53C3313B.1080500@cn.fujitsu.com> (raw)
In-Reply-To: <20140713142918.GH10641@merlins.org>
-------- Original Message --------
Subject: Re: btrfs is related to OOM death problems on my 8GB server
with both 3.15.1 and 3.14?
From: Marc MERLIN <marc@merlins.org>
To: Andrew E. Mileski <andrewm@isoar.ca>
Date: 2014年07月13日 22:29
> On Sun, Jul 06, 2014 at 07:58:15AM -0700, Marc MERLIN wrote:
>>> As an update, after 1.7 days of scrubbing, the system has started
>>> getting sluggish, I'm getting synchronization problems/crashes in some of
>>> my tools that talk to serial ports (likely due to mini deadlocks in the
>>> kernel), and I'm now getting a few btrfs hangs.
>> Predictably, it died yesterday afternoon after going into memory death
>> (it was answering pings, but userspace was dead, and even sysrq-o did
>> not respond, I had to power cycle the power outlet).
>>
>> This happened just before my 3rd scrub finished, so I'm now 2 out of 2:
>> running scrub on my 3 filesystems kills the system half way through the
>> 3rd scrub.
> Ok, I now changed the subject line to confirm that btrfs is to blame.
>
> I had my system booted 6 days and it held steady at 6.4GB free.
> I mounted 2 of my 4 btrfs filesystems (one by one) and waited a few
> days, and no problems with RAM going down.
>
> Then I mounted my 3rd btrfs filesystem, the one that holds many files
> that has rsync backups running, and I lost 1.4GB of RAM overnight.
> I've just umounted one of its mountpoints, the one where all the backups
> happen while leaving its main pool mounted and will see if RAM keeps
> going down or not (i.e. whether the memory leak is due to rsync activity
> or some other background btrfs process).
>
> But generally, is there a tool to locate which kernel function allocated
> all that RAM that seems to get allocated and forgotten?
This can be done by kernel memleak detection.
Location:
-> Kernel hacking
-> Memory Debugging
-> Kernel memory leak detector
Then you can check /sys/kernel/debug/memleak to see which function call
caused the problem.
Thanks,
Qu
>
> Is /proc/slabinfo supposed to show anything useful?
>
> This is the filesystem in question:
> gargamel:~# btrfs fi df /mnt/btrfs_pool2/
> Data, single: total=3.34TiB, used=3.32TiB
> System, DUP: total=8.00MiB, used=400.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=77.50GiB, used=59.87GiB
> Metadata, single: total=8.00MiB, used=0.00
>
>
> Thanks,
> Marc
next prev parent reply other threads:[~2014-07-14 1:24 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-04 1:19 Is btrfs related to OOM death problems on my 8GB server with both 3.15.1 and 3.14? Marc MERLIN
2014-07-04 4:33 ` Russell Coker
2014-07-04 6:04 ` Marc MERLIN
2014-07-04 6:23 ` Satoru Takeuchi
2014-07-04 14:24 ` Marc MERLIN
2014-07-04 14:45 ` Russell Coker
2014-07-04 15:07 ` Marc MERLIN
2014-07-04 22:13 ` Duncan
2014-07-05 13:47 ` Andrew E. Mileski
2014-07-05 14:43 ` Marc MERLIN
2014-07-05 15:17 ` Andrew E. Mileski
2014-07-06 14:58 ` Marc MERLIN
2014-07-13 14:29 ` btrfs is " Marc MERLIN
2014-07-13 15:37 ` Marc MERLIN
2014-07-13 15:45 ` btrfs quotas " Marc MERLIN
2014-07-14 1:36 ` Qu Wenruo
2014-07-14 2:43 ` Marc MERLIN
2014-07-14 1:24 ` Qu Wenruo [this message]
2014-07-16 0:36 ` btrfs is " Jérôme Poulin
2014-07-16 15:55 ` Marc MERLIN
2014-07-17 2:22 ` Marc MERLIN
2014-07-16 0:45 ` Is btrfs " Jérôme Poulin
2014-07-05 14:27 ` Andrew E. Mileski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53C3313B.1080500@cn.fujitsu.com \
--to=quwenruo@cn.fujitsu.com \
--cc=andrewm@isoar.ca \
--cc=linux-btrfs@vger.kernel.org \
--cc=marc@merlins.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).