From: Qu Wenruo <quwenruo@cn.fujitsu.com>
To: Marc MERLIN <marc@merlins.org>, "Andrew E. Mileski" <andrewm@isoar.ca>
Cc: <linux-btrfs@vger.kernel.org>
Subject: Re: btrfs is related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?
Date: Mon, 14 Jul 2014 09:24:11 +0800 [thread overview]
Message-ID: <53C3313B.1080500@cn.fujitsu.com> (raw)
In-Reply-To: <20140713142918.GH10641@merlins.org>
-------- Original Message --------
Subject: Re: btrfs is related to OOM death problems on my 8GB server
with both 3.15.1 and 3.14?
From: Marc MERLIN <marc@merlins.org>
To: Andrew E. Mileski <andrewm@isoar.ca>
Date: 2014年07月13日 22:29
> On Sun, Jul 06, 2014 at 07:58:15AM -0700, Marc MERLIN wrote:
>>> As an update, after 1.7 days of scrubbing, the system has started
>>> getting sluggish, I'm getting synchronization problems/crashes in some of
>>> my tools that talk to serial ports (likely due to mini deadlocks in the
>>> kernel), and I'm now getting a few btrfs hangs.
>> Predictably, it died yesterday afternoon after going into memory death
>> (it was answering pings, but userspace was dead, and even sysrq-o did
>> not respond, I had to power cycle the power outlet).
>>
>> This happened just before my 3rd scrub finished, so I'm now 2 out of 2:
>> running scrub on my 3 filesystems kills the system half way through the
>> 3rd scrub.
> Ok, I now changed the subject line to confirm that btrfs is to blame.
>
> I had my system booted 6 days and it held steady at 6.4GB free.
> I mounted 2 of my 4 btrfs filesystems (one by one) and waited a few
> days, and no problems with RAM going down.
>
> Then I mounted my 3rd btrfs filesystem, the one that holds many files
> that has rsync backups running, and I lost 1.4GB of RAM overnight.
> I've just umounted one of its mountpoints, the one where all the backups
> happen while leaving its main pool mounted and will see if RAM keeps
> going down or not (i.e. whether the memory leak is due to rsync activity
> or some other background btrfs process).
>
> But generally, is there a tool to locate which kernel function allocated
> all that RAM that seems to get allocated and forgotten?
This can be done by kernel memleak detection.
Location:
-> Kernel hacking
-> Memory Debugging
-> Kernel memory leak detector
Then you can check /sys/kernel/debug/memleak to see which function call
caused the problem.
Thanks,
Qu
>
> Is /proc/slabinfo supposed to show anything useful?
>
> This is the filesystem in question:
> gargamel:~# btrfs fi df /mnt/btrfs_pool2/
> Data, single: total=3.34TiB, used=3.32TiB
> System, DUP: total=8.00MiB, used=400.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=77.50GiB, used=59.87GiB
> Metadata, single: total=8.00MiB, used=0.00
>
>
> Thanks,
> Marc
next prev parent reply other threads:[~2014-07-14 1:24 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-07-04 1:19 Is btrfs related to OOM death problems on my 8GB server with both 3.15.1 and 3.14? Marc MERLIN
2014-07-04 4:33 ` Russell Coker
2014-07-04 6:04 ` Marc MERLIN
2014-07-04 6:23 ` Satoru Takeuchi
2014-07-04 14:24 ` Marc MERLIN
2014-07-04 14:45 ` Russell Coker
2014-07-04 15:07 ` Marc MERLIN
2014-07-04 22:13 ` Duncan
2014-07-05 13:47 ` Andrew E. Mileski
2014-07-05 14:43 ` Marc MERLIN
2014-07-05 15:17 ` Andrew E. Mileski
2014-07-06 14:58 ` Marc MERLIN
2014-07-13 14:29 ` btrfs is " Marc MERLIN
2014-07-13 15:37 ` Marc MERLIN
2014-07-13 15:45 ` btrfs quotas " Marc MERLIN
2014-07-14 1:36 ` Qu Wenruo
2014-07-14 2:43 ` Marc MERLIN
2014-07-14 1:24 ` Qu Wenruo [this message]
2014-07-16 0:36 ` btrfs is " Jérôme Poulin
2014-07-16 15:55 ` Marc MERLIN
2014-07-17 2:22 ` Marc MERLIN
2014-07-16 0:45 ` Is btrfs " Jérôme Poulin
2014-07-05 14:27 ` Andrew E. Mileski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53C3313B.1080500@cn.fujitsu.com \
--to=quwenruo@cn.fujitsu.com \
--cc=andrewm@isoar.ca \
--cc=linux-btrfs@vger.kernel.org \
--cc=marc@merlins.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.