* Tracking down a memory leak
@ 2005-06-20 10:33 Marco Colombo
2005-06-28 9:18 ` Marco Colombo
0 siblings, 1 reply; 4+ messages in thread
From: Marco Colombo @ 2005-06-20 10:33 UTC (permalink / raw)
To: linux-kernel
Hi,
today I've found a server in a OOM condition, the funny thing is that
after some investigation I've found no process that has mem allocated
to. I even switched to single user, here's what I've found:
sh-2.05b# ps axu; free
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2708 432 ? S May16 0:10 init [S]
root 2 0.0 0.0 0 0 ? SWN May16 0:03 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? SW< May16 1:49 [events/0]
root 4 0.0 0.0 0 0 ? SW< May16 0:00 [khelper]
root 16 0.0 0.0 0 0 ? SW< May16 0:00 [kacpid]
root 87 0.0 0.0 0 0 ? SW< May16 0:33 [kblockd/0]
root 95 0.0 0.0 0 0 ? SW May16 0:00 [khubd]
root 152 0.0 0.0 0 0 ? SW< May16 0:00 [aio/0]
root 151 0.0 0.0 0 0 ? SW May16 45:44 [kswapd0]
root 245 0.0 0.0 0 0 ? SW May16 0:00 [kseriod]
root 473 0.0 0.0 0 0 ? SW May16 0:00 [md7_raid1]
root 475 0.0 0.0 0 0 ? SW May16 0:11 [md6_raid1]
root 477 0.0 0.0 0 0 ? SW May16 0:00 [md5_raid1]
root 479 0.0 0.0 0 0 ? SW May16 0:00 [md4_raid1]
root 481 0.0 0.0 0 0 ? SW May16 0:00 [md3_raid1]
root 483 0.0 0.0 0 0 ? SW May16 0:08 [md2_raid1]
root 485 0.0 0.0 0 0 ? SW May16 0:00 [md1_raid1]
root 486 0.0 0.0 0 0 ? SW May16 0:02 [md0_raid1]
root 487 0.0 0.0 0 0 ? SW May16 0:38 [kjournald]
root 690 0.0 0.0 0 0 ? SW May16 0:03 [kjournald]
root 691 0.0 0.0 0 0 ? SW May16 4:05 [kjournald]
root 692 0.0 0.0 0 0 ? SW May16 0:03 [kjournald]
root 693 0.0 0.0 0 0 ? SW May16 0:03 [kjournald]
root 694 0.0 0.0 0 0 ? SW May16 0:03 [kjournald]
root 695 0.0 0.0 0 0 ? SW May16 6:14 [kjournald]
root 696 0.0 0.0 0 0 ? SW May16 0:00 [kjournald]
root 16545 0.0 0.0 0 0 ? SW 11:47 0:00 [pdflush]
root 16551 0.0 0.0 0 0 ? SW 11:47 0:00 [pdflush]
root 16889 0.0 0.0 3072 744 ? S 11:59 0:00 minilogd
root 17235 0.0 0.0 2708 444 ttyS0 S 12:00 0:00 init [S]
root 17236 0.0 0.1 3176 1180 ttyS0 S 12:00 0:00 /bin/sh
root 17248 0.0 0.0 2732 748 ttyS0 R 12:07 0:00 ps axu
total used free shared buffers cached
Mem: 1035812 898524 137288 0 3588 16732
-/+ buffers/cache: 878204 157608
Swap: 1049248 788 1048460
sh-2.05b# uptime
12:13:28 up 35 days, 1:48, 0 users, load average: 0.00, 0.59, 16.13
sh-2.05b# uname -a
Linux xxxx.example.org 2.6.10-1.12_FC2.marco #1 Mon Feb 7 14:53:42 CET 2005
i686 athlon i386 GNU/Linux
I know this is an old Fedora Core 2 kernel, eventually I'll bring the
issue on thier lists. An upgrade has already been scheduled for this
host, so I'm not really pressed in tracking this specific bug (unless it
occurs on the new system, of course).
Anyway, I just wonder if generally there's a way to find out where those
850+ MBs are allocated. Since there are no big user processes, I'm
assuming it's a memory leak in kernel space. I'm curious, this is the
first time I see something like this. Any suggestion what to look at
besides 'ps' and 'free'?
The server has been mainly running PostgreSQL at a fairly high load for
the last 35 days, BTW.
TIA,
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Tracking down a memory leak
2005-06-20 10:33 Tracking down a memory leak Marco Colombo
@ 2005-06-28 9:18 ` Marco Colombo
2005-06-28 11:02 ` Andrew Morton
0 siblings, 1 reply; 4+ messages in thread
From: Marco Colombo @ 2005-06-28 9:18 UTC (permalink / raw)
To: linux-kernel
On Mon, 2005-06-20 at 12:33 +0200, Marco Colombo wrote:
> Hi,
> today I've found a server in a OOM condition, the funny thing is that
> after some investigation I've found no process that has mem allocated
> to. I even switched to single user, here's what I've found:
[...]
> total used free shared buffers cached
> Mem: 1035812 898524 137288 0 3588 16732
> -/+ buffers/cache: 878204 157608
> Swap: 1049248 788 1048460
> sh-2.05b# uptime
> 12:13:28 up 35 days, 1:48, 0 users, load average: 0.00, 0.59, 16.13
> sh-2.05b# uname -a
> Linux xxxx.example.org 2.6.10-1.12_FC2.marco #1 Mon Feb 7 14:53:42 CET 2005
> i686 athlon i386 GNU/Linux
>
> I know this is an old Fedora Core 2 kernel, eventually I'll bring the
> issue on thier lists. An upgrade has already been scheduled for this
> host, so I'm not really pressed in tracking this specific bug (unless it
> occurs on the new system, of course).
>
> Anyway, I just wonder if generally there's a way to find out where those
> 850+ MBs are allocated. Since there are no big user processes, I'm
> assuming it's a memory leak in kernel space. I'm curious, this is the
> first time I see something like this. Any suggestion what to look at
> besides 'ps' and 'free'?
>
> The server has been mainly running PostgreSQL at a fairly high load for
> the last 35 days, BTW.
>
> TIA,
> .TM.
Thanks to everybody who replied to me. Here's more data:
sh-2.05b# sort -rn +1 /proc/slabinfo | head -5
biovec-1 7502216 7502296 16 226 1 : tunables 120 60 0 : slabdata 33196 33196 0
bio 7502216 7502262 96 41 1 : tunables 120 60 0 : slabdata 182982 182982 0
size-64 4948 5307 64 61 1 : tunables 120 60 0 : slabdata 87 87 0
buffer_head 3691 3750 52 75 1 : tunables 120 60 0 : slabdata 50 50 0
dentry_cache 2712 2712 164 24 1 : tunables 120 60 0 : slabdata 113 113 0
I've found no way do free that memory, so decided to reboot it.
In the following days I had been monitoring the system after upgrading
to kernel-2.6.10-1.770_FC2. Here are the results I got, day by day:
bio 115333 115333 96 41 1 : tunables 120 60 0 : slabdata 2813 2813 0
biovec-1 115322 115486 16 226 1 : tunables 120 60 0 : slabdata 511 511 0
biovec-1 325006 325440 16 226 1 : tunables 120 60 0 : slabdata 1440 1440 0
bio 324987 325212 96 41 1 : tunables 120 60 0 : slabdata 7930 7932 0
bio 538535 538535 96 41 1 : tunables 120 60 0 : slabdata 13135 13135 0
biovec-1 538528 538784 16 226 1 : tunables 120 60 0 : slabdata 2384 2384 0
bio 749870 750218 96 41 1 : tunables 120 60 0 : slabdata 18296 18298 0
biovec-1 749886 750772 16 226 1 : tunables 120 60 0 : slabdata 3322 3322 0
bio 960630 960630 96 41 1 : tunables 120 60 0 : slabdata 23430 23430 0
biovec-1 960642 960726 16 226 1 : tunables 120 60 0 : slabdata 4251 4251 0
bio 1170079 1170345 96 41 1 : tunables 120 60 0 : slabdata 28543 28545 0
biovec-1 1170066 1170906 16 226 1 : tunables 120 60 0 : slabdata 5181 5181 0
bio 1379857 1380019 96 41 1 : tunables 120 60 0 : slabdata 33658 33659 0
biovec-1 1379854 1380408 16 226 1 : tunables 120 60 0 : slabdata 6108 6108 0
Clearly, something was going on. So I decided to run a vanilla kernel
instead.
I'm running 2.6.12.1 right now, and after about one day of uptime:
bio 345 369 96 41 1 : tunables 120 60 0 : slabdata 9 9 0
biovec-1 376 678 16 226 1 : tunables 120 60 0 : slabdata 3 3 0
which seem to me sane values (and they stay like that as far as I can
see). No more daily increase of more than 200,000.
I'll keep an eye on it in the next days, but I think 2.6.12.1 is not
affected.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Tracking down a memory leak
2005-06-28 9:18 ` Marco Colombo
@ 2005-06-28 11:02 ` Andrew Morton
2005-06-28 13:38 ` Marco Colombo
0 siblings, 1 reply; 4+ messages in thread
From: Andrew Morton @ 2005-06-28 11:02 UTC (permalink / raw)
To: Marco Colombo; +Cc: linux-kernel
Marco Colombo <marco@esi.it> wrote:
>
> Thanks to everybody who replied to me. Here's more data:
>
> sh-2.05b# sort -rn +1 /proc/slabinfo | head -5
> biovec-1 7502216 7502296 16 226 1 : tunables 120 60 0 : slabdata 33196 33196 0
> bio 7502216 7502262 96 41 1 : tunables 120 60 0 : slabdata 182982 182982 0
Are you using RAID?
From: Neil Brown <neilb@cse.unsw.edu.au>
insert a missing bio_put when writing the md superblock.
Without this we have a steady growth in the "bio" slab.
Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>
Signed-off-by: Andrew Morton <akpm@osdl.org>
---
drivers/md/md.c | 1 +
1 files changed, 1 insertion(+)
diff -puN drivers/md/md.c~md-bio-leak-fix drivers/md/md.c
--- 25/drivers/md/md.c~md-bio-leak-fix 2005-06-27 22:29:13.000000000 -0700
+++ 25-akpm/drivers/md/md.c 2005-06-27 22:29:13.000000000 -0700
@@ -338,6 +338,7 @@ static int super_written(struct bio *bio
if (atomic_dec_and_test(&rdev->mddev->pending_writes))
wake_up(&rdev->mddev->sb_wait);
+ bio_put(bio);
return 0;
}
_
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Tracking down a memory leak
2005-06-28 11:02 ` Andrew Morton
@ 2005-06-28 13:38 ` Marco Colombo
0 siblings, 0 replies; 4+ messages in thread
From: Marco Colombo @ 2005-06-28 13:38 UTC (permalink / raw)
To: Andrew Morton; +Cc: linux-kernel
On Tue, 2005-06-28 at 04:02 -0700, Andrew Morton wrote:
> Marco Colombo <marco@esi.it> wrote:
> >
> > Thanks to everybody who replied to me. Here's more data:
> >
> > sh-2.05b# sort -rn +1 /proc/slabinfo | head -5
> > biovec-1 7502216 7502296 16 226 1 : tunables 120 60 0 : slabdata 33196 33196 0
> > bio 7502216 7502262 96 41 1 : tunables 120 60 0 : slabdata 182982 182982 0
>
> Are you using RAID?
Thanks Andrew,
yes I'm using RAID:
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [multipath]
md1 : active raid1 hdc7[1] hda7[0]
524544 blocks [2/2] [UU]
md2 : active raid1 hdc8[1] hda8[0]
524544 blocks [2/2] [UU]
md3 : active raid1 hdc9[1] hda9[0]
3145856 blocks [2/2] [UU]
md4 : active raid1 hdc10[1] hda10[0]
1048192 blocks [2/2] [UU]
md5 : active raid1 hdc11[1] hda11[0]
2097024 blocks [2/2] [UU]
md6 : active raid1 hdc12[1] hda12[0]
25961408 blocks [2/2] [UU]
md7 : active raid1 hdc13[1] hda13[0]
25961408 blocks [2/2] [UU]
md0 : active raid1 hdc5[1] hda5[0]
262400 blocks [2/2] [UU]
unused devices: <none>
This is a pretty standard configuration for me here, I use it almost
everywhere as a mean to protect data from simple disk failures (that are
way to common these days). I have at least 5-6 servers at hand, all with
disk mirroring. Some of them match the version exactly, and the hardware
is very similar, but there was only one host showing the leak. Two
things are different on the affected server:
- it uses the dc395x driver (only a DAT device is attached);
- it's a database (PostgreSQL) server (that is, with some shm usage).
The dc395x sometimes spits some error messages:
Jun 28 05:37:47 xxxx kernel: dc395x: Interrupt from DMA engine: 0x63!
Jun 28 05:37:47 xxxx kernel: dc395x: Ignoring DMA error (probably a bad
thing) ...
I have to live with those for now. They seem to be harmless. The same
messages occur with 2.6.12.1, but the bio leak is gone (or so it seems
after 1 day of uptime), I guess they may be unrelated (the dc395x driver
used to be my suspect #1).
> From: Neil Brown <neilb@cse.unsw.edu.au>
>
> insert a missing bio_put when writing the md superblock.
>
> Without this we have a steady growth in the "bio" slab.
>
> Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>
> Signed-off-by: Andrew Morton <akpm@osdl.org>
> ---
>
> drivers/md/md.c | 1 +
> 1 files changed, 1 insertion(+)
>
> diff -puN drivers/md/md.c~md-bio-leak-fix drivers/md/md.c
> --- 25/drivers/md/md.c~md-bio-leak-fix 2005-06-27 22:29:13.000000000 -0700
> +++ 25-akpm/drivers/md/md.c 2005-06-27 22:29:13.000000000 -0700
> @@ -338,6 +338,7 @@ static int super_written(struct bio *bio
>
> if (atomic_dec_and_test(&rdev->mddev->pending_writes))
> wake_up(&rdev->mddev->sb_wait);
> + bio_put(bio);
> return 0;
> }
>
I'm sorry, but I'm missing something: I'm not able to locate that piece
of code in neither the (FC2) linux-2.6.10 tree nor the (unpatched)
linux-2.6.12.1 tree. There's no super_written() function at all, BTW.
The code looks completely different.
[...google digging...]
Is it related to this patch?
http://marc.theaimsgroup.com/?l=linux-raid&m=111155701929593&w=2
If so, I don't think I'm using it.
But the symptoms looks like the same.
Anyway, it seems solved in 2.6.12.1, so I'm not that worried. I'll wait
and see if after the upgrade to FC4 the problem is still there (with the
official FC4 kernel). I'll bring the issue to the Fedora lists, in case.
As far as I can see, the problem is already solved in mainstream. Thanks
again.
.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ Colombo@ESI.it
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2005-06-28 13:44 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-06-20 10:33 Tracking down a memory leak Marco Colombo
2005-06-28 9:18 ` Marco Colombo
2005-06-28 11:02 ` Andrew Morton
2005-06-28 13:38 ` Marco Colombo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox