Tracking down leaking applications

Tracking down leaking applications

[Rahul Karnik]

We are using an old P3 866 MHz with 512 MB of RAM to host a
development Apache server, and over the past few days have experienced
oom kills on a regular basis. Kernel being used is Fedora Core's
2.6.15-1_1831. A typical oom trace is shown below.

Apr  4 14:41:13 fedora kernel: oom-killer: gfp_mask=0x201d2, order=0
Apr  4 14:41:13 fedora kernel: Mem-info:
Apr  4 14:41:13 fedora kernel: DMA per-cpu:
Apr  4 14:41:13 fedora kernel: cpu 0 hot: low 0, high 0, batch 1 used:0
Apr  4 14:41:13 fedora kernel: cpu 0 cold: low 0, high 0, batch 1 used:0
Apr  4 14:41:13 fedora kernel: DMA32 per-cpu: empty
Apr  4 14:41:13 fedora kernel: Normal per-cpu:
Apr  4 14:41:13 fedora kernel: cpu 0 hot: low 0, high 186, batch 31 used:171
Apr  4 14:41:13 fedora kernel: cpu 0 cold: low 0, high 62, batch 15 used:49
Apr  4 14:41:13 fedora kernel: HighMem per-cpu: empty
Apr  4 14:41:13 fedora kernel: Free pages:        5320kB (0kB HighMem)
Apr  4 14:41:13 fedora kernel: Active:122584 inactive:645 dirty:0
writeback:0 unstable:0 free:1330 slab:2313 mapped:122684
pagetables:726
Apr  4 14:41:13 fedora kernel: DMA free:2068kB min:88kB low:108kB
high:132kB active:10280kB inactive:8kB present:16384kB
pages_scanned:12133 all_unreclaimable? yes
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 495 495
Apr  4 14:41:13 fedora kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable?
no
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 495 495
Apr  4 14:41:13 fedora kernel: Normal free:3252kB min:2800kB
low:3500kB high:4200kB active:480056kB inactive:2572kB
present:506880kB pages_scanned:671522 all_unreclaimable? yes
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 0 0
Apr  4 14:41:13 fedora kernel: HighMem free:0kB min:128kB low:128kB
high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Apr  4 14:41:13 fedora kernel: lowmem_reserve[]: 0 0 0 0
Apr  4 14:41:13 fedora kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2068kB
Apr  4 14:41:13 fedora kernel: DMA32: empty
Apr  4 14:41:13 fedora kernel: Normal: 137*4kB 2*8kB 2*16kB 1*32kB
1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3252kB
Apr  4 14:41:13 fedora kernel: HighMem: empty
Apr  4 14:41:13 fedora kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Apr  4 14:41:13 fedora kernel: Free swap  = 0kB
Apr  4 14:41:13 fedora kernel: Total swap = 0kB
Apr  4 14:41:13 fedora kernel: Free swap:            0kB
Apr  4 14:41:13 fedora kernel: 130816 pages of RAM
Apr  4 14:41:13 fedora kernel: 0 pages of HIGHMEM
Apr  4 14:41:13 fedora kernel: 2143 reserved pages
Apr  4 14:41:13 fedora kernel: 74708 pages shared
Apr  4 14:41:13 fedora kernel: 0 pages swap cached
Apr  4 14:41:13 fedora kernel: 0 pages dirty
Apr  4 14:41:13 fedora kernel: 0 pages writeback
Apr  4 14:41:13 fedora kernel: 122684 pages mapped
Apr  4 14:41:13 fedora kernel: 2313 pages slab
Apr  4 14:41:13 fedora kernel: 726 pages pagetables
Apr  4 14:41:13 fedora kernel: Out of Memory: Killed process 24188 (httpd).

The process killed has been either httpd or cronolog so far. For now,
I have upgraded to FC4's 2.6.16-1_1069 and added some swap, where
previously there was none.

Is there a way to:
- confirm that it is a userspace and not a kernel issue?
- track down the application that is leaking memory?

Thanks for the help,
Rahul
--
Rahul Karnik
rahul@genebrew.com

Re: Tracking down leaking applications

[Leonid Kalev]

Rahul Karnik wrote:

>
>The process killed has been either httpd or cronolog so far. For now,
>I have upgraded to FC4's 2.6.16-1_1069 and added some swap, where
>previously there was none.
>
>Is there a way to:
>- confirm that it is a userspace and not a kernel issue?
>- track down the application that is leaking memory?
>  
>
This seems a bit off-topic for LKML, because you should *always* check 
user-space for memory leaks before blaming the kernel. A few things that 
can help you with your questions:
- the 'ps' utility, to see who eats the memory
- valgrind - an excellent tool for tracking down memory leaks (and other 
bugs, too). Comes with Fedora, but check the Web for a newer version.

Regards,

Leo.

>Thanks for the help,
>Rahul
>--
>Rahul Karnik
>rahul@genebrew.com
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>

Re: Tracking down leaking applications

[Rahul Karnik]

On 4/10/06, Leonid Kalev <lion@odcnet.com> wrote:
> This seems a bit off-topic for LKML, because you should *always* check
> user-space for memory leaks before blaming the kernel. A few things that
> can help you with your questions:

Not blaming the kernel yet, just wondering if it can help me track
down the offending app.

> - the 'ps' utility, to see who eats the memory

Well, httpd is at the top of the list, as it should be. What I cannot
tell is if the processes are leaking memory, or if all is well.

> - valgrind - an excellent tool for tracking down memory leaks (and other
> bugs, too). Comes with Fedora, but check the Web for a newer version.

I will try valgrind on the web application.

Thanks for the help!

-Rahul
--
Rahul Karnik
rahul@genebrew.com

Re: Tracking down leaking applications

[Alan Cox]

On Sul, 2006-04-09 at 11:42 -0400, Rahul Karnik wrote:
> The process killed has been either httpd or cronolog so far. For now,
> I have upgraded to FC4's 2.6.16-1_1069 and added some swap, where
> previously there was none.

Under load apache can want a lot of memory as you have a lot of server
processes compared to say thttpd or boa.

> Is there a way to:
> - confirm that it is a userspace and not a kernel issue?
> - track down the application that is leaking memory?

Apache allows you to control the memory usage of CGIs and also the
number of servers etc. That may be a good starting point for tuning. See
the apache docs and apacheweek