Re: 2.6.25-rc4 OOMs itself dead on bootup

Frank Sorenson
2008-03-07T19:28:45Z

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ingo Molnar wrote:
> * Frank Sorenson <frank@tuxrocks.com> wrote:
>
>> 2.6.25-rc4 invokes the oom-killer repeatedly when attempting to boot,
>> eventually panicing with "Out of memory and no killable processes."
>> This happens since at least 2.6.25-rc3, but 2.6.24 boots just fine,
>>
>> The system is a Dell Inspiron E1705 running Fedora 8 (x86_64).
>>
>> My .config is at http://tuxrocks.com/tmp/config-2.6.25-rc4, and a
>> syslog of the system up until the point where it oom-killed syslog
>> (just before the panic) is at
>> http://tuxrocks.com/tmp/oom-2.6.25-rc4.txt
>
> i've picked up your .config and enabled a few drivers in it to make it
> boot on a testsystem of mine and it doesnt OOM:
>
> 20:47:17 up 10 min, 1 user, load average: 0.01, 0.07, 0.03
> total used free shared buffers cached
> Mem: 1025768 258544 767224 0 12956 169556
> -/+ buffers/cache: 76032 949736
> Swap: 3911816 0 3911816
>
> (config and bootlog from my box attached.)
>
> So it's probably not a .config dependent generic kernel problem, but
> probably something specific to your hardware.
>
> Since the first oom happens about 9 minutes into the bootup:
>
> [ 569.755853] sh invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
>
> do you have any chance to log in and capture MM statistics? The
> following script will capture a bunch of statistics:
>
> http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
>
> and takes less than a minute to run - should be enough time in theory.
> If it's possible to run it then send us the output file it produces.
>
> Ingo

Thank you for the help, and for the time.

I did some additional debugging, and I believe you're correct about it
being specific to my system. The system seems to run fine until some
time during the boot. I booted with "init=/bin/sh" (that's how the
system stayed up for 9 minutes), then it died when I tried starting
things up. I've further narrowed the OOM down to udev (though it's not
entirely udev's fault, since 2.6.24 runs fine).

I ran your debug info tool before killing the box by running
/sbin/start_udev. The output of the tool is at
http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24

Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
causes udev (or something it calls) to behave very badly.

I'll keep looking further into the cause. Thanks again for the help.

Frank
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
frank@tuxrocks.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFH0ZcXaI0dwg4A47wRAoy7AJ9ILIlACjitvOpNghRNxmOgiygk1QCfb3Oi
8Drhxc4Tvu0K+1KD0U6XUOE=
=SmQj
-----END PGP SIGNATURE-----