Re: Very High Load, kernel 2.4.18, apache/mysql

Simon Kirby
2002-09-25T05:19:01Z

On Tue, Sep 24, 2002 at 10:38:56PM -0400, Adam Goldstein wrote:

> [root@nosferatu whitewlf]# vmstat -n 1
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 5 5 2 94076 1181592 61740 219676 0 0 10 16 125 111 69 12 19
> 7 2 4 94076 1186024 61752 219664 0 0 0 948 454 1421 95 5 0
> 10 2 2 94076 1172288 61764 219672 0 0 0 1024 468 1425 88 12 0
> 7 2 3 94076 1175220 61772 219660 0 0 0 1236 509 1513 93 7 0
> 5 2 2 94076 1187824 61784 219664 0 0 0 864 419 1524 87 13 0
> 8 1 2 94076 1170140 61792 219656 0 0 0 656 362 945 88 12 0
> 5 7 3 94076 1182448 61800 219712 0 0 36 696 580 1616 93 7 0
> 5 4 3 94076 1186500 61808 219740 0 0 12 1252 595 1766 90 10 0
> 8 1 3 94076 1177424 61812 219744 0 0 0 1124 497 1588 96 4 0
> 8 3 3 94076 1167564 61824 219748 0 0 0 1136 485 1476 88 12 0
> 5 4 2 94076 1187024 61836 219740 0 0 0 1204 473 1659 93 7 0
> 10 6 3 94076 1180816 61840 219832 0 0 52 1124 668 3079 73 27 0
> 6 6 2 94076 1184404 61840 219932 0 0 88 1356 1110 1886 94 6 0
> 8 4 2 94076 1176276 61852 219948 0 0 0 1324 683 1819 89 11 0
> 6 4 3 94076 1183948 61860 219932 0 0 0 984 441 1296 92 8 0
> 11 1 2 94076 1177320 61872 219940 0 0 0 948 448 1351 88 12 0
> 12 2 2 94076 1150268 61880 219952 0 0 0 952 438 1206 88 12 0

(Yes, I reformatted your vmstat.)

It's mostly CPU bound (see first column), but there is some disk waiting
going on too (next two). Most of the disk activity shows writing ("bo"),
not reading ("bi"). There is some swap use, but no swap occurred during
your dump ("si", "so"), so it's probably fine.

Free memory is huge, which indicates either the box hasn't been up long,
some huge process just exited and cleared a lot of memory with it, or
your site really is small and doesn't need anywhere near that much
memory. Judging by the rate of disk reads ("bi"), it looks like it
probably has more than enough memory.

A lot of writeouts are happening, and they're happening all the time (not
in five second bursts which would indicate regular asynchronous write
out). Are applications sync()ing, fsync()ing, fdatasync()ing, or using
O_SYNC? Are you using a journalling FS and are doing a lot of metadata
(directory) changes? We saw huge problems on our mail servers when we
switched to ext3 from ext2 when with ext2 they were almost always idle
(load went from 0.2, 0.4 to 20, 30) because we're using dotlocking which
seems to annoy ext3.

If you're using a database, try disabling fsync() mode. Data integrity
after crashes might be more interesting (insert fsync() flamewar here),
but it mith help a lot. At least try it temporarily to see if this is
what is causing the load.

Always mount your filesystems with "noatime" and "nodiratime". I mount
hundreds of servers this way and nobody ever notices (except that disks
last a lot longer and there are a lot less writeouts on servers that do a
lot of reading, such as web servers). If you don't do this, _every_ file
read will result in a scheduled writeback to disk to update the atime
(last accessed time). Writing atime to disk is usually a dumb idea,
because almost nothing uses it. I think the only program in the wild
I've ever seen that uses the atime field is the finger daemon (wow).

> CPU0 states: 87.5% user, 12.0% system, 0.0% nice, 0.0% idle
> CPU1 states: 90.2% user, 9.4% system, 0.0% nice, 0.0% idle

Looks like mostly user CPU.

> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
> 16800 apache 20 0 4732 4260 2988 R 37.7 0.2 0:35 httpd
> 21171 apache 16 0 4976 4548 3268 R 36.6 0.2 2:02 httpd
> 6949 apache 17 0 4604 4132 2936 R 36.5 0.2 0:53 httpd
> 29183 apache 17 0 4900 4468 3192 R 36.0 0.2 6:18 httpd

First, check /tmp for .bugtraq.c, etc., and make sure this isn't the
Slapper worm. :)

Next, figure out why these processes have taken _minutes_ of CPU time and
are still running! If these aren't the worm, you're likely using
mod_perl or mod_php or something which can make the httpd proess take
that much CPU. Check which scripts and what conditions are creating
those processes. Play around in /proc/16800/fd, look at /proc/16800/cwd,
etc., if you can't determine what is happening by the logs. If you're
still stuck, try tracing them (see below). If it's hard to catch them
(though it appears they are slugs), switching mod_perl/mod_php to
standalone CGIs may help.

To summarize, it looks like the box is both CPU bound (above Apache
processes) and blocking on disk writes. The processes using the CPU are
not responsible for the writing out because they are in 'R' state
(running); if they were writing, they would be in mostly 'D' state.

If you want to see which processes are writing out, try:

ps auxw | grep ' D '

(Might give false positives -- just looking for 'D' state.)

If you want to see whether the journalling code is doing the writing,
try:

ps -eo pid,stat,args,wchan | grep ' D '

...and see which functions the 'D' state processes are blocking in
(requires your System.map file to be up-to-date). If you see something
about do_get_write_access (a function in fs/jbd/transaction.c), it's
likely the ext3 journalling causing all of the writing. This is what I
saw in our case with the mail servers.

This "ps" command is also useful for figuring out what other non-running
processes are doing, too. However, the wchan field often shows just
"down", which isn't very helpful.

If you are getting a lot of processes sleeping in "down" and want to
figure out where they are actually stuck, try heading over to the console
and hit right_control-scroll_lock. Modern kernels will print a stack
backtrace for each process, and you can manually translate the the EIP
locations in /System.map or /boot/System.map (whatever matches your
kernel) to the function names to find functions the kernel is/was in.

To find the function in System.map, first make make sure it is sorted.
Next, incrementally search for the first EIP, number by number. The EIP
provided in the process list dump will always be higher than the actual
function offset, because it will be somewhere in the middle of the
function (System.map lists the beginning of each function). If you don't
have incremental search, this might be tedious. Some versions of "klogd"
will do this translation for you; you might want to check your kern.log.
You may also be able to coax "ksymoops" into doing the translation for
you.

If you cannot find a match in System.map, the EIP may be in a module
(requires loading modules with a symbol dump to trace). Try the next EIP
first, you can often get a good idea of what is happening by just tracing
further back. Once you've done this a few times, you'll get used to
seeing the module offsets being quite different from built-in offsets.

If you want to figure out what a running ('R') process is doing, first
try "strace -p <pid>". If it's not making many or any system calls (eg:
an endless loop or very user-CPU-intensive loop), try ltrace. If that
provides nothing useful the only other option is to try attaching to it
with gdb and do a backtrace:

gdb /proc/<pid>/exe
attach <pid>
bt

...but you may need to compile with debugging symbols for this to provide
useful output. Chances are you won't need to do this, and "strace"
will give you a pretty good idea about what is happening.

There should be enough information you can gather from these tools to
figure out what is happening. "vmstat 1" is usually the quickest way to
get a general idea of what is happening, and "ps auxwr" and "ps aux |
grep ' D '" are useful for starting to narrow it down.

Hope this helps. :)

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ sim@stormix.com ][ sim@netnation.com ]
[ Opinions expressed are not necessarily those of my employers. ]