Re: PROBLEM: high system usage / poor SMP network performance

[Andrew Morton <akpm@zip.com.au>]

Vincent Sweeney wrote:
> 
> Naturally I would really like to know where / what is using up all this
> system cpu so I would like to try profiling the kernel as I'm sure this is a
> pure kernel network layer performance issue but I have no idea where to
> start so does anyone have some advice / tips on where I should start?

Yes, profiling the kernel is clearly the next step.  And it's
really easy.

1: If possible, rebuild your kernel with as few modules as possible.
   Current profiler doesn't cope with code which is in modules.

2: If you're on uniprocessor, enable the "Local APIC support on
   Uniprocessors" option.  This allows higher-resolution profiling.

3: Arrange for the kernel to be provided the `profile=1' boot
   option.  I use

	append="profile=1"

   in /etc/lilo.conf

   After a reboot, the kernel is profiling itself.  The overhead is
   really low.

4: Bring the server online and wait until it starts to play up.

Now we can profile it.  I use this script:

mnm:/home/akpm> cat $(which akpm-prof)
#!/bin/sh
TIMEFILE=/tmp/$(basename $1).time
sudo readprofile -r
sudo readprofile -M10
time "$@"
readprofile -v -m /boot/System.map | sort -n +2 | tail -40 | tee $TIMEFILE
echo created $TIMEFILE

Let's go through it:

	readprofile -r

		This clears out the kernel's current profiling info

	readprofile -M10

		This attempts to set the profiling interval to 10*HZ
		(1000 Hz).  This requires a local APIC, and a recent
		util-linux package.  Not very important if this fails.
		This command also cleans out the kernel's current
		profiling info (it's a superset of -r).

	time "$@"

		Runs the command which we wish to profile

	readprofile ...

		Emits the profile info, sorted in useful order.
		You must make sure that /boot/System.map is the
		correct one for the currently-running kernel!

So in your situation, the command which you want to profile isn't
important - you want to profile kernel activity arising from
*existing* processes.  So you can use:

	akpm-prof sleep 30

Please send the results!

-