Increased Latencies when upgrading kernel version

[Taylor Lewick <taylor.lewick@gmail.com>]

For some time now we've been running an older kernel, 2.6.16.60.  When
we tried to upgrade, first going to 2.6.27.19 and then to 2.6.32.1 and
2.6.33.1 we noticed that latencies increased.  At first we noticed it
by doing network tests via udpping, netperf, etc.  We made some
tweaks, and were able to get network latency to within 1 to 2
microseconds of where we were previously on 2.6.16.60.  Then we did
some more testing, and noticed that system latency also seems higher.

We've done our tests on identical hardware servers, same NICs,
connected through same network gear.  Basically, we've tried to keep
everything identical except the kernel versions, and we are unable to
achieve the same performance for system latency on the newer kernels,
despite adjusting various kernel settings and recompiling.

The latency differences are about 15 microseconds per transaction.

At this point, I don't know what else to try.  I haven't played around
with the /proc/sys/kernel/sched_* paramaters under the newer kernels
yet.  Have tried changing pre-emption modes with little effect, in
fact, voluntary preemption seems to be peforming the best for us.

At this time the realtime patch isn't really an option for us to
consider, at least not yet.

Any suggestions?  Is this a known issue when upgrading to more recent
kernel versions?

Thanks,
Taylor