Re: >10% performance degradation since 2.6.18

[Jens Axboe <jens.axboe@oracle.com>]

On Fri, Jul 03 2009, Matthew Wilcox wrote:
> On Fri, Jul 03, 2009 at 08:54:14PM +0200, Jens Axboe wrote:
> > On Fri, Jul 03 2009, Andi Kleen wrote:
> > > 
> > > Matthew Wilcox <matthew@wil.cx> writes:
> > > >
> > > > ======oprofile CPU_CLK_UNHALTED for top 30 functions
> > > > Cycles% 2.6.18-92.el5-op           Cycles% 2.6.30
> > > > 70.1409 <database>                 67.0207 <database>
> > > > 1.3556 mpt_interrupt               1.7029 mpt_interrupt
> > > 
> > > It's strange that mpt_interrupt is that more costly in 2.6.30
> > > than in 2.6.18. I diffed 2.6.30's drivers/message/fusion/mptbase.c
> > > to a rhel 5.3s and they seem to be about the same. 
> > > 
> > > So why does it cost 0.5% more in 2.6.30?
> > > 
> > > [adding MPT maintainers]
> > 
> > Look at the irqs/sec rate, it's higher by about the same percentage. So
> > it's likely not a more costly irq handler, it's likely just called that
> > much more. It could be IO pattern, causing more commands to be issued
> > (which leads to more interrupts, etc).
> 
> Yes, but the irqs/sec increase doesn't appear to be due to MPT interrupts.
> In the /proc/interrupt summaries, RH5 did 388666895 IOC interrupts and
> 2.6.30 did 378419042.  As a percentage of interrupts, the IOC interrupts
> were 59.4% with RH and 51.8% with 2.6.30.

OK. So where are the extra irqs from?

> This isn't quite conclusive since the collection of /proc/interrupts is
> over the entire life of the system, not during the measurement period.
> But I do find it persuasive.

Since the total is so high, it's probably good enough and sampling only
during the measurement likely wouldn't change that picture a lot.

-- 
Jens Axboe