RE: >10% performance degradation since 2.6.18

Chetan.Loke
2009-07-06T21:59:09Z

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of
> Daniel J Blueman
> Sent: Sunday, July 05, 2009 7:01 AM
> To: Matthew Wilcox; Andi Kleen
> Cc: Linux Kernel; Jens Axboe; Arjan van de Ven
> Subject: Re: >10% performance degradation since 2.6.18
>
> On Jul 3, 9:10 pm, Arjan van de Ven <ar...@infradead.org> wrote:
> > On Fri, 3 Jul 2009 21:54:58 +0200
> >
> > Andi Kleen <a...@firstfloor.org> wrote:
> > > > That would seem to be a fruitful avenue of investigation --
> > > > whether limiting the cards to a single RX/TX interrupt would be
> > > > advantageous, or whether spreading the eight interrupts
> out over
> > > > the CPUs would be advantageous.
> >
> > > The kernel should really do the per cpu binding of MSIs
> by default.
> >
> > ... so that you can't do power management on a per socket basis?
> > hardly a good idea.
> >
> > just need to use a new enough irqbalance and it will spread out the
> > interrupts unless your load is low enough to go into low power mode.
>
> I was finding newer kernels (>~2.6.24) would set the
> Redirection Hint bit in the MSI address vector, allowing the
> processors to deliver the interrupt to the lowest interrupt
> priority (eg idle, no powersave) core
> (http://www.intel.com/Assets/PDF/manual/253668.pdf pp10-66)
> and older irqbalance daemons would periodically naively
> rewrite the bitmask of cores, delivering the interrupt to a
> static one.
>
> Thus, it may be worth checking if disabling any older
> irqbalance daemon gives any win.
>
> Perhaps there is value in writing different subsets of cores
> to the MSI address vector core bitmask (with the redirection
> hint enabled) for different I/O queues on heavy interrupt
> sources? By default, it's all cores.
>

Possible enhancement -

1) Drain the responses in the xmit_frame() path. That is, post the TX-request() and just before returning see if there are
any more responses in the RX-queue. This will minimize(only if the NIC f/w coalesces) interrupt load.
The n/w core should drain the responses rather than calling the drain-routine from the adapter's xmit_frame() handler. This way there won't be any need to
modify individual xmit_frame handlers.

PS - I'm not familiar with the networking code.

Chetan Loke