Re: >10% performance degradation since 2.6.18

[Matthew Wilcox <matthew@wil.cx>]

On Mon, Jul 06, 2009 at 10:57:09AM -0700, Ma, Chinang wrote:
> >-----Original Message-----
> >From: Matthew Wilcox [mailto:matthew@wil.cx]
> >Sent: Monday, July 06, 2009 10:42 AM
> >To: Ma, Chinang
> >Cc: Rick Jones; Herbert Xu; Jeff Garzik; andi@firstfloor.org;
> >arjan@infradead.org; jens.axboe@oracle.com; linux-kernel@vger.kernel.org;
> >Styner, Douglas W; Prickett, Terry O; Wilcox, Matthew R;
> >netdev@vger.kernel.org; Brandeburg, Jesse
> >Subject: Re: >10% performance degradation since 2.6.18
> >
> >On Mon, Jul 06, 2009 at 10:36:11AM -0700, Ma, Chinang wrote:
> >> For OLTP workload we are not pushing much network throughput. Lower
> >network latency is more important for OLTP performance. For the original
> >Nehalem 2 sockets OLTP result in this mail thread, we bound the two NIC
> >interrupts to cpu1 and cpu9 (one NIC per sockets). Database processes are
> >divided into two groups and pinned to socket and each processe only
> >received request from the NIC it bound to. This binding scheme gave us >1%
> >performance boost pre-Nehalem date. We also see positive impact on this NHM
> >system.
> >
> >So you've tried spreading the four RX and TX interrupts for each card
> >out over, say, CPUs 1, 3, 5, 7 for eth1 and then 9, 11, 13, 15 for eth0,
> >and it produces worse performance than having all four tied to CPUs 1
> >and 9?  Interesting.
> 
> I was comparing 2 NIC on 2 sockets to 2 NIC on the same socket. I have not tried spreading out the interrupt for a NIC to cpus in the same sockets. Is there good reason for trying this?

If spreading the network load from one CPU to two CPUs increases the
performance, spreading the network load from two to eight might get even
better performance.

Are the clients now pinned to the CPU package on which they receive their
network interrupts?

> >Can you try changing IGB_MAX_RX_QUEUES (in drivers/net/igb/igb.h, about
> >line 60) to 1, and seeing if performance improves that way?
> 
> I suppose this should wait until we find out whether spread out NIC interrupt in socket helps or not.

Yes, let's try having the driver work the way that LAD designed it
first ;-)

They may not have been optimising for the database client workload,
of course.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."