Re: Timeline of IPoIB performance

[Roland Dreier <rolandd@cisco.com>]

     > 2.6.12-rc5      in-kernel    1     405   <<<<<
     > 2.6.12-rc4      in-kernel    1     470   <<<<<

I was optimistic when I saw this, because the changeover to git
occurred with 2.6.12-rc2, so I thought I could use git bisect to track
down exactly when the performance regression happened.

However, I haven't been able to get numbers that are stable enough to
track this down.  I have two systems, both HP DL145s with dual Opteron
875s and two-port mem-free PCI Express HCAs.  I use MSI-X with the
completion interrupt affinity set to CPU 0, and "taskset 2" to run
netserver and netperf on CPU 1.

With default netperf parameters (just "-H otherguy") I get numbers
between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
The numbers are quite consistent between reboots, but if I reboot the
system (even keeping the kernel identical), I see large performance
changes.  Presumably something is happening like the cache coloring of
some hot data structures changing semi-randomly depending on the
timing of various initialations.

Matt, how stable are your numbers?

 - R.