David S. Miller wrote:

> I'll test this once you work out that obvious bug.
> 

David's got it tested and working. No difference in the tbench
test reported for SPARC64.

Following are some numbers for a tbench 3.03 test: 1 client, 1 server
in different configurations.

Xeon and G5 seem to be significantly improved on the order of 1-5%.
I2 may be slightly down, but if it is significant I expect real world
workloads to be either not impacted, or hopefully some might see a
small improvement.

tbench, MB/s, higher is better.
Dual Nocona Xeon (mwait idle):
same thread     2.6.12-rc4      -sched
                 185.6           186.5
                 185.3           186.8
                 185.6           187.4
other thread
                 186.4           187.3
                 187.3           187.1
                 187.8           188.4
other CPU
                 173.0           174.0
                 170.7           174.2
                 169.5           175.7

Dual G5:
same CPU
                 256.3           259.0
                 255.4           262.3
                 256.5           259.7
other CPU
                 150.3           155.4
                 148.5           155.4
                 150.1           154.0

Itanium 2:
same CPU
                 133.1           131.8
                 128.7           131.6
                 133.2           132.2
other CPU
                 84.9            83.9
                 84.6            83.9
                 84.5            84.0

Real performance testing would be good, if anyone is interested.
Updated patch attached.

Unless anyone has an objection, I'm going to hack up untested
implementations for the rest of the architectures and see if Andrew
will put the patch in -mm for a while.