Re: VolanoMark regression with 2.6.27-rc1

["Zhang, Yanmin" <yanmin_zhang@linux.intel.com>]

On Thu, 2008-08-21 at 14:48 +0800, Zhang, Yanmin wrote:
> On Thu, 2008-08-21 at 08:16 +0200, Ingo Molnar wrote:
> > * Zhang, Yanmin <yanmin_zhang@linux.intel.com> wrote:
> > 
> > > > ok, i've applied this one to tip/sched/urgent instead of the 
> > > > feature-disabling patchlet. Yanmin, could you please check whether this 
> > > > one does the trick?
> > >
> > > This new patch almost doesn't help volanoMark. Pls. use the patch 
> > > which sets LB_BIAS=1 by default.
> > 
> > ok. That also removes the kernel.h complications ;-)
> Sorry, I have new update.
> Originally, I worked on 2.6.27-rc1. I just move to 2.6.27-rc3 and found
> something defferent when CONFIG_GROUP_SCHED=n.
> 
> With 2.6.27-rc3, on my 8-core stoakley, all volanoMark regression disappears,
> no matter if I enable LB_BIAS. On 16-core tigerton, the regression is still
> there if I don't enable LB_BIAS and regression becomes 11% from 65%.
I have new updates on this regression. I checked volanoMark web page and
found the client command line has option rooms and users. rooms means how many
chat room will be started. users means how many users are in 1 room. The default
rooms is 10 and users is 20, so every room has about 800 threads. As all threads of a
room just communicate within this room, so the rooms number is important.

All my previous volanoMark testing uses default rooms 10 and users 20. With wake_offine
in kernel, waker/sleeper will be moved to the same cpu gradually. However, if the
rooms is not multiple of cpu number, due to load balance, kernel will move threads from
one cpu to another cpu continually. If there are too many threads to weaken the cache-hot
effect, load balance is more important. But if there are not too many threads running,
cache-hot is more important than load balance. Should we prefer to wake_affine more?

Below is some data I collected with numerous testing on 3 machines.


On 2-quadcore processor stoakley (8-core):
kernel\rooms           |      8        |      10      |      16    |     32
-------------------------------------------------------------------------------------------
2.6.26_nogroup         |   385617      |    351247    |	323324	   |  231934
-------------------------------------------------------------------------------------------
2.6.27-rc4_nogroup     |    359124     |    336984    |	335180     |  235258
-------------------------------------------------------------------------------------------
2.6.26group            |    381425     |    343636    |  312280    |  179673
-------------------------------------------------------------------------------------------
2.6.27-rc4group        |   212112      |   270000     |	300188	    |  228465
-------------------------------------------------------------------------------------------


On 2-quadcore+HT processor new x86_64 (8-core+HT, total 16 threads):
kernel\rooms        |    10    |   16    |   24    |   32     |   64
-------------------------------------------------------------------------
2.6.26_nogroup      |  667668  | 671860  | 671662  |  621900  | 509482
-------------------------------------------------------------------------
2.6.27-rc4_nogroup  |  732346  |  800290 | 709272  |  648561  | 497243
-------------------------------------------------------------------------
2.6.26group         |  705579  |  759464 | 693697  |  636019  | 500744
-------------------------------------------------------------------------
2.6.27-rc4group     |  572426  |  674977 | 627410  |  590984  | 445651
-------------------------------------------------------------------------


On 4-quadcore tigerton processor(16-core)(32 rooms testing isn't stable on the machine, so no 32):
kernel\rooms           |      8        |      10      |   16   
------------------------------------------------------------------
2.6.26_nogroup         |   346410      |    382938    | 349405
------------------------------------------------------------------
2.6.27-rc4_nogroup     |    359124     |    336984    |	335180    
------------------------------------------------------------------
2.6.26group            |    504802     |    376513    | 319020   
------------------------------------------------------------------
2.6.27-rc4group        |   247652      |   284784     | 355132	   
------------------------------------------------------------------

I also tried different users with rooms 8 and found the results of users 20/40/60 are very close.

With group scheduing, mostly, 2.6.26 is better than 2.6.27-rc4.
Without group scheduling, the result depends on specific machine.

I also rerun hackbench with group 10/16/32, and found the result difference between 2 kernels
varies among group 10/16/32.

What's the most reasonable group/rooms we should use to test?

In the other hand, tbench(start CPU_NUM*2 clients) has about 4~5% regression with 2.6.27-rc kernels.
With 30second schedstat data during the testing, I found there is almost no wake remote and wake
affine with 2.6.26, but there are many either wake_affine or wake remote with 2.6.27-rc.

-yanmin