Re: Inconsistent timing results of multithreaded program on an SMP machine.

[Con Kolivas <kernel@kolivas.org>]

On Sun, 20 Nov 2005 20:27, Marcel Zalmanovici wrote:
> Hi,
>
> I am trying, as part of my thesis, to make some improvement to the linux
> scheduler.
> Therefore I've written a small multithreaded application and tested how
> long on average it takes for it to complete.
>
> The results were very surprising. I expected to see the completion time
> vary 1 to 2 seconds at most.
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16
> sec, and the same happened to longer tests where the min was ~1min and the
> max ~2:30min.
>
> Does anyone have any clue as to what might happen ?
> Is there anything I can do to get stable results ?
>
> Here is a small test case program:
>
> (See attached file: sched_test.c)
>
> The test was always done on a pretty much empty machine. I've tried both
> kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
>
> I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
                                                       ^^^^^^^^^^^^^^

I suspect what you're seeing is the random nature of threads to bind either to 
a hyperthread sibling or a real core. If all your threads land on only one 
physical cpu and both hyperthread siblings it will run much slower than if 
half land on one physical cpu and half on the other physical cpu. To confirm 
this, try setting cpu affinity just for one logical core of each phyiscal cpu 
(like affinity for cpu 1 and 3 only for example) or disabling hyperthread in 
the bios.

Cheers,
Con