Re: VM: 2.4.10 vs. 2.4.10-ac2 and qsort()

[Rob Landley <landley@trommello.org>]

On Thursday 04 October 2001 20:03, Alexei Podtelezhnikov wrote:
> Hi guys,
>
> I've already expressed my concern about using srand(1) in private e-mails.
> I think it's unscientific to use one particular random sequence. Since
> no one checked if that matters, I changed srand(1) to srand(time(NULL))
> and I'm posting my results. I don't do testing of Alan or Linus's kernels,
> but use recent Red Hat kernel. I think I've shown that it does matter.

One advantage of srand(1) is that it has a chance of being repeatable.  You 
don't WANT a truly random sequence, you want a sequence that simulates a 
reproducable work load.  That's why it makes sense to initialize the random 
number generator with a known seed, so we can do it again after applying a 
patch and see what kind of difference it made.

If you vary the initialization of the random number generator, your work load 
isn't reproducible.  It's the same TYPE of load, but not the same actual load 
from one run to the next.  If we're talking a 2% improvement expected out of 
a particular tweak, you want to reduce random variation in the test case as 
much as possible that might swamp your change.  (We get enough variation 
already from scheduling and random interrupts flushing cache state and 
such...)

There's nothing wrong with using different seed values for srand, but testing 
different VMs with different seed values has the possibility of being apples 
to oranges.

If you want to run your test in a loop for a while to invoke statistics to
counter the randomness of your tests, you could do that too.  But that's a 
different kind of test.  And one that's not nearly as convenient to run from 
tweak to tweak...

Rob