Re: Call to write tests for osstest

[George Dunlap <george.dunlap@eu.citrix.com>]

On 09/08/13 13:00, Wei Liu wrote:
> On Fri, Aug 09, 2013 at 01:46:42PM +0200, Sander Eikelenboom wrote:
>> Friday, August 9, 2013, 1:19:54 PM, you wrote:
>>
>>> On gio, 2013-08-08 at 19:32 +0200, Sander Eikelenboom wrote:
>>>> * Some performance testing (network, block, cpu/mem/fork/real apps benchmarks, some other metrics)
>>>>     - perhaps makes separate graphs of these, so one can see performance increase or decrease over a larger time frame, and see at around what commits that occurred.
>>>>     - only after all basic tests succeeded and a push was done.
>>>>
>>> We are after this too. I am looking at how to make it possible to do
>>> something like that on top of OSSTest.
>>> Perf benchmarking is a little bit different from regression
>>> smoke-testing, but still I think (hope? :-)) that most of the
>>> infrastructure can be reused.
>> Yes it does require to keep the rest of the circumstances the same for a longer period of time.
>> But it could be quite valuable, slow introduced and minor performance regressions are hard to discover.
>>
>>> Also, what hardware to use and how to properly schedule these kind of
>>> "tests" is something that needs a bit more of thinking/discussion, I
>>> think.
>> Yes since you have to keep the "environment" the same, the machine should only run the perf tests at that time.
>>
> This worries me. It is really hard to keep the "environment" the same.
> AIUI network / block etc. performance can be affected by other kernel
> subsystems. Also the kernel config could also have impact on
> performance.

The same is true for functional regressions -- a bug could be introduced 
by Xen, qemu, or the kernel.  The tester already keeps track of this, as 
far as I know, and only moves one thing at a time.

  -George