* [Qemu-devel] Suggestion for testing framework
@ 2008-06-03 20:59 Balazs Attila-Mihaly (Cd-MaN)
2008-06-03 21:50 ` Erik de Castro Lopo
` (2 more replies)
0 siblings, 3 replies; 18+ messages in thread
From: Balazs Attila-Mihaly (Cd-MaN) @ 2008-06-03 20:59 UTC (permalink / raw)
To: Qemu Devel
Hello all
It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines:
- several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc)
- a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-))
- the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source
- if the compile fails, it collects the error logs
- if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test...
- results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list
Please comment if you find the testing methodology good and what performance and functionality test should the process include...
Best regards.
__________________________________________________________
Sent from Yahoo! Mail.
A Smarter Email http://uk.docs.yahoo.com/nowyoucan.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN)
@ 2008-06-03 21:50 ` Erik de Castro Lopo
2008-06-03 21:53 ` Anthony Liguori
2008-06-03 22:02 ` Paul Brook
2008-06-04 10:36 ` Fabrice Bellard
2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil
2 siblings, 2 replies; 18+ messages in thread
From: Erik de Castro Lopo @ 2008-06-03 21:50 UTC (permalink / raw)
To: qemu-devel; +Cc: Balazs Attila-Mihaly (Cd-MaN)
Balazs Attila-Mihaly (Cd-MaN) wrote:
> Hello all
>
> It seems that there is agreement that some sort of automated
> testing is "a good thing" ;-).
I am a huge fan of testing and think that qemu developers and users
would both benefit from more automated testing.
> - several qemu images (one with Debian, one with Windows XP - I
> can get a free student license for it, etc)
This is testing of the final product. Especially for something as
complex as qemu, final product testing is difficult. Wherever possible
units should be tested in isolation. Debugging test failures of unit
tests are also far easier than debugging problems in a whole qemu
VM.
However, if you do pursue your original plan, it would be really nice
if either WinXP 64bit or Vista64 could be added to this list.
Cheers,
Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"Re graphics: A picture is worth 10K words - but only those to
describe the picture. Hardly any sets of 10K words can be
adequately described with pictures." -- Alan Perlis
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 21:50 ` Erik de Castro Lopo
@ 2008-06-03 21:53 ` Anthony Liguori
2008-06-03 22:02 ` Paul Brook
1 sibling, 0 replies; 18+ messages in thread
From: Anthony Liguori @ 2008-06-03 21:53 UTC (permalink / raw)
To: qemu-devel; +Cc: Balazs Attila-Mihaly (Cd-MaN)
Erik de Castro Lopo wrote:
> Balazs Attila-Mihaly (Cd-MaN) wrote:
>
>
>> Hello all
>>
>> It seems that there is agreement that some sort of automated
>> testing is "a good thing" ;-).
>>
>
> I am a huge fan of testing and think that qemu developers and users
> would both benefit from more automated testing.
>
>
>> - several qemu images (one with Debian, one with Windows XP - I
>> can get a free student license for it, etc)
>>
>
> This is testing of the final product. Especially for something as
> complex as qemu, final product testing is difficult. Wherever possible
> units should be tested in isolation. Debugging test failures of unit
> tests are also far easier than debugging problems in a whole qemu
> VM.
>
> However, if you do pursue your original plan, it would be really nice
> if either WinXP 64bit or Vista64 could be added to this list.
>
http://kvm.qumranet.com/kvmwiki/KVMTest
I use that today to automatically test Windows installs (including
Vista). It's designed for KVM but since the userspace for KVM and QEMU
have identical interfaces, it should Just Work with QEMU.
Regards,
Anthony Liguori
> Cheers,
> Erik
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 21:50 ` Erik de Castro Lopo
2008-06-03 21:53 ` Anthony Liguori
@ 2008-06-03 22:02 ` Paul Brook
2008-06-03 22:05 ` Glauber Costa
1 sibling, 1 reply; 18+ messages in thread
From: Paul Brook @ 2008-06-03 22:02 UTC (permalink / raw)
To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
On Tuesday 03 June 2008, Erik de Castro Lopo wrote:
> Balazs Attila-Mihaly (Cd-MaN) wrote:
> > Hello all
> >
> > It seems that there is agreement that some sort of automated
> > testing is "a good thing" ;-).
>
> I am a huge fan of testing and think that qemu developers and users
> would both benefit from more automated testing.
IMHO Automated testing by itself is pretty much worthless.
The value comes from having someone look at the results, and actively fix
problems as they are discovered. Once you've allocated resources to do this
bugfixing setting up the testing is fairly trivial.
Paul
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:02 ` Paul Brook
@ 2008-06-03 22:05 ` Glauber Costa
2008-06-03 22:17 ` Anthony Liguori
2008-06-03 22:25 ` Paul Brook
0 siblings, 2 replies; 18+ messages in thread
From: Glauber Costa @ 2008-06-03 22:05 UTC (permalink / raw)
To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote:
> On Tuesday 03 June 2008, Erik de Castro Lopo wrote:
>> Balazs Attila-Mihaly (Cd-MaN) wrote:
>> > Hello all
>> >
>> > It seems that there is agreement that some sort of automated
>> > testing is "a good thing" ;-).
>>
>> I am a huge fan of testing and think that qemu developers and users
>> would both benefit from more automated testing.
>
> IMHO Automated testing by itself is pretty much worthless.
> The value comes from having someone look at the results, and actively fix
> problems as they are discovered. Once you've allocated resources to do this
> bugfixing setting up the testing is fairly trivial.
Not at all. A developer writing something new for qemu will have a way
to make sure his code works before submitting it upstream.
Right now, each one has to write its own testing, each time, which can
be failed in itself, and not do a full coverage.
So while it may not do much for existing code, it can certainly help
the quality of new code to improve.
--
Glauber Costa.
"Free as in Freedom"
http://glommer.net
"The less confident you are, the more serious you have to act."
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:05 ` Glauber Costa
@ 2008-06-03 22:17 ` Anthony Liguori
2008-06-03 22:25 ` Paul Brook
1 sibling, 0 replies; 18+ messages in thread
From: Anthony Liguori @ 2008-06-03 22:17 UTC (permalink / raw)
To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
Glauber Costa wrote:
> On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote:
>
>> On Tuesday 03 June 2008, Erik de Castro Lopo wrote:
>>
>>> Balazs Attila-Mihaly (Cd-MaN) wrote:
>>>
>>>> Hello all
>>>>
>>>> It seems that there is agreement that some sort of automated
>>>> testing is "a good thing" ;-).
>>>>
>>> I am a huge fan of testing and think that qemu developers and users
>>> would both benefit from more automated testing.
>>>
>> IMHO Automated testing by itself is pretty much worthless.
>> The value comes from having someone look at the results, and actively fix
>> problems as they are discovered. Once you've allocated resources to do this
>> bugfixing setting up the testing is fairly trivial.
>>
>
> Not at all. A developer writing something new for qemu will have a way
> to make sure his code works before submitting it upstream.
>
Test cases are great. I think the question is how useful is a fully
automated testing suite that runs continuously. Without people
dedicating time to track down all of the various failures, it tends to
suffer the same fate as a poorly maintained bugzilla.
But yeah, we want reproducible test cases. That's what KVMTest is all
about--automating things that normally require user intervention.
Regards,
Anthony Liguori
> Right now, each one has to write its own testing, each time, which can
> be failed in itself, and not do a full coverage.
>
> So while it may not do much for existing code, it can certainly help
> the quality of new code to improve.
>
>
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:05 ` Glauber Costa
2008-06-03 22:17 ` Anthony Liguori
@ 2008-06-03 22:25 ` Paul Brook
2008-06-03 22:35 ` Thiemo Seufer
` (2 more replies)
1 sibling, 3 replies; 18+ messages in thread
From: Paul Brook @ 2008-06-03 22:25 UTC (permalink / raw)
To: qemu-devel
Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
On Tuesday 03 June 2008, Glauber Costa wrote:
> On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote:
> > On Tuesday 03 June 2008, Erik de Castro Lopo wrote:
> >> Balazs Attila-Mihaly (Cd-MaN) wrote:
> >> > Hello all
> >> >
> >> > It seems that there is agreement that some sort of automated
> >> > testing is "a good thing" ;-).
> >>
> >> I am a huge fan of testing and think that qemu developers and users
> >> would both benefit from more automated testing.
> >
> > IMHO Automated testing by itself is pretty much worthless.
> > The value comes from having someone look at the results, and actively fix
> > problems as they are discovered. Once you've allocated resources to do
> > this bugfixing setting up the testing is fairly trivial.
>
> Not at all. A developer writing something new for qemu will have a way
> to make sure his code works before submitting it upstream.
> Right now, each one has to write its own testing, each time, which can
> be failed in itself, and not do a full coverage.
We were talking about a tester that does periodic long running tests off svn
trunk, and reports the results. Individual developers are not directly
involved.
You're talking about some sort of testsuite that can be distributed to all
developers and reasonably run before every patch is submitted, which is a
significantly different beast.
I'm pretty certain the proposed tests would not be suitable for routine use by
the majority of developers are part of normal developers. They will be too
large, probably take a long time to run, and contain proprietary software
that can't be redistributed.
Paul
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:25 ` Paul Brook
@ 2008-06-03 22:35 ` Thiemo Seufer
2008-06-04 6:41 ` Laurent Desnogues
2008-06-04 10:28 ` Avi Kivity
2008-06-05 10:13 ` Ian Jackson
2 siblings, 1 reply; 18+ messages in thread
From: Thiemo Seufer @ 2008-06-03 22:35 UTC (permalink / raw)
To: qemu-devel
Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
Paul Brook wrote:
> On Tuesday 03 June 2008, Glauber Costa wrote:
> > On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote:
> > > On Tuesday 03 June 2008, Erik de Castro Lopo wrote:
> > >> Balazs Attila-Mihaly (Cd-MaN) wrote:
> > >> > Hello all
> > >> >
> > >> > It seems that there is agreement that some sort of automated
> > >> > testing is "a good thing" ;-).
> > >>
> > >> I am a huge fan of testing and think that qemu developers and users
> > >> would both benefit from more automated testing.
> > >
> > > IMHO Automated testing by itself is pretty much worthless.
> > > The value comes from having someone look at the results, and actively fix
> > > problems as they are discovered. Once you've allocated resources to do
> > > this bugfixing setting up the testing is fairly trivial.
> >
> > Not at all. A developer writing something new for qemu will have a way
> > to make sure his code works before submitting it upstream.
> > Right now, each one has to write its own testing, each time, which can
> > be failed in itself, and not do a full coverage.
>
> We were talking about a tester that does periodic long running tests off svn
> trunk, and reports the results. Individual developers are not directly
> involved.
>
> You're talking about some sort of testsuite that can be distributed to all
> developers and reasonably run before every patch is submitted, which is a
> significantly different beast.
>
> I'm pretty certain the proposed tests would not be suitable for routine use by
> the majority of developers are part of normal developers. They will be too
> large, probably take a long time to run, and contain proprietary software
> that can't be redistributed.
OTOH, something like the CRIS testsuite (which was ported from GNU sim)
for other target architectures could be very useful.
Thiemo
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:35 ` Thiemo Seufer
@ 2008-06-04 6:41 ` Laurent Desnogues
2008-06-04 7:49 ` Laurent Desnogues
2008-06-04 9:44 ` Edgar E. Iglesias
0 siblings, 2 replies; 18+ messages in thread
From: Laurent Desnogues @ 2008-06-04 6:41 UTC (permalink / raw)
To: qemu-devel
On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote:
> OTOH, something like the CRIS testsuite (which was ported from GNU sim)
> for other target architectures could be very useful.
That would certainly be something very valuable but it would
need a tremendous amount of work and would have to be
done by a different person than the original coder (or else
you could not catch misunderstandings of the instruction
descriptions for instance).
I would also like to point out that non ia32/x86_64 targets
tests were not covered by the discussion. This probably
represents a small percentage of qemu usage, but it
definitely needs some kind of automated testing too.
Laurent
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-04 6:41 ` Laurent Desnogues
@ 2008-06-04 7:49 ` Laurent Desnogues
2008-06-04 9:44 ` Edgar E. Iglesias
1 sibling, 0 replies; 18+ messages in thread
From: Laurent Desnogues @ 2008-06-04 7:49 UTC (permalink / raw)
To: qemu-devel
On Wed, Jun 4, 2008 at 8:41 AM, Laurent Desnogues
<laurent.desnogues@gmail.com> wrote:
> On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote:
>> OTOH, something like the CRIS testsuite (which was ported from GNU sim)
>> for other target architectures could be very useful.
>
> That would certainly be something very valuable but it would
> need a tremendous amount of work and would have to be
> done by a different person than the original coder (or else
> you could not catch misunderstandings of the instruction
> descriptions for instance).
>
> I would also like to point out that non ia32/x86_64 targets
I meant "hosts" instead of "targets"...
> tests were not covered by the discussion. This probably
> represents a small percentage of qemu usage, but it
> definitely needs some kind of automated testing too.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-04 6:41 ` Laurent Desnogues
2008-06-04 7:49 ` Laurent Desnogues
@ 2008-06-04 9:44 ` Edgar E. Iglesias
1 sibling, 0 replies; 18+ messages in thread
From: Edgar E. Iglesias @ 2008-06-04 9:44 UTC (permalink / raw)
To: Laurent Desnogues; +Cc: qemu-devel
On Wed, Jun 04, 2008 at 08:41:58AM +0200, Laurent Desnogues wrote:
> On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote:
> > OTOH, something like the CRIS testsuite (which was ported from GNU sim)
> > for other target architectures could be very useful.
>
> That would certainly be something very valuable but it would
> need a tremendous amount of work and would have to be
> done by a different person than the original coder (or else
> you could not catch misunderstandings of the instruction
> descriptions for instance).
It's good if someone else writes the tests but I don't feel it's that critical for qemu. The tests certainly dont _have_ to be written by others to have great value. I agree that it might require a bit of work but luckily it can be done incrementally.
An annoying thing with the CRIS testsuite (the way it runs now) is that it requires the CRIS compiler to build the test-cases before they can run. It would be nice to have a check-notools rule (or similar) that would download a pre-compiled testsuite so anyone could run it. I don't think we can expect developers to have toolchains for all the supported targets and to keep them all up to date as new testcases are added which may require modern tool versions.
Best regards
--
Edgar E. Iglesias
Axis Communications AB
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:25 ` Paul Brook
2008-06-03 22:35 ` Thiemo Seufer
@ 2008-06-04 10:28 ` Avi Kivity
2008-06-04 12:32 ` Glauber Costa
2008-06-05 10:13 ` Ian Jackson
2 siblings, 1 reply; 18+ messages in thread
From: Avi Kivity @ 2008-06-04 10:28 UTC (permalink / raw)
To: qemu-devel
Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN)
Paul Brook wrote:
> We were talking about a tester that does periodic long running tests off svn
> trunk, and reports the results. Individual developers are not directly
> involved.
>
>
Even that is tremendously useful. Once you identify a regression, it is
easy to bisect and pinpoint the offending patch, which also locates the
author.
> You're talking about some sort of testsuite that can be distributed to all
> developers and reasonably run before every patch is submitted, which is a
> significantly different beast.
>
> I'm pretty certain the proposed tests would not be suitable for routine use by
> the majority of developers are part of normal developers. They will be too
> large, probably take a long time to run, and contain proprietary software
> that can't be redistributed.
>
There's no need to start at the screen while the test is running.
Proprietary software can be worked around by having the user provide a
CD image and licensing keys in a configuration file.
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN)
2008-06-03 21:50 ` Erik de Castro Lopo
@ 2008-06-04 10:36 ` Fabrice Bellard
2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil
2 siblings, 0 replies; 18+ messages in thread
From: Fabrice Bellard @ 2008-06-04 10:36 UTC (permalink / raw)
To: qemu-devel
Balazs Attila-Mihaly (Cd-MaN) wrote:
> Hello all
>
> It seems that there is agreement that some sort of automated testing
> is "a good thing" ;-). I'll have some free time in the next couple of
> days and plan on throwing something like this together on a spare
> box. I was thinking along the lines:
>
> - several qemu images (one with Debian, one with Windows XP - I can
> get a free student license for it, etc) - a script does a checkout of
> the trunk, checks if the version number is different from the last
> checkout (to avoid spamming the list :-)) - the script introduces the
> source in each VM, starts the VM and lets the different compilers
> available in the VM (like gcc 3.3, 3.4, mingw) compile the source -
> if the compile fails, it collects the error logs - if the compile
> succeeds, performance and functionality tests are run with the
> resulting binary - the is the most nebulous part for the moment for
> me - if I recall Fabrice said that compiling something inside a VM is
> a good performance test... - results are sumitted to the list - if
> you are ok with that, I wouldn't want to spam the list
>
> Please comment if you find the testing methodology good and what
> performance and functionality test should the process include...
I want to do such automated testing and performance measurements since a
long time, but never found the time to do them. My idea was close to
what you suggest, i.e:
Every day:
- do a checkout
- compile all the versions on i386 and x86_64
- publish the binary packages
- run the linux-user tests with the user emulators
- run some of the disk images with the system emulators.
- run benchmarks (gcc compilation, nbench)
- install and test some commercial OSes (this part cannot be
redistributed of course)
The results should be published on a web site, not on the mailing list.
Fabrice.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-04 10:28 ` Avi Kivity
@ 2008-06-04 12:32 ` Glauber Costa
2008-06-04 12:36 ` Avi Kivity
0 siblings, 1 reply; 18+ messages in thread
From: Glauber Costa @ 2008-06-04 12:32 UTC (permalink / raw)
To: Avi Kivity; +Cc: Erik de Castro Lopo, qemu-devel, Balazs Attila-Mihaly (Cd-MaN)
On Wed, Jun 4, 2008 at 7:28 AM, Avi Kivity <avi@qumranet.com> wrote:
> Paul Brook wrote:
>>
>> We were talking about a tester that does periodic long running tests off
>> svn trunk, and reports the results. Individual developers are not directly
>> involved.
>>
>>
>
> Even that is tremendously useful. Once you identify a regression, it is
> easy to bisect and pinpoint the offending patch, which also locates the
> author.
Not that easy, regarding bisection. Qemu does not have an as strict
commit policy as the linux kernel, and you bet most of the commits
won't build fine by their own.
>
>> You're talking about some sort of testsuite that can be distributed to all
>> developers and reasonably run before every patch is submitted, which is a
>> significantly different beast.
>>
>> I'm pretty certain the proposed tests would not be suitable for routine
>> use by the majority of developers are part of normal developers. They will
>> be too large, probably take a long time to run, and contain proprietary
>> software that can't be redistributed.
>>
>
> There's no need to start at the screen while the test is running.
> Proprietary software can be worked around by having the user provide a CD
> image and licensing keys in a configuration file.
>
That's a good solution, but we do have an awful lot of Free Software
that covers a wide range of qemu targets, right? So even if one does
not have, or does not want to run a CD, or so, it's still better than
nothing at all
--
Glauber Costa.
"Free as in Freedom"
http://glommer.net
"The less confident you are, the more serious you have to act."
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-04 12:32 ` Glauber Costa
@ 2008-06-04 12:36 ` Avi Kivity
0 siblings, 0 replies; 18+ messages in thread
From: Avi Kivity @ 2008-06-04 12:36 UTC (permalink / raw)
To: Glauber Costa
Cc: Erik de Castro Lopo, qemu-devel, Balazs Attila-Mihaly (Cd-MaN)
Glauber Costa wrote:
>> Even that is tremendously useful. Once you identify a regression, it is
>> easy to bisect and pinpoint the offending patch, which also locates the
>> author.
>>
>
> Not that easy, regarding bisection. Qemu does not have an as strict
> commit policy as the linux kernel, and you bet most of the commits
> won't build fine by their own.
>
If we have such a testsuite, the motivation for bisect friendly commits
will increase.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-03 22:25 ` Paul Brook
2008-06-03 22:35 ` Thiemo Seufer
2008-06-04 10:28 ` Avi Kivity
@ 2008-06-05 10:13 ` Ian Jackson
2008-06-05 12:42 ` Avi Kivity
2 siblings, 1 reply; 18+ messages in thread
From: Ian Jackson @ 2008-06-05 10:13 UTC (permalink / raw)
To: qemu-devel
Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"):
> We were talking about a tester that does periodic long running tests off svn
> trunk, and reports the results. Individual developers are not directly
> involved.
Just reporting the results is all very well, but as others have said
testing is only useful with a very firm commitment to deal with the
results of reports. Few Free Software projects can manage to do this
without some kind of formal and automatic linkage of QA passes into
code distribution or propagation. Without that, continuous discipline
is needed - and if it ever slips, the test failures become an
ever-deepening swamp.
One common approach to this problem used with success in very
different ways by both Debian and Xen and probably many others, is to
maintain parallel branches: one (call it `unstable') is changed
freely, and the other (call it `testing') is updated from unstable
only when the QA criteria (whatever those are) are met.
If there is some reason why the testing version has to be up to date,
this naturally leads to developers working on fixing test failures
because they end up blocking on it. In Debian this is achieved (more
or less) by the fact that `testing' will form the basis of the next
release and is the only route to wide deployment; in Xen the testing
tree (called `xen-unstable') is actually the one that's widely
publicised and against which all submitters are expected to submit
patches.
I'm not sure exactly what would be the best corresponding model for
Qemu. One idea might be to decree that while the testing tree is
broken, changes other than ones to fix it are forbidden.
There is one other unfortunate problem with the scheme above: this
mode of working is much harder to support without a version control
system that does merge-tracking. With such a vcs, everything can be
largely automatic, including adjusting patches submitted against
testing to apply to unstable. With svn such a workflow would be
nearly impossible.
Ian.
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework
2008-06-05 10:13 ` Ian Jackson
@ 2008-06-05 12:42 ` Avi Kivity
0 siblings, 0 replies; 18+ messages in thread
From: Avi Kivity @ 2008-06-05 12:42 UTC (permalink / raw)
To: qemu-devel
Ian Jackson wrote:
> Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"):
>
>> We were talking about a tester that does periodic long running tests off svn
>> trunk, and reports the results. Individual developers are not directly
>> involved.
>>
>
> Just reporting the results is all very well, but as others have said
> testing is only useful with a very firm commitment to deal with the
> results of reports. Few Free Software projects can manage to do this
> without some kind of formal and automatic linkage of QA passes into
> code distribution or propagation. Without that, continuous discipline
> is needed - and if it ever slips, the test failures become an
> ever-deepening swamp.
>
> One common approach to this problem used with success in very
> different ways by both Debian and Xen and probably many others, is to
> maintain parallel branches: one (call it `unstable') is changed
> freely, and the other (call it `testing') is updated from unstable
> only when the QA criteria (whatever those are) are met.
>
IMO parallel branches are overkill, especially given the current lack of
reviewing and committing.
I'd suggest having just two rules:
- patch authors make a reasonable effort to ensure no regression in
compilation or execution; this does not include running the entire test
suite
- if a regression is discovered by the test suite, the offending patch
is either fixed in a short time or reverted
This way progress can still be made, but quality is maintained
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel][Patch] Suggestion for testing framework
2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN)
2008-06-03 21:50 ` Erik de Castro Lopo
2008-06-04 10:36 ` Fabrice Bellard
@ 2008-06-15 18:52 ` Stefan Weil
2 siblings, 0 replies; 18+ messages in thread
From: Stefan Weil @ 2008-06-15 18:52 UTC (permalink / raw)
To: QEMU Developers
[-- Attachment #1.1: Type: text/plain, Size: 3078 bytes --]
Balazs Attila-Mihaly (Cd-MaN) schrieb:
> Hello all
>
> It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines:
>
> - several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc)
> - a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-))
> - the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source
> - if the compile fails, it collects the error logs
> - if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test...
> - results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list
>
> Please comment if you find the testing methodology good and what performance and functionality test should the process include...
>
> Best regards.
Hi,
using an existing framework for continuous integration tests will
provide many of the
features needed. Here is an example for an automated mail (which might
be sent to
the Qemu mailing list). I used CruiseControl, a Java based framework. The
configuration and a helper script are appended to this mail and could be
added to
Qemu trunk, so everyone can run his/her own continuous integration.
The current configuration is just a simple compile test for all targets.
It can be extended to add more information to the mail, to inform those who
changed the code about success or failure, to run a web interface,
to cross compile, to run tests...
Of course, running the continuous integration on a server with access
from the
Internet would be better than running it on my private machine, but even my
private machine could send mails to the list when something goes wrong.
I tried to run CI on a virtual server but it did not have enough
resources to run Java...
Regards
Stefan
Example of mail sent by CruiseControl:
View results here -> http://localhost/cruisecontrol/buildresults/qemu
trunk x86?log=log20080615200821Lbuild.7
<http://localhost/cruisecontrol/buildresults/qemu%20trunk%20x86?log=log20080615200821Lbuild.7>
BUILD COMPLETE - build.7
Date of build: 06/15/2008 20:08:21
Time to build: 3 minute(s) 11 second(s)
Last changed: 06/15/2008 20:06:39
Last log entry: Avoid temporary variable use across basic blocks for udivx
Unit Tests: (0)
No Tests Run
This project doesn't have any tests
Modifications since last successful build: (2)
modified blueswir1 /trunk/target-sparc/translate.c 06/15/2008
20:06:39 Avoid temporary variable use across basic blocks for udivx
modified blueswir1 /trunk/linux-user/main.c 06/15/2008 20:02:48 Fix
Sparc32plus & Sparc64 debug output
[-- Attachment #1.2: Type: text/html, Size: 8913 bytes --]
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: cruisecontrol.patch --]
[-- Type: text/x-diff; name="cruisecontrol.patch", Size: 3157 bytes --]
Index: tests/cruisecontrol/config.xml
===================================================================
--- tests/cruisecontrol/config.xml (Revision 0)
+++ tests/cruisecontrol/config.xml (Revision 0)
@@ -0,0 +1,71 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+CruiseControl configuration file for continuous integration tests of Qemu.
+-->
+
+<cruisecontrol>
+ <!-- fix next two lines for your installation -->
+ <property name="srcdir" value="/home/stefan/src/qemu-trunk/trunk" />
+ <property name="mailto" value="name@domain.de" />
+
+ <!-- Native compilation with default configuration. -->
+ <project name="qemu trunk x86" buildafterfailed="false">
+ <listeners>
+ <currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
+ </listeners>
+ <bootstrappers>
+ <svnbootstrapper localworkingcopy="${srcdir}" />
+ </bootstrappers>
+ <modificationset quietperiod="0">
+ <svn localworkingcopy="${srcdir}" uselocalrevision="true" />
+ </modificationset>
+ <schedule interval="600">
+ <exec
+ command="tests/cruisecontrol/cc-build.sh"
+ workingdir="${srcdir}"
+ errorstr="build failed"
+ />
+ </schedule>
+ <publishers>
+ <artifactspublisher file="${srcdir}/bin/native/make.log" dest="artifacts/${project.name}" />
+ <htmlemail
+ mailhost="localhost"
+ reportsuccess="always"
+ returnaddress="qemu-devel@nongnu.org"
+ returnname="QEMU Developers CruiseControl"
+ subjectprefix="[Qemu-devel][CruiseControl]"
+ skipusers="true"
+ buildresultsurl="http://localhost/cruisecontrol/buildresults/${project.name}"
+ logdir="logs/${project.name}"
+ >
+ <always address="${mailto}" />
+ </htmlemail>
+
+ </publishers>
+ </project>
+
+ <!-- Native compilation (64 bit) with default configuration. -->
+<!--
+ <project name="qemu trunk amd64" buildafterfailed="false">
+ </project>
+-->
+
+ <!-- Some tests. Might also be added to one of the compilation projects. -->
+<!--
+ <project name="qemu trunk tests" buildafterfailed="false">
+ </project>
+-->
+
+ <!-- Cross compilation (Win32) with default configuration. -->
+<!--
+ <project name="qemu trunk win32" buildafterfailed="false">
+ </project>
+-->
+
+ <!-- Cross compilation (Mips) with default configuration. -->
+ <!-- Not working because of missing tcg support for mips host. -->
+<!--
+ <project name="qemu trunk mips" buildafterfailed="false">
+ </project>
+-->
+</cruisecontrol>
Index: tests/cruisecontrol/cc-build.sh
===================================================================
--- tests/cruisecontrol/cc-build.sh (Revision 0)
+++ tests/cruisecontrol/cc-build.sh (Revision 0)
@@ -0,0 +1,19 @@
+#!/bin/sh
+
+# $Id$
+
+# Build script for Continuous Integration with Cruise Control.
+# Used by CI for Qemu.
+
+export LANG=C
+
+bindir=bin/native
+
+mkdir -p $bindir
+cd $bindir
+
+../../configure
+
+(make || echo build failed) 2>&1 | tee make.log
+
+# eof
Eigenschaftsänderungen: tests/cruisecontrol/cc-build.sh
___________________________________________________________________
Name: svn:executable
+ *
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2008-06-15 18:53 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN)
2008-06-03 21:50 ` Erik de Castro Lopo
2008-06-03 21:53 ` Anthony Liguori
2008-06-03 22:02 ` Paul Brook
2008-06-03 22:05 ` Glauber Costa
2008-06-03 22:17 ` Anthony Liguori
2008-06-03 22:25 ` Paul Brook
2008-06-03 22:35 ` Thiemo Seufer
2008-06-04 6:41 ` Laurent Desnogues
2008-06-04 7:49 ` Laurent Desnogues
2008-06-04 9:44 ` Edgar E. Iglesias
2008-06-04 10:28 ` Avi Kivity
2008-06-04 12:32 ` Glauber Costa
2008-06-04 12:36 ` Avi Kivity
2008-06-05 10:13 ` Ian Jackson
2008-06-05 12:42 ` Avi Kivity
2008-06-04 10:36 ` Fabrice Bellard
2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).