* [Qemu-devel] Suggestion for testing framework @ 2008-06-03 20:59 Balazs Attila-Mihaly (Cd-MaN) 2008-06-03 21:50 ` Erik de Castro Lopo ` (2 more replies) 0 siblings, 3 replies; 18+ messages in thread From: Balazs Attila-Mihaly (Cd-MaN) @ 2008-06-03 20:59 UTC (permalink / raw) To: Qemu Devel Hello all It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines: - several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc) - a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-)) - the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source - if the compile fails, it collects the error logs - if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test... - results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list Please comment if you find the testing methodology good and what performance and functionality test should the process include... Best regards. __________________________________________________________ Sent from Yahoo! Mail. A Smarter Email http://uk.docs.yahoo.com/nowyoucan.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN) @ 2008-06-03 21:50 ` Erik de Castro Lopo 2008-06-03 21:53 ` Anthony Liguori 2008-06-03 22:02 ` Paul Brook 2008-06-04 10:36 ` Fabrice Bellard 2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil 2 siblings, 2 replies; 18+ messages in thread From: Erik de Castro Lopo @ 2008-06-03 21:50 UTC (permalink / raw) To: qemu-devel; +Cc: Balazs Attila-Mihaly (Cd-MaN) Balazs Attila-Mihaly (Cd-MaN) wrote: > Hello all > > It seems that there is agreement that some sort of automated > testing is "a good thing" ;-). I am a huge fan of testing and think that qemu developers and users would both benefit from more automated testing. > - several qemu images (one with Debian, one with Windows XP - I > can get a free student license for it, etc) This is testing of the final product. Especially for something as complex as qemu, final product testing is difficult. Wherever possible units should be tested in isolation. Debugging test failures of unit tests are also far easier than debugging problems in a whole qemu VM. However, if you do pursue your original plan, it would be really nice if either WinXP 64bit or Vista64 could be added to this list. Cheers, Erik -- ----------------------------------------------------------------- Erik de Castro Lopo ----------------------------------------------------------------- "Re graphics: A picture is worth 10K words - but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures." -- Alan Perlis ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 21:50 ` Erik de Castro Lopo @ 2008-06-03 21:53 ` Anthony Liguori 2008-06-03 22:02 ` Paul Brook 1 sibling, 0 replies; 18+ messages in thread From: Anthony Liguori @ 2008-06-03 21:53 UTC (permalink / raw) To: qemu-devel; +Cc: Balazs Attila-Mihaly (Cd-MaN) Erik de Castro Lopo wrote: > Balazs Attila-Mihaly (Cd-MaN) wrote: > > >> Hello all >> >> It seems that there is agreement that some sort of automated >> testing is "a good thing" ;-). >> > > I am a huge fan of testing and think that qemu developers and users > would both benefit from more automated testing. > > >> - several qemu images (one with Debian, one with Windows XP - I >> can get a free student license for it, etc) >> > > This is testing of the final product. Especially for something as > complex as qemu, final product testing is difficult. Wherever possible > units should be tested in isolation. Debugging test failures of unit > tests are also far easier than debugging problems in a whole qemu > VM. > > However, if you do pursue your original plan, it would be really nice > if either WinXP 64bit or Vista64 could be added to this list. > http://kvm.qumranet.com/kvmwiki/KVMTest I use that today to automatically test Windows installs (including Vista). It's designed for KVM but since the userspace for KVM and QEMU have identical interfaces, it should Just Work with QEMU. Regards, Anthony Liguori > Cheers, > Erik > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 21:50 ` Erik de Castro Lopo 2008-06-03 21:53 ` Anthony Liguori @ 2008-06-03 22:02 ` Paul Brook 2008-06-03 22:05 ` Glauber Costa 1 sibling, 1 reply; 18+ messages in thread From: Paul Brook @ 2008-06-03 22:02 UTC (permalink / raw) To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) On Tuesday 03 June 2008, Erik de Castro Lopo wrote: > Balazs Attila-Mihaly (Cd-MaN) wrote: > > Hello all > > > > It seems that there is agreement that some sort of automated > > testing is "a good thing" ;-). > > I am a huge fan of testing and think that qemu developers and users > would both benefit from more automated testing. IMHO Automated testing by itself is pretty much worthless. The value comes from having someone look at the results, and actively fix problems as they are discovered. Once you've allocated resources to do this bugfixing setting up the testing is fairly trivial. Paul ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:02 ` Paul Brook @ 2008-06-03 22:05 ` Glauber Costa 2008-06-03 22:17 ` Anthony Liguori 2008-06-03 22:25 ` Paul Brook 0 siblings, 2 replies; 18+ messages in thread From: Glauber Costa @ 2008-06-03 22:05 UTC (permalink / raw) To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote: > On Tuesday 03 June 2008, Erik de Castro Lopo wrote: >> Balazs Attila-Mihaly (Cd-MaN) wrote: >> > Hello all >> > >> > It seems that there is agreement that some sort of automated >> > testing is "a good thing" ;-). >> >> I am a huge fan of testing and think that qemu developers and users >> would both benefit from more automated testing. > > IMHO Automated testing by itself is pretty much worthless. > The value comes from having someone look at the results, and actively fix > problems as they are discovered. Once you've allocated resources to do this > bugfixing setting up the testing is fairly trivial. Not at all. A developer writing something new for qemu will have a way to make sure his code works before submitting it upstream. Right now, each one has to write its own testing, each time, which can be failed in itself, and not do a full coverage. So while it may not do much for existing code, it can certainly help the quality of new code to improve. -- Glauber Costa. "Free as in Freedom" http://glommer.net "The less confident you are, the more serious you have to act." ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:05 ` Glauber Costa @ 2008-06-03 22:17 ` Anthony Liguori 2008-06-03 22:25 ` Paul Brook 1 sibling, 0 replies; 18+ messages in thread From: Anthony Liguori @ 2008-06-03 22:17 UTC (permalink / raw) To: qemu-devel; +Cc: Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) Glauber Costa wrote: > On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote: > >> On Tuesday 03 June 2008, Erik de Castro Lopo wrote: >> >>> Balazs Attila-Mihaly (Cd-MaN) wrote: >>> >>>> Hello all >>>> >>>> It seems that there is agreement that some sort of automated >>>> testing is "a good thing" ;-). >>>> >>> I am a huge fan of testing and think that qemu developers and users >>> would both benefit from more automated testing. >>> >> IMHO Automated testing by itself is pretty much worthless. >> The value comes from having someone look at the results, and actively fix >> problems as they are discovered. Once you've allocated resources to do this >> bugfixing setting up the testing is fairly trivial. >> > > Not at all. A developer writing something new for qemu will have a way > to make sure his code works before submitting it upstream. > Test cases are great. I think the question is how useful is a fully automated testing suite that runs continuously. Without people dedicating time to track down all of the various failures, it tends to suffer the same fate as a poorly maintained bugzilla. But yeah, we want reproducible test cases. That's what KVMTest is all about--automating things that normally require user intervention. Regards, Anthony Liguori > Right now, each one has to write its own testing, each time, which can > be failed in itself, and not do a full coverage. > > So while it may not do much for existing code, it can certainly help > the quality of new code to improve. > > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:05 ` Glauber Costa 2008-06-03 22:17 ` Anthony Liguori @ 2008-06-03 22:25 ` Paul Brook 2008-06-03 22:35 ` Thiemo Seufer ` (2 more replies) 1 sibling, 3 replies; 18+ messages in thread From: Paul Brook @ 2008-06-03 22:25 UTC (permalink / raw) To: qemu-devel Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) On Tuesday 03 June 2008, Glauber Costa wrote: > On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote: > > On Tuesday 03 June 2008, Erik de Castro Lopo wrote: > >> Balazs Attila-Mihaly (Cd-MaN) wrote: > >> > Hello all > >> > > >> > It seems that there is agreement that some sort of automated > >> > testing is "a good thing" ;-). > >> > >> I am a huge fan of testing and think that qemu developers and users > >> would both benefit from more automated testing. > > > > IMHO Automated testing by itself is pretty much worthless. > > The value comes from having someone look at the results, and actively fix > > problems as they are discovered. Once you've allocated resources to do > > this bugfixing setting up the testing is fairly trivial. > > Not at all. A developer writing something new for qemu will have a way > to make sure his code works before submitting it upstream. > Right now, each one has to write its own testing, each time, which can > be failed in itself, and not do a full coverage. We were talking about a tester that does periodic long running tests off svn trunk, and reports the results. Individual developers are not directly involved. You're talking about some sort of testsuite that can be distributed to all developers and reasonably run before every patch is submitted, which is a significantly different beast. I'm pretty certain the proposed tests would not be suitable for routine use by the majority of developers are part of normal developers. They will be too large, probably take a long time to run, and contain proprietary software that can't be redistributed. Paul ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:25 ` Paul Brook @ 2008-06-03 22:35 ` Thiemo Seufer 2008-06-04 6:41 ` Laurent Desnogues 2008-06-04 10:28 ` Avi Kivity 2008-06-05 10:13 ` Ian Jackson 2 siblings, 1 reply; 18+ messages in thread From: Thiemo Seufer @ 2008-06-03 22:35 UTC (permalink / raw) To: qemu-devel Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) Paul Brook wrote: > On Tuesday 03 June 2008, Glauber Costa wrote: > > On Tue, Jun 3, 2008 at 7:02 PM, Paul Brook <paul@codesourcery.com> wrote: > > > On Tuesday 03 June 2008, Erik de Castro Lopo wrote: > > >> Balazs Attila-Mihaly (Cd-MaN) wrote: > > >> > Hello all > > >> > > > >> > It seems that there is agreement that some sort of automated > > >> > testing is "a good thing" ;-). > > >> > > >> I am a huge fan of testing and think that qemu developers and users > > >> would both benefit from more automated testing. > > > > > > IMHO Automated testing by itself is pretty much worthless. > > > The value comes from having someone look at the results, and actively fix > > > problems as they are discovered. Once you've allocated resources to do > > > this bugfixing setting up the testing is fairly trivial. > > > > Not at all. A developer writing something new for qemu will have a way > > to make sure his code works before submitting it upstream. > > Right now, each one has to write its own testing, each time, which can > > be failed in itself, and not do a full coverage. > > We were talking about a tester that does periodic long running tests off svn > trunk, and reports the results. Individual developers are not directly > involved. > > You're talking about some sort of testsuite that can be distributed to all > developers and reasonably run before every patch is submitted, which is a > significantly different beast. > > I'm pretty certain the proposed tests would not be suitable for routine use by > the majority of developers are part of normal developers. They will be too > large, probably take a long time to run, and contain proprietary software > that can't be redistributed. OTOH, something like the CRIS testsuite (which was ported from GNU sim) for other target architectures could be very useful. Thiemo ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:35 ` Thiemo Seufer @ 2008-06-04 6:41 ` Laurent Desnogues 2008-06-04 7:49 ` Laurent Desnogues 2008-06-04 9:44 ` Edgar E. Iglesias 0 siblings, 2 replies; 18+ messages in thread From: Laurent Desnogues @ 2008-06-04 6:41 UTC (permalink / raw) To: qemu-devel On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote: > OTOH, something like the CRIS testsuite (which was ported from GNU sim) > for other target architectures could be very useful. That would certainly be something very valuable but it would need a tremendous amount of work and would have to be done by a different person than the original coder (or else you could not catch misunderstandings of the instruction descriptions for instance). I would also like to point out that non ia32/x86_64 targets tests were not covered by the discussion. This probably represents a small percentage of qemu usage, but it definitely needs some kind of automated testing too. Laurent ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-04 6:41 ` Laurent Desnogues @ 2008-06-04 7:49 ` Laurent Desnogues 2008-06-04 9:44 ` Edgar E. Iglesias 1 sibling, 0 replies; 18+ messages in thread From: Laurent Desnogues @ 2008-06-04 7:49 UTC (permalink / raw) To: qemu-devel On Wed, Jun 4, 2008 at 8:41 AM, Laurent Desnogues <laurent.desnogues@gmail.com> wrote: > On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote: >> OTOH, something like the CRIS testsuite (which was ported from GNU sim) >> for other target architectures could be very useful. > > That would certainly be something very valuable but it would > need a tremendous amount of work and would have to be > done by a different person than the original coder (or else > you could not catch misunderstandings of the instruction > descriptions for instance). > > I would also like to point out that non ia32/x86_64 targets I meant "hosts" instead of "targets"... > tests were not covered by the discussion. This probably > represents a small percentage of qemu usage, but it > definitely needs some kind of automated testing too. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-04 6:41 ` Laurent Desnogues 2008-06-04 7:49 ` Laurent Desnogues @ 2008-06-04 9:44 ` Edgar E. Iglesias 1 sibling, 0 replies; 18+ messages in thread From: Edgar E. Iglesias @ 2008-06-04 9:44 UTC (permalink / raw) To: Laurent Desnogues; +Cc: qemu-devel On Wed, Jun 04, 2008 at 08:41:58AM +0200, Laurent Desnogues wrote: > On Wed, Jun 4, 2008 at 12:35 AM, Thiemo Seufer <ths@networkno.de> wrote: > > OTOH, something like the CRIS testsuite (which was ported from GNU sim) > > for other target architectures could be very useful. > > That would certainly be something very valuable but it would > need a tremendous amount of work and would have to be > done by a different person than the original coder (or else > you could not catch misunderstandings of the instruction > descriptions for instance). It's good if someone else writes the tests but I don't feel it's that critical for qemu. The tests certainly dont _have_ to be written by others to have great value. I agree that it might require a bit of work but luckily it can be done incrementally. An annoying thing with the CRIS testsuite (the way it runs now) is that it requires the CRIS compiler to build the test-cases before they can run. It would be nice to have a check-notools rule (or similar) that would download a pre-compiled testsuite so anyone could run it. I don't think we can expect developers to have toolchains for all the supported targets and to keep them all up to date as new testcases are added which may require modern tool versions. Best regards -- Edgar E. Iglesias Axis Communications AB ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:25 ` Paul Brook 2008-06-03 22:35 ` Thiemo Seufer @ 2008-06-04 10:28 ` Avi Kivity 2008-06-04 12:32 ` Glauber Costa 2008-06-05 10:13 ` Ian Jackson 2 siblings, 1 reply; 18+ messages in thread From: Avi Kivity @ 2008-06-04 10:28 UTC (permalink / raw) To: qemu-devel Cc: Glauber Costa, Erik de Castro Lopo, Balazs Attila-Mihaly (Cd-MaN) Paul Brook wrote: > We were talking about a tester that does periodic long running tests off svn > trunk, and reports the results. Individual developers are not directly > involved. > > Even that is tremendously useful. Once you identify a regression, it is easy to bisect and pinpoint the offending patch, which also locates the author. > You're talking about some sort of testsuite that can be distributed to all > developers and reasonably run before every patch is submitted, which is a > significantly different beast. > > I'm pretty certain the proposed tests would not be suitable for routine use by > the majority of developers are part of normal developers. They will be too > large, probably take a long time to run, and contain proprietary software > that can't be redistributed. > There's no need to start at the screen while the test is running. Proprietary software can be worked around by having the user provide a CD image and licensing keys in a configuration file. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-04 10:28 ` Avi Kivity @ 2008-06-04 12:32 ` Glauber Costa 2008-06-04 12:36 ` Avi Kivity 0 siblings, 1 reply; 18+ messages in thread From: Glauber Costa @ 2008-06-04 12:32 UTC (permalink / raw) To: Avi Kivity; +Cc: Erik de Castro Lopo, qemu-devel, Balazs Attila-Mihaly (Cd-MaN) On Wed, Jun 4, 2008 at 7:28 AM, Avi Kivity <avi@qumranet.com> wrote: > Paul Brook wrote: >> >> We were talking about a tester that does periodic long running tests off >> svn trunk, and reports the results. Individual developers are not directly >> involved. >> >> > > Even that is tremendously useful. Once you identify a regression, it is > easy to bisect and pinpoint the offending patch, which also locates the > author. Not that easy, regarding bisection. Qemu does not have an as strict commit policy as the linux kernel, and you bet most of the commits won't build fine by their own. > >> You're talking about some sort of testsuite that can be distributed to all >> developers and reasonably run before every patch is submitted, which is a >> significantly different beast. >> >> I'm pretty certain the proposed tests would not be suitable for routine >> use by the majority of developers are part of normal developers. They will >> be too large, probably take a long time to run, and contain proprietary >> software that can't be redistributed. >> > > There's no need to start at the screen while the test is running. > Proprietary software can be worked around by having the user provide a CD > image and licensing keys in a configuration file. > That's a good solution, but we do have an awful lot of Free Software that covers a wide range of qemu targets, right? So even if one does not have, or does not want to run a CD, or so, it's still better than nothing at all -- Glauber Costa. "Free as in Freedom" http://glommer.net "The less confident you are, the more serious you have to act." ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-04 12:32 ` Glauber Costa @ 2008-06-04 12:36 ` Avi Kivity 0 siblings, 0 replies; 18+ messages in thread From: Avi Kivity @ 2008-06-04 12:36 UTC (permalink / raw) To: Glauber Costa Cc: Erik de Castro Lopo, qemu-devel, Balazs Attila-Mihaly (Cd-MaN) Glauber Costa wrote: >> Even that is tremendously useful. Once you identify a regression, it is >> easy to bisect and pinpoint the offending patch, which also locates the >> author. >> > > Not that easy, regarding bisection. Qemu does not have an as strict > commit policy as the linux kernel, and you bet most of the commits > won't build fine by their own. > If we have such a testsuite, the motivation for bisect friendly commits will increase. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 22:25 ` Paul Brook 2008-06-03 22:35 ` Thiemo Seufer 2008-06-04 10:28 ` Avi Kivity @ 2008-06-05 10:13 ` Ian Jackson 2008-06-05 12:42 ` Avi Kivity 2 siblings, 1 reply; 18+ messages in thread From: Ian Jackson @ 2008-06-05 10:13 UTC (permalink / raw) To: qemu-devel Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"): > We were talking about a tester that does periodic long running tests off svn > trunk, and reports the results. Individual developers are not directly > involved. Just reporting the results is all very well, but as others have said testing is only useful with a very firm commitment to deal with the results of reports. Few Free Software projects can manage to do this without some kind of formal and automatic linkage of QA passes into code distribution or propagation. Without that, continuous discipline is needed - and if it ever slips, the test failures become an ever-deepening swamp. One common approach to this problem used with success in very different ways by both Debian and Xen and probably many others, is to maintain parallel branches: one (call it `unstable') is changed freely, and the other (call it `testing') is updated from unstable only when the QA criteria (whatever those are) are met. If there is some reason why the testing version has to be up to date, this naturally leads to developers working on fixing test failures because they end up blocking on it. In Debian this is achieved (more or less) by the fact that `testing' will form the basis of the next release and is the only route to wide deployment; in Xen the testing tree (called `xen-unstable') is actually the one that's widely publicised and against which all submitters are expected to submit patches. I'm not sure exactly what would be the best corresponding model for Qemu. One idea might be to decree that while the testing tree is broken, changes other than ones to fix it are forbidden. There is one other unfortunate problem with the scheme above: this mode of working is much harder to support without a version control system that does merge-tracking. With such a vcs, everything can be largely automatic, including adjusting patches submitted against testing to apply to unstable. With svn such a workflow would be nearly impossible. Ian. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-05 10:13 ` Ian Jackson @ 2008-06-05 12:42 ` Avi Kivity 0 siblings, 0 replies; 18+ messages in thread From: Avi Kivity @ 2008-06-05 12:42 UTC (permalink / raw) To: qemu-devel Ian Jackson wrote: > Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"): > >> We were talking about a tester that does periodic long running tests off svn >> trunk, and reports the results. Individual developers are not directly >> involved. >> > > Just reporting the results is all very well, but as others have said > testing is only useful with a very firm commitment to deal with the > results of reports. Few Free Software projects can manage to do this > without some kind of formal and automatic linkage of QA passes into > code distribution or propagation. Without that, continuous discipline > is needed - and if it ever slips, the test failures become an > ever-deepening swamp. > > One common approach to this problem used with success in very > different ways by both Debian and Xen and probably many others, is to > maintain parallel branches: one (call it `unstable') is changed > freely, and the other (call it `testing') is updated from unstable > only when the QA criteria (whatever those are) are met. > IMO parallel branches are overkill, especially given the current lack of reviewing and committing. I'd suggest having just two rules: - patch authors make a reasonable effort to ensure no regression in compilation or execution; this does not include running the entire test suite - if a regression is discovered by the test suite, the offending patch is either fixed in a short time or reverted This way progress can still be made, but quality is maintained -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel] Suggestion for testing framework 2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN) 2008-06-03 21:50 ` Erik de Castro Lopo @ 2008-06-04 10:36 ` Fabrice Bellard 2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil 2 siblings, 0 replies; 18+ messages in thread From: Fabrice Bellard @ 2008-06-04 10:36 UTC (permalink / raw) To: qemu-devel Balazs Attila-Mihaly (Cd-MaN) wrote: > Hello all > > It seems that there is agreement that some sort of automated testing > is "a good thing" ;-). I'll have some free time in the next couple of > days and plan on throwing something like this together on a spare > box. I was thinking along the lines: > > - several qemu images (one with Debian, one with Windows XP - I can > get a free student license for it, etc) - a script does a checkout of > the trunk, checks if the version number is different from the last > checkout (to avoid spamming the list :-)) - the script introduces the > source in each VM, starts the VM and lets the different compilers > available in the VM (like gcc 3.3, 3.4, mingw) compile the source - > if the compile fails, it collects the error logs - if the compile > succeeds, performance and functionality tests are run with the > resulting binary - the is the most nebulous part for the moment for > me - if I recall Fabrice said that compiling something inside a VM is > a good performance test... - results are sumitted to the list - if > you are ok with that, I wouldn't want to spam the list > > Please comment if you find the testing methodology good and what > performance and functionality test should the process include... I want to do such automated testing and performance measurements since a long time, but never found the time to do them. My idea was close to what you suggest, i.e: Every day: - do a checkout - compile all the versions on i386 and x86_64 - publish the binary packages - run the linux-user tests with the user emulators - run some of the disk images with the system emulators. - run benchmarks (gcc compilation, nbench) - install and test some commercial OSes (this part cannot be redistributed of course) The results should be published on a web site, not on the mailing list. Fabrice. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Qemu-devel][Patch] Suggestion for testing framework 2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN) 2008-06-03 21:50 ` Erik de Castro Lopo 2008-06-04 10:36 ` Fabrice Bellard @ 2008-06-15 18:52 ` Stefan Weil 2 siblings, 0 replies; 18+ messages in thread From: Stefan Weil @ 2008-06-15 18:52 UTC (permalink / raw) To: QEMU Developers [-- Attachment #1.1: Type: text/plain, Size: 3078 bytes --] Balazs Attila-Mihaly (Cd-MaN) schrieb: > Hello all > > It seems that there is agreement that some sort of automated testing is "a good thing" ;-). I'll have some free time in the next couple of days and plan on throwing something like this together on a spare box. I was thinking along the lines: > > - several qemu images (one with Debian, one with Windows XP - I can get a free student license for it, etc) > - a script does a checkout of the trunk, checks if the version number is different from the last checkout (to avoid spamming the list :-)) > - the script introduces the source in each VM, starts the VM and lets the different compilers available in the VM (like gcc 3.3, 3.4, mingw) compile the source > - if the compile fails, it collects the error logs > - if the compile succeeds, performance and functionality tests are run with the resulting binary - the is the most nebulous part for the moment for me - if I recall Fabrice said that compiling something inside a VM is a good performance test... > - results are sumitted to the list - if you are ok with that, I wouldn't want to spam the list > > Please comment if you find the testing methodology good and what performance and functionality test should the process include... > > Best regards. Hi, using an existing framework for continuous integration tests will provide many of the features needed. Here is an example for an automated mail (which might be sent to the Qemu mailing list). I used CruiseControl, a Java based framework. The configuration and a helper script are appended to this mail and could be added to Qemu trunk, so everyone can run his/her own continuous integration. The current configuration is just a simple compile test for all targets. It can be extended to add more information to the mail, to inform those who changed the code about success or failure, to run a web interface, to cross compile, to run tests... Of course, running the continuous integration on a server with access from the Internet would be better than running it on my private machine, but even my private machine could send mails to the list when something goes wrong. I tried to run CI on a virtual server but it did not have enough resources to run Java... Regards Stefan Example of mail sent by CruiseControl: View results here -> http://localhost/cruisecontrol/buildresults/qemu trunk x86?log=log20080615200821Lbuild.7 <http://localhost/cruisecontrol/buildresults/qemu%20trunk%20x86?log=log20080615200821Lbuild.7> BUILD COMPLETE - build.7 Date of build: 06/15/2008 20:08:21 Time to build: 3 minute(s) 11 second(s) Last changed: 06/15/2008 20:06:39 Last log entry: Avoid temporary variable use across basic blocks for udivx Unit Tests: (0) No Tests Run This project doesn't have any tests Modifications since last successful build: (2) modified blueswir1 /trunk/target-sparc/translate.c 06/15/2008 20:06:39 Avoid temporary variable use across basic blocks for udivx modified blueswir1 /trunk/linux-user/main.c 06/15/2008 20:02:48 Fix Sparc32plus & Sparc64 debug output [-- Attachment #1.2: Type: text/html, Size: 8913 bytes --] [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: cruisecontrol.patch --] [-- Type: text/x-diff; name="cruisecontrol.patch", Size: 3157 bytes --] Index: tests/cruisecontrol/config.xml =================================================================== --- tests/cruisecontrol/config.xml (Revision 0) +++ tests/cruisecontrol/config.xml (Revision 0) @@ -0,0 +1,71 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- +CruiseControl configuration file for continuous integration tests of Qemu. +--> + +<cruisecontrol> + <!-- fix next two lines for your installation --> + <property name="srcdir" value="/home/stefan/src/qemu-trunk/trunk" /> + <property name="mailto" value="name@domain.de" /> + + <!-- Native compilation with default configuration. --> + <project name="qemu trunk x86" buildafterfailed="false"> + <listeners> + <currentbuildstatuslistener file="logs/${project.name}/status.txt"/> + </listeners> + <bootstrappers> + <svnbootstrapper localworkingcopy="${srcdir}" /> + </bootstrappers> + <modificationset quietperiod="0"> + <svn localworkingcopy="${srcdir}" uselocalrevision="true" /> + </modificationset> + <schedule interval="600"> + <exec + command="tests/cruisecontrol/cc-build.sh" + workingdir="${srcdir}" + errorstr="build failed" + /> + </schedule> + <publishers> + <artifactspublisher file="${srcdir}/bin/native/make.log" dest="artifacts/${project.name}" /> + <htmlemail + mailhost="localhost" + reportsuccess="always" + returnaddress="qemu-devel@nongnu.org" + returnname="QEMU Developers CruiseControl" + subjectprefix="[Qemu-devel][CruiseControl]" + skipusers="true" + buildresultsurl="http://localhost/cruisecontrol/buildresults/${project.name}" + logdir="logs/${project.name}" + > + <always address="${mailto}" /> + </htmlemail> + + </publishers> + </project> + + <!-- Native compilation (64 bit) with default configuration. --> +<!-- + <project name="qemu trunk amd64" buildafterfailed="false"> + </project> +--> + + <!-- Some tests. Might also be added to one of the compilation projects. --> +<!-- + <project name="qemu trunk tests" buildafterfailed="false"> + </project> +--> + + <!-- Cross compilation (Win32) with default configuration. --> +<!-- + <project name="qemu trunk win32" buildafterfailed="false"> + </project> +--> + + <!-- Cross compilation (Mips) with default configuration. --> + <!-- Not working because of missing tcg support for mips host. --> +<!-- + <project name="qemu trunk mips" buildafterfailed="false"> + </project> +--> +</cruisecontrol> Index: tests/cruisecontrol/cc-build.sh =================================================================== --- tests/cruisecontrol/cc-build.sh (Revision 0) +++ tests/cruisecontrol/cc-build.sh (Revision 0) @@ -0,0 +1,19 @@ +#!/bin/sh + +# $Id$ + +# Build script for Continuous Integration with Cruise Control. +# Used by CI for Qemu. + +export LANG=C + +bindir=bin/native + +mkdir -p $bindir +cd $bindir + +../../configure + +(make || echo build failed) 2>&1 | tee make.log + +# eof Eigenschaftsänderungen: tests/cruisecontrol/cc-build.sh ___________________________________________________________________ Name: svn:executable + * ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2008-06-15 18:53 UTC | newest] Thread overview: 18+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-06-03 20:59 [Qemu-devel] Suggestion for testing framework Balazs Attila-Mihaly (Cd-MaN) 2008-06-03 21:50 ` Erik de Castro Lopo 2008-06-03 21:53 ` Anthony Liguori 2008-06-03 22:02 ` Paul Brook 2008-06-03 22:05 ` Glauber Costa 2008-06-03 22:17 ` Anthony Liguori 2008-06-03 22:25 ` Paul Brook 2008-06-03 22:35 ` Thiemo Seufer 2008-06-04 6:41 ` Laurent Desnogues 2008-06-04 7:49 ` Laurent Desnogues 2008-06-04 9:44 ` Edgar E. Iglesias 2008-06-04 10:28 ` Avi Kivity 2008-06-04 12:32 ` Glauber Costa 2008-06-04 12:36 ` Avi Kivity 2008-06-05 10:13 ` Ian Jackson 2008-06-05 12:42 ` Avi Kivity 2008-06-04 10:36 ` Fabrice Bellard 2008-06-15 18:52 ` [Qemu-devel][Patch] " Stefan Weil
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).