[Qemu-devel] Status and RFC of patchew testings on QEMU

[Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

Hi all,

Today I've included a fourth type of the automatic patchew replies: FreeBSD.

So far we have these tests running by patchew on each patch series:

  * Docker tests
    Basically it is
        make docker-test-quick@centos6 \
             docker-test-build@min-glib \
             docker-test-mingw@fedora"

  * checkpatch.pl
    Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.

  * s390x
    It runs on a machine shared by Fedora team, basically only "./configure and
    make", because "make check" hanging is tricky to deal with from an
    automation perspective. (Ideas?)

  * FreeBSD
    Like s390x.

Q1: In the worst case, you get four individual auto replies from patchew. Is
that too many? Do you prefer one reply with all the results concatenated into
one?

Q2: Some think the full log in the mail body is more than necessary. Is it
better or worse if it is a "tail -n 200" of the log in the body and the full log
attached?

Q3: What other tests do maintainers want? Different hosts? Different configure
combinations?

Q4: Any other improvements/features you want? (E.g. some documentation? :)

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Mon, Jul 17, 2017 at 02:35:21PM +0800, Fam Zheng wrote:
> Q1: In the worst case, you get four individual auto replies from patchew. Is
> that too many? Do you prefer one reply with all the results concatenated into
> one?

I'd like to avoid situations where one of the failure emails is sent
hours after the others, because it's possible that the patch series
author has already sent the next (still broken) revision by that time.
The simplest way to avoid that is by sending just one email.

> Q2: Some think the full log in the mail body is more than necessary. Is it
> better or worse if it is a "tail -n 200" of the log in the body and the full log
> attached?

tail output and a link to the full logs would be nice.  Often there is a
lot of irrelevant output.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 17 July 2017 at 10:05, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, Jul 17, 2017 at 02:35:21PM +0800, Fam Zheng wrote:
>> Q1: In the worst case, you get four individual auto replies from patchew. Is
>> that too many? Do you prefer one reply with all the results concatenated into
>> one?
>
> I'd like to avoid situations where one of the failure emails is sent
> hours after the others, because it's possible that the patch series
> author has already sent the next (still broken) revision by that time.
> The simplest way to avoid that is by sending just one email.
>
>> Q2: Some think the full log in the mail body is more than necessary. Is it
>> better or worse if it is a "tail -n 200" of the log in the body and the full log
>> attached?
>
> tail output and a link to the full logs would be nice.  Often there is a
> lot of irrelevant output.

Ideally we'd streamline our make process to not produce so much
irrelevant output :-)

-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Mon, 07/17 10:28, Peter Maydell wrote:
> On 17 July 2017 at 10:05, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Mon, Jul 17, 2017 at 02:35:21PM +0800, Fam Zheng wrote:
> >> Q1: In the worst case, you get four individual auto replies from patchew. Is
> >> that too many? Do you prefer one reply with all the results concatenated into
> >> one?
> >
> > I'd like to avoid situations where one of the failure emails is sent
> > hours after the others, because it's possible that the patch series
> > author has already sent the next (still broken) revision by that time.
> > The simplest way to avoid that is by sending just one email.
> >
> >> Q2: Some think the full log in the mail body is more than necessary. Is it
> >> better or worse if it is a "tail -n 200" of the log in the body and the full log
> >> attached?
> >
> > tail output and a link to the full logs would be nice.  Often there is a
> > lot of irrelevant output.
> 
> Ideally we'd streamline our make process to not produce so much
> irrelevant output :-)

Does that mean to make "quite-command" absolutely quiet if V=1 is not specified?
;-)

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 17 July 2017 at 10:39, Fam Zheng <famz@redhat.com> wrote:
> On Mon, 07/17 10:28, Peter Maydell wrote:
>> Ideally we'd streamline our make process to not produce so much
>> irrelevant output :-)
>
> Does that mean to make "quite-command" absolutely quiet if V=1 is not specified?

The current 'quiet' mode is not quite aimed at the same purpose:
it's good for interactive use where you don't want the long detail
of command lines but you do want some periodic indication of
progress through the compile. For entirely noninteractive
setups like travis and patchew there's no need to produce
what is in effect a rather verbose progress bar...

thanks
-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 11:06:12AM +0100, Peter Maydell wrote:
> On 17 July 2017 at 10:39, Fam Zheng <famz@redhat.com> wrote:
> > On Mon, 07/17 10:28, Peter Maydell wrote:
> >> Ideally we'd streamline our make process to not produce so much
> >> irrelevant output :-)
> >
> > Does that mean to make "quite-command" absolutely quiet if V=1 is not specified?
> 
> The current 'quiet' mode is not quite aimed at the same purpose:
> it's good for interactive use where you don't want the long detail
> of command lines but you do want some periodic indication of
> progress through the compile. For entirely noninteractive
> setups like travis and patchew there's no need to produce
> what is in effect a rather verbose progress bar...

For unattended automated systems though, I think running with V=1 is
preferrable to quiet mode. When you don't have direct access to the
test system to reproduce systems, every bit of information that you
can get from the logs is potentially useful, including full compiler
args. The flipside is that in other cases the verbose output can
obscure the error messages you are after - depends what your're trying
to debug - code problems vs build system problems.

Perhaps it would be best to extract stderr separately. ie run with V=1
and have a full log that contains stdout+stderr as normal, but have a
second log which only contains stderr contents to make it easier to
see the error / warning messages in isolation from the "noise".


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 17 July 2017 at 11:26, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Mon, Jul 17, 2017 at 11:06:12AM +0100, Peter Maydell wrote:
>> The current 'quiet' mode is not quite aimed at the same purpose:
>> it's good for interactive use where you don't want the long detail
>> of command lines but you do want some periodic indication of
>> progress through the compile. For entirely noninteractive
>> setups like travis and patchew there's no need to produce
>> what is in effect a rather verbose progress bar...
>
> For unattended automated systems though, I think running with V=1 is
> preferrable to quiet mode. When you don't have direct access to the
> test system to reproduce systems, every bit of information that you
> can get from the logs is potentially useful, including full compiler
> args. The flipside is that in other cases the verbose output can
> obscure the error messages you are after - depends what your're trying
> to debug - code problems vs build system problems.

The problem in both cases is that it makes the logs nearly
unusable. For travis, their log-pretty-printer takes forever
to try to render our enormously long build logs, to the point
where you're better off grabbing the raw text-only log.
For patchew, I basically ignore patchew compile failure emails
because it is too painful to get to the bottom of the email
where the actual error message is because of the pages and
pages and pages of useless progress output :-(

thanks
-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Tue, 07/25 10:58, Peter Maydell wrote:
> For patchew, I basically ignore patchew compile failure emails
> because it is too painful to get to the bottom of the email
> where the actual error message is because of the pages and
> pages and pages of useless progress output :-(

Isn't it very easy to go to the bottom with a shortcut key and start looking
from there? That's why I thought including the full log in the email isn't a
problem, while not everyone likes web interface when it comes to working on a
long log (I personally find travis web page annoyingly inefficient).

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Gerd Hoffmann]

On Thu, 2017-07-27 at 19:03 +0800, Fam Zheng wrote:
> On Tue, 07/25 10:58, Peter Maydell wrote:
> > For patchew, I basically ignore patchew compile failure emails
> > because it is too painful to get to the bottom of the email
> > where the actual error message is because of the pages and
> > pages and pages of useless progress output :-(
> 
> Isn't it very easy to go to the bottom with a shortcut key and start
> looking
> from there? That's why I thought including the full log in the email
> isn't a
> problem, while not everyone likes web interface when it comes to
> working on a
> long log (I personally find travis web page annoyingly inefficient).

IMO it is fine to have a weblink only for the full build log if you
manage to include the relevant log snippet in the mail, so there is
rarely a need to actually lookup the full log online.

I think there is room for improvement on the latter.  You could try to
find the first "error" in the build log, go backwards 10 or 20 lines, 
from there include the next 50 to 100 lines inline in the mail. 
Probably needs some experimentation to figure which line counts work
best in practice.

cheers,
  Gerd

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 10:05:31AM +0100, Stefan Hajnoczi wrote:
> On Mon, Jul 17, 2017 at 02:35:21PM +0800, Fam Zheng wrote:
> > Q1: In the worst case, you get four individual auto replies from patchew. Is
> > that too many? Do you prefer one reply with all the results concatenated into
> > one?
> 
> I'd like to avoid situations where one of the failure emails is sent
> hours after the others, because it's possible that the patch series
> author has already sent the next (still broken) revision by that time.
> The simplest way to avoid that is by sending just one email.

If you do that, then you delay reporting of all 4 test systems, if only
one of them gets backlogged. I'd much rather see each test failure
reported immediately without waiting for other jobs which might complete
hours or days later in the worst case.

> > Q2: Some think the full log in the mail body is more than necessary. Is it
> > better or worse if it is a "tail -n 200" of the log in the body and the full log
> > attached?
> 
> tail output and a link to the full logs would be nice.  Often there is a
> lot of irrelevant output.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Thomas Huth]

On 17.07.2017 08:35, Fam Zheng wrote:
> Hi all,
> 
> Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> 
> So far we have these tests running by patchew on each patch series:
> 
>   * Docker tests
>     Basically it is
>         make docker-test-quick@centos6 \
>              docker-test-build@min-glib \
>              docker-test-mingw@fedora"
> 
>   * checkpatch.pl
>     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> 
>   * s390x
>     It runs on a machine shared by Fedora team, basically only "./configure and
>     make", because "make check" hanging is tricky to deal with from an
>     automation perspective. (Ideas?)

Is there any check that could hang "forever"? I think most of the checks
should have a proper timeout of one or two minutes, don't they?

Maybe you could also simply run the "make check" with the "timeout"
command to avoid that it hangs forever?

>   * FreeBSD
>     Like s390x.
> 
> Q1: In the worst case, you get four individual auto replies from patchew. Is
> that too many? Do you prefer one reply with all the results concatenated into
> one?

One result would be nicer, I think.

> Q2: Some think the full log in the mail body is more than necessary. Is it
> better or worse if it is a "tail -n 200" of the log in the body and the full log
> attached?
> 
> Q3: What other tests do maintainers want? Different hosts? Different configure
> combinations?

Not sure, but should we maybe also check compiling with the configure
switches set to non-default values? E.g. --enable-debug --disable-slirp
--enable-tcg-interpreter --disable-tcg etc. ?

 Thomas

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 11:41:38AM +0200, Thomas Huth wrote:
> On 17.07.2017 08:35, Fam Zheng wrote:
> > Hi all,
> > 
> > Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> > 
> > So far we have these tests running by patchew on each patch series:
> > 
> >   * Docker tests
> >     Basically it is
> >         make docker-test-quick@centos6 \
> >              docker-test-build@min-glib \
> >              docker-test-mingw@fedora"
> > 
> >   * checkpatch.pl
> >     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> > 
> >   * s390x
> >     It runs on a machine shared by Fedora team, basically only "./configure and
> >     make", because "make check" hanging is tricky to deal with from an
> >     automation perspective. (Ideas?)
> 
> Is there any check that could hang "forever"? I think most of the checks
> should have a proper timeout of one or two minutes, don't they?
> 
> Maybe you could also simply run the "make check" with the "timeout"
> command to avoid that it hangs forever?
> 
> >   * FreeBSD
> >     Like s390x.
> > 
> > Q1: In the worst case, you get four individual auto replies from patchew. Is
> > that too many? Do you prefer one reply with all the results concatenated into
> > one?
> 
> One result would be nicer, I think.
> 
> > Q2: Some think the full log in the mail body is more than necessary. Is it
> > better or worse if it is a "tail -n 200" of the log in the body and the full log
> > attached?
> > 
> > Q3: What other tests do maintainers want? Different hosts? Different configure
> > combinations?
> 
> Not sure, but should we maybe also check compiling with the configure
> switches set to non-default values? E.g. --enable-debug --disable-slirp
> --enable-tcg-interpreter --disable-tcg etc. ?

I'm a little concerned about the fact that we've now got three different
sets of tests that are being run on pull requests. There are the tests
that Peter runs on various combinations at time of merge, the tests run 
by patchw at time of submissions, and the tests run by travis after
merge. Each of them is covering a different set of scenarios with only
partial overlap between them.

As a maintainer sending pull requests, I want to try to run all relevant
tests before sending a PR, to minimize chance of it being rejected. It
is hard to come up with a workflow that maximises my coverage across all
three test systems that are run.

I mostly trigger running of the travis tests by pushing to a github banch 
in my qemu clone, but that misses coverage of things done by patchw and by
Peter, sometimes requiring many re-submissions of a PR before all three 
test processes pass. I find this is quite frustrating & a time sink for
everyone involved.

My ideal view of the world would be a single testing system which we
feed with jobs from different places. eg patchw can feed it mail submissions,
Peter can feed it candidate merges before pushing to master, something can
feed it git master after push, and/or periodically, and contributors can
feed it personal branches. IOW, no matter who/what triggers the tests, we
always run the exact same set of tests.  BUilding and maintaining such a
system is hard work, potentially expensive (in terms of hardware required),
and an ongoing full time job for at least one person, ideally more. So I
realize I'm asking for magical dancing ponies here, but it is nice to be 
able to dream.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 17 July 2017 at 11:20, Daniel P. Berrange <berrange@redhat.com> wrote:
> I'm a little concerned about the fact that we've now got three different
> sets of tests that are being run on pull requests. There are the tests
> that Peter runs on various combinations at time of merge, the tests run
> by patchw at time of submissions, and the tests run by travis after
> merge. Each of them is covering a different set of scenarios with only
> partial overlap between them.

I agree -- I have my ad-hoc test system not because I enjoy
collecting random things to run tests on but because we don't
have an automated system that does what I need:
 * wide coverage, focusing on oddball host architectures and OSes
 * ability to ssh in to a machine where tests have broken if need be
 * decently fast to complete a test run (travis is terrible for this)

Note that "fast to complete a test run for me" is in conflict with
"usable by anybody who wants to submit a pull request" :-)

thanks
-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 12:00:22PM +0100, Peter Maydell wrote:
> On 17 July 2017 at 11:20, Daniel P. Berrange <berrange@redhat.com> wrote:
> > I'm a little concerned about the fact that we've now got three different
> > sets of tests that are being run on pull requests. There are the tests
> > that Peter runs on various combinations at time of merge, the tests run
> > by patchw at time of submissions, and the tests run by travis after
> > merge. Each of them is covering a different set of scenarios with only
> > partial overlap between them.
> 
> I agree -- I have my ad-hoc test system not because I enjoy
> collecting random things to run tests on but because we don't
> have an automated system that does what I need:
>  * wide coverage, focusing on oddball host architectures and OSes
>  * ability to ssh in to a machine where tests have broken if need be
>  * decently fast to complete a test run (travis is terrible for this)
> 
> Note that "fast to complete a test run for me" is in conflict with
> "usable by anybody who wants to submit a pull request" :-)

Not neccessarily. It does mean though that if we had an automated test
system, we would have to reserve some portion of hardware resources for
exclusively jobs triggered by the person responsible for git master and
stable branch merges, so that it can never get delayed by a backlog of
jobs from the mailing list.

OpenStack has this kind of CI system - they have 2 separate job queues,
one testing all patches submitted for code review, and one queue only
processing patches for merge, so the former can't negatively impact the
progress of the latter. Availability of (money/sponsorship for) hardware
though is an obvious limiting factor here.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Philippe Mathieu-Daudé]

On Mon, Jul 17, 2017 at 7:20 AM, Daniel P. Berrange <berrange@redhat.com> wrote:
> I'm a little concerned about the fact that we've now got three different
> sets of tests that are being run on pull requests. There are the tests

hmm 4? there is also Shippable cross-builds:

  https://app.shippable.com/github/qemu/qemu/dashboard

[...]
> always run the exact same set of tests.  BUilding and maintaining such a
> system is hard work, potentially expensive (in terms of hardware required),
> and an ongoing full time job for at least one person, ideally more. So I

True :(

> realize I'm asking for magical dancing ponies here, but it is nice to be
> able to dream.

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Mon, 07/17 11:41, Thomas Huth wrote:
> On 17.07.2017 08:35, Fam Zheng wrote:
> > Hi all,
> > 
> > Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> > 
> > So far we have these tests running by patchew on each patch series:
> > 
> >   * Docker tests
> >     Basically it is
> >         make docker-test-quick@centos6 \
> >              docker-test-build@min-glib \
> >              docker-test-mingw@fedora"
> > 
> >   * checkpatch.pl
> >     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> > 
> >   * s390x
> >     It runs on a machine shared by Fedora team, basically only "./configure and
> >     make", because "make check" hanging is tricky to deal with from an
> >     automation perspective. (Ideas?)
> 
> Is there any check that could hang "forever"? I think most of the checks
> should have a proper timeout of one or two minutes, don't they?
> 
> Maybe you could also simply run the "make check" with the "timeout"
> command to avoid that it hangs forever?

Not every operation has a timeout in our tests, with the usual culprit being
vhost-user-test.

timeout can terminate make and directly invoked qtest commands, but qemu
processes that block or hang cannot be cleaned up.

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Thomas Huth]

On 18.07.2017 01:17, Fam Zheng wrote:
> On Mon, 07/17 11:41, Thomas Huth wrote:
>> On 17.07.2017 08:35, Fam Zheng wrote:
>>> Hi all,
>>>
>>> Today I've included a fourth type of the automatic patchew replies: FreeBSD.
>>>
>>> So far we have these tests running by patchew on each patch series:
>>>
>>>   * Docker tests
>>>     Basically it is
>>>         make docker-test-quick@centos6 \
>>>              docker-test-build@min-glib \
>>>              docker-test-mingw@fedora"
>>>
>>>   * checkpatch.pl
>>>     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
>>>
>>>   * s390x
>>>     It runs on a machine shared by Fedora team, basically only "./configure and
>>>     make", because "make check" hanging is tricky to deal with from an
>>>     automation perspective. (Ideas?)
>>
>> Is there any check that could hang "forever"? I think most of the checks
>> should have a proper timeout of one or two minutes, don't they?
>>
>> Maybe you could also simply run the "make check" with the "timeout"
>> command to avoid that it hangs forever?
> 
> Not every operation has a timeout in our tests, with the usual culprit being
> vhost-user-test.
> 
> timeout can terminate make and directly invoked qtest commands, but qemu
> processes that block or hang cannot be cleaned up.

Could the test be rewritten to provide a proper timeout handling
instead? Tests should clearly fail after a while instead of hanging
forever...
Or maybe we could add some magic that the troublesome tests are not
executed if a certain environment variable is set, so we could skip them
in these automated setups here?

 Thomas

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 18 July 2017 at 10:11, Thomas Huth <thuth@redhat.com> wrote:
> Could the test be rewritten to provide a proper timeout handling
> instead? Tests should clearly fail after a while instead of hanging
> forever...
> Or maybe we could add some magic that the troublesome tests are not
> executed if a certain environment variable is set, so we could skip them
> in these automated setups here?

I think it's always going to be possible that some bug in
QEMU results in a test hanging -- so it must be down to
the test harness to deal with hanging tests. And then
once you have that functionality in the test harness it
doesn't matter quite so much whether an individual test
has its own timeout handling or not.

thanks
-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Tue, Jul 18, 2017 at 10:37:30AM +0100, Peter Maydell wrote:
> On 18 July 2017 at 10:11, Thomas Huth <thuth@redhat.com> wrote:
> > Could the test be rewritten to provide a proper timeout handling
> > instead? Tests should clearly fail after a while instead of hanging
> > forever...
> > Or maybe we could add some magic that the troublesome tests are not
> > executed if a certain environment variable is set, so we could skip them
> > in these automated setups here?
> 
> I think it's always going to be possible that some bug in
> QEMU results in a test hanging -- so it must be down to
> the test harness to deal with hanging tests. And then
> once you have that functionality in the test harness it
> doesn't matter quite so much whether an individual test
> has its own timeout handling or not.

If the tests are all inside a docker container, then dealing with a hanging
test is no more difficult than waiting a suitable amount of time and then
telling docker to destroy the container.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Tue, 07/18 10:42, Daniel P. Berrange wrote:
> On Tue, Jul 18, 2017 at 10:37:30AM +0100, Peter Maydell wrote:
> > On 18 July 2017 at 10:11, Thomas Huth <thuth@redhat.com> wrote:
> > > Could the test be rewritten to provide a proper timeout handling
> > > instead? Tests should clearly fail after a while instead of hanging
> > > forever...
> > > Or maybe we could add some magic that the troublesome tests are not
> > > executed if a certain environment variable is set, so we could skip them
> > > in these automated setups here?
> > 
> > I think it's always going to be possible that some bug in
> > QEMU results in a test hanging -- so it must be down to
> > the test harness to deal with hanging tests. And then
> > once you have that functionality in the test harness it
> > doesn't matter quite so much whether an individual test
> > has its own timeout handling or not.
> 
> If the tests are all inside a docker container, then dealing with a hanging
> test is no more difficult than waiting a suitable amount of time and then
> telling docker to destroy the container.

Yes, the context here is non-x86 or non-Linux which are not as easy to run
docker.

Running tests in a VM would be great, but the harness is missing.

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 02:35:21PM +0800, Fam Zheng wrote:
> Hi all,
> 
> Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> 
> So far we have these tests running by patchew on each patch series:
> 
>   * Docker tests
>     Basically it is
>         make docker-test-quick@centos6 \
>              docker-test-build@min-glib \
>              docker-test-mingw@fedora"
> 
>   * checkpatch.pl
>     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> 
>   * s390x
>     It runs on a machine shared by Fedora team, basically only "./configure and
>     make", because "make check" hanging is tricky to deal with from an
>     automation perspective. (Ideas?)
> 
>   * FreeBSD
>     Like s390x.
> 
> Q1: In the worst case, you get four individual auto replies from patchew. Is
> that too many? Do you prefer one reply with all the results concatenated into
> one?

The risk with combining everything into one reply is that if only one of the
four test systems fails / hangs / gets very backlogged, you're going to
delay reporting of failures from all four systems. Personally, I would like
to see any failure reported as soon as it happens, without waiting to see
if other things fail or pass.

> Q2: Some think the full log in the mail body is more than necessary. Is it
> better or worse if it is a "tail -n 200" of the log in the body and the full log
> attached?

The mails are rather large alredy. It is common with Jenkins to only report
the tail of the logs, and then provide a hyperlink to a web site with the
complete log. This avoids bloating everyone's INBOXs with many 100 KB of
logs.

> Q3: What other tests do maintainers want? Different hosts? Different configure
> combinations?
> 
> Q4: Any other improvements/features you want? (E.g. some documentation? :)

Sometimes the test system seems to get pretty backlogged and there's no
way of knowing this, as its indistinguishable from the situation where
it doesn't send results because everything passed.

I'd like to see a web page that provides a list of all mail threads that
the test system has queued, with status of which jobs and running, and
once completed, provides the full logs.

That way we can quickly check whether patchw has started processing a
particular series or not.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Mon, 07/17 11:02, Daniel P. Berrange wrote:
> I'd like to see a web page that provides a list of all mail threads that
> the test system has queued, with status of which jobs and running, and
> once completed, provides the full logs.

The index page already exists (no running jobs information, though):

http://patchew.org/QEMU/

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Kevin Wolf]

Am 17.07.2017 um 08:35 hat Fam Zheng geschrieben:
> Hi all,
> 
> Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> 
> So far we have these tests running by patchew on each patch series:
> 
>   * Docker tests
>     Basically it is
>         make docker-test-quick@centos6 \
>              docker-test-build@min-glib \
>              docker-test-mingw@fedora"
> 
>   * checkpatch.pl
>     Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> 
>   * s390x
>     It runs on a machine shared by Fedora team, basically only "./configure and
>     make", because "make check" hanging is tricky to deal with from an
>     automation perspective. (Ideas?)
> 
>   * FreeBSD
>     Like s390x.
> 
> Q1: In the worst case, you get four individual auto replies from patchew. Is
> that too many? Do you prefer one reply with all the results concatenated into
> one?

checkpatch.pl is different enough from the other build/test errors that
I would prefer keeping a separate reply for that one.

But it seems that if your code doesn't compile (e.g. with different
configure options than on the developer's machine), chances are that all
three other tests fail, and then one reply for all of them is good
enough.

> Q2: Some think the full log in the mail body is more than necessary. Is it
> better or worse if it is a "tail -n 200" of the log in the body and the full log
> attached?

If you would attach the full log anyway, I'd say keep it in the body.
The other option is what others proposed in this thread, 'tail -n 200'
in the body and then include just a link to the full log in a web
interface.

> Q3: What other tests do maintainers want? Different hosts? Different configure
> combinations?

Would running qemu-iotests (at least the 'quick' group) be possible or
would that take too many resources?

Only today I noticed again that two recently merged pull requests broke
qemu-iotests cases, so I must assume that apart from some block
maintainers, nobody runs it regularly.

Kevin

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Peter Maydell]

On 17 July 2017 at 11:39, Kevin Wolf <kwolf@redhat.com> wrote:
> Only today I noticed again that two recently merged pull requests broke
> qemu-iotests cases, so I must assume that apart from some block
> maintainers, nobody runs it regularly.

If "make check" doesn't run it, it doesn't get run :-)
I actually looked into trying to run qemu-iotests tests
for the recent patches I sent for vpc and vmdk image format
handler fixes, but I couldn't get it to do anything sensible
and I couldn't find any documentation, so I gave up and
assumed that "make check" plus manual testing was good enough...

thanks
-- PMM

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Kevin Wolf]

Am 17.07.2017 um 12:49 hat Peter Maydell geschrieben:
> On 17 July 2017 at 11:39, Kevin Wolf <kwolf@redhat.com> wrote:
> > Only today I noticed again that two recently merged pull requests broke
> > qemu-iotests cases, so I must assume that apart from some block
> > maintainers, nobody runs it regularly.
> 
> If "make check" doesn't run it, it doesn't get run :-)
> I actually looked into trying to run qemu-iotests tests
> for the recent patches I sent for vpc and vmdk image format
> handler fixes, but I couldn't get it to do anything sensible
> and I couldn't find any documentation, so I gave up and
> assumed that "make check" plus manual testing was good enough...

I think we had it in 'make check' for a while, but I seem to remember
that people complained about things like the disk space that some of the
tests needed for temporary files, and some exotic systems didn't support
sparse files or something like that.

There is still 'make check-block', but nobody uses that. If you want to
add it back to 'make check', I wouldn't object (but others might).

Kevin

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On 2017-07-17 13:31, Kevin Wolf wrote:
> Am 17.07.2017 um 12:49 hat Peter Maydell geschrieben:
>> On 17 July 2017 at 11:39, Kevin Wolf <kwolf@redhat.com> wrote:
>>> Only today I noticed again that two recently merged pull requests broke
>>> qemu-iotests cases, so I must assume that apart from some block
>>> maintainers, nobody runs it regularly.
>>
>> If "make check" doesn't run it, it doesn't get run :-)
>> I actually looked into trying to run qemu-iotests tests
>> for the recent patches I sent for vpc and vmdk image format
>> handler fixes, but I couldn't get it to do anything sensible
>> and I couldn't find any documentation, so I gave up and
>> assumed that "make check" plus manual testing was good enough...
> 
> I think we had it in 'make check' for a while, but I seem to remember
> that people complained about things like the disk space that some of the
> tests needed for temporary files, and some exotic systems didn't support
> sparse files or something like that.

Wasn't that exotic system HFS+ on OS X/macOS? As far as I'm informed,
macOS has recently switched to APFS which does have sparse files (and I
think they have automatically converted running systems to APFS).

Max



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Philippe Mathieu-Daudé]

On Mon, Jul 17, 2017 at 10:40 AM, Max Reitz <mreitz@redhat.com> wrote:
> On 2017-07-17 13:31, Kevin Wolf wrote:
>> I think we had it in 'make check' for a while, but I seem to remember
>> that people complained about things like the disk space that some of the
>> tests needed for temporary files, and some exotic systems didn't support
>> sparse files or something like that.
>
> Wasn't that exotic system HFS+ on OS X/macOS? As far as I'm informed,
> macOS has recently switched to APFS which does have sparse files (and I
> think they have automatically converted running systems to APFS).

if anybody can run tests except those, we can add:

if $OS == OSX && $FS == HFS+; then
  "no 'make check' on OSX with HFS+, sorry"
fi

and have more ppl running tests...

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Mon, 07/17 12:39, Kevin Wolf wrote:
 
> > Q3: What other tests do maintainers want? Different hosts? Different configure
> > combinations?
> 
> Would running qemu-iotests (at least the 'quick' group) be possible or
> would that take too many resources?

As long as it can be done in several minutes on the test machine, we should be
good.

> 
> Only today I noticed again that two recently merged pull requests broke
> qemu-iotests cases, so I must assume that apart from some block
> maintainers, nobody runs it regularly.
> 

This has been on my list for a long time, the difficulty has been the frequent
breakage - I need to enable it when all tests pass, maybe 2.10 freeze is a good
time.

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Tue, Jul 18, 2017 at 07:28:50AM +0800, Fam Zheng wrote:
> On Mon, 07/17 12:39, Kevin Wolf wrote:
>  
> > > Q3: What other tests do maintainers want? Different hosts? Different configure
> > > combinations?
> > 
> > Would running qemu-iotests (at least the 'quick' group) be possible or
> > would that take too many resources?
> 
> As long as it can be done in several minutes on the test machine, we should be
> good.
> 
> > 
> > Only today I noticed again that two recently merged pull requests broke
> > qemu-iotests cases, so I must assume that apart from some block
> > maintainers, nobody runs it regularly.
> > 
> 
> This has been on my list for a long time, the difficulty has been the frequent
> breakage - I need to enable it when all tests pass, maybe 2.10 freeze is a good
> time.

IME the I/O tests runs for  raw & qcow2  formats are pretty stable in
general - the current breakage is the first I've noticed in a while.
So I'd certainly encourage enabling those two formats immediately, once
the current regresison is fix.

I also had patches merged recently that should fix it for 'qcow' and 'luks'
formats too, so it'd be good to get those enabled for actively testing
too.

The pain point I've seen come with a few tests that are sensitive to load
due to hidden race conditions. Tracking those down & fixing them is likely
only practical once we are actively running them on every patch, since it
needs a high volume of runs to expose the problems.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Philippe Mathieu-Daudé]

Hi Fam,

On 07/17/2017 03:35 AM, Fam Zheng wrote:
> So far we have these tests running by patchew on each patch series:
> 
>    * Docker tests
>      Basically it is
>          make docker-test-quick@centos6 \
>               docker-test-build@min-glib \
>               docker-test-mingw@fedora"
> 
>    * checkpatch.pl
>      Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> 
>    * s390x
>      It runs on a machine shared by Fedora team, basically only "./configure and
>      make", because "make check" hanging is tricky to deal with from an
>      automation perspective. (Ideas?)
> 
>    * FreeBSD
>      Like s390x.
> 
[...]
> Q4: Any other improvements/features you want? (E.g. some documentation? :)

I'm unsure if someone already asked about it, but is it possible to sort 
the tests, like:
- fastest/cheapest
- more likely to fail
- ...
- scarce resources at last

And abort/send report on first failure.

Regards,

Phil.

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Fam Zheng]

On Fri, 07/28 02:46, Philippe Mathieu-Daudé wrote:
> Hi Fam,
> 
> On 07/17/2017 03:35 AM, Fam Zheng wrote:
> > So far we have these tests running by patchew on each patch series:
> > 
> >    * Docker tests
> >      Basically it is
> >          make docker-test-quick@centos6 \
> >               docker-test-build@min-glib \
> >               docker-test-mingw@fedora"
> > 
> >    * checkpatch.pl
> >      Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> > 
> >    * s390x
> >      It runs on a machine shared by Fedora team, basically only "./configure and
> >      make", because "make check" hanging is tricky to deal with from an
> >      automation perspective. (Ideas?)
> > 
> >    * FreeBSD
> >      Like s390x.
> > 
> [...]
> > Q4: Any other improvements/features you want? (E.g. some documentation? :)
> 
> I'm unsure if someone already asked about it, but is it possible to sort the
> tests, like:
> - fastest/cheapest
> - more likely to fail
> - ...
> - scarce resources at last
> 
> And abort/send report on first failure.

Different hosts run tests in parallel, currently they are three: FreeBSD, Fedora
s390 and Fedora x86_64, and there isn't a strict order between them.

Also I'm not sure it's a good idea to _not_ run build tests if checkpatch test
fails.

One thing I agree is that checkpatch reports should be generated as soon as
possible, which is not the case at the moment. Thanks.

Fam

Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Cornelia Huck]

On Fri, 28 Jul 2017 14:33:37 +0800
Fam Zheng <famz@redhat.com> wrote:

> On Fri, 07/28 02:46, Philippe Mathieu-Daudé wrote:
> > Hi Fam,
> > 
> > On 07/17/2017 03:35 AM, Fam Zheng wrote:  
> > > So far we have these tests running by patchew on each patch series:
> > > 
> > >    * Docker tests
> > >      Basically it is
> > >          make docker-test-quick@centos6 \
> > >               docker-test-build@min-glib \
> > >               docker-test-mingw@fedora"
> > > 
> > >    * checkpatch.pl
> > >      Each patch is fed to ./scripts/checkpatch.pl and all errors are reported.
> > > 
> > >    * s390x
> > >      It runs on a machine shared by Fedora team, basically only "./configure and
> > >      make", because "make check" hanging is tricky to deal with from an
> > >      automation perspective. (Ideas?)
> > > 
> > >    * FreeBSD
> > >      Like s390x.
> > >   
> > [...]  
> > > Q4: Any other improvements/features you want? (E.g. some documentation? :)  
> > 
> > I'm unsure if someone already asked about it, but is it possible to sort the
> > tests, like:
> > - fastest/cheapest
> > - more likely to fail
> > - ...
> > - scarce resources at last
> > 
> > And abort/send report on first failure.  
> 
> Different hosts run tests in parallel, currently they are three: FreeBSD, Fedora
> s390 and Fedora x86_64, and there isn't a strict order between them.
> 
> Also I'm not sure it's a good idea to _not_ run build tests if checkpatch test
> fails.

Especially as sometimes checkpatch.pl reports things that can be
overridden by the maintainer with a good conscience.

Generally, I find it useful to get *all* problems instead of playing
whack-a-mole.

> 
> One thing I agree is that checkpatch reports should be generated as
> soon as possible, which is not the case at the moment. Thanks.

+1