libvirt / libxl testing emails

libvirt / libxl testing emails

[George Dunlap]

Ian / Ian,

Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and 
asked about testing the libvirt git repo against libxl.  I mentioned 
that we already had some basic tests to do that, and he asked whether it 
would make sense to have the push gate e-mail the libvirt list.

I have no idea how the libvirt people feel about that, so I'll let DV 
make his case, and also let he Ians give their opinions.

  -George

Re: [libvirt] libvirt / libxl testing emails

[Daniel P. Berrange]

On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> Ian / Ian,
> 
> Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> about testing the libvirt git repo against libxl.  I mentioned that we
> already had some basic tests to do that, and he asked whether it would make
> sense to have the push gate e-mail the libvirt list.
> 
> I have no idea how the libvirt people feel about that, so I'll let DV make
> his case, and also let he Ians give their opinions.

I guess from my POV the answer probably hinges on the reliability of the
test harness. If it has a very low false failure rate, then it'd be ok
to send it to this list. Conversely if there's a non-trivial false failure
rate it'd be better to have it go to an individual who filters it and just
forwards the real failures to the list.

An alternative idea would be for us to setup a new 'libvirt-testing'
mailing list, exclusively for spamming by CI systems, that people could
opt-in to subscribing to.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [libvirt] libvirt / libxl testing emails

[George Dunlap]

On 06/03/2014 03:30 PM, Daniel P. Berrange wrote:
> On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
>> Ian / Ian,
>>
>> Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
>> about testing the libvirt git repo against libxl.  I mentioned that we
>> already had some basic tests to do that, and he asked whether it would make
>> sense to have the push gate e-mail the libvirt list.
>>
>> I have no idea how the libvirt people feel about that, so I'll let DV make
>> his case, and also let he Ians give their opinions.
> I guess from my POV the answer probably hinges on the reliability of the
> test harness. If it has a very low false failure rate, then it'd be ok
> to send it to this list. Conversely if there's a non-trivial false failure
> rate it'd be better to have it go to an individual who filters it and just
> forwards the real failures to the list.
>
> An alternative idea would be for us to setup a new 'libvirt-testing'
> mailing list, exclusively for spamming by CI systems, that people could
> opt-in to subscribing to.

In my experience the test e-mails are fairly chatty, and fail for any 
number of reasons.  The most recent test mail can be found here, for 
instance:

http://www.gossamer-threads.com/lists/xen/devel/332583

And it seems to have failed because of a problem with the ARM testing 
hardware (since builds of Xen and Linux fail as well).

My suspicion is that most people on libvir-list will learn to ignore the 
mails unless someone raises a particular failure as something worth 
looking at; in which case it might be better to have a separate opt-in 
list.  (Or to wait until someone on xen-devel raises the issue.)

  -George

Re: [libvirt] libvirt / libxl testing emails

[Ian Campbell]

On Tue, 2014-06-03 at 15:37 +0100, George Dunlap wrote:
> On 06/03/2014 03:30 PM, Daniel P. Berrange wrote:
> > On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> >> Ian / Ian,
> >>
> >> Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> >> about testing the libvirt git repo against libxl.  I mentioned that we
> >> already had some basic tests to do that, and he asked whether it would make
> >> sense to have the push gate e-mail the libvirt list.
> >>
> >> I have no idea how the libvirt people feel about that, so I'll let DV make
> >> his case, and also let he Ians give their opinions.
> > I guess from my POV the answer probably hinges on the reliability of the
> > test harness. If it has a very low false failure rate, then it'd be ok
> > to send it to this list. Conversely if there's a non-trivial false failure
> > rate it'd be better to have it go to an individual who filters it and just
> > forwards the real failures to the list.
> >
> > An alternative idea would be for us to setup a new 'libvirt-testing'
> > mailing list, exclusively for spamming by CI systems, that people could
> > opt-in to subscribing to.
> 
> In my experience the test e-mails are fairly chatty, and fail for any 
> number of reasons.  The most recent test mail can be found here, for 
> instance:
> 
> http://www.gossamer-threads.com/lists/xen/devel/332583
> 
> And it seems to have failed because of a problem with the ARM testing 
> hardware (since builds of Xen and Linux fail as well).
> 
> My suspicion is that most people on libvir-list will learn to ignore the 
> mails unless someone raises a particular failure as something worth 
> looking at; in which case it might be better to have a separate opt-in 
> list.  (Or to wait until someone on xen-devel raises the issue.)

The majority of the failures so far have been infrastructure issues, so
I expect you are right.

I tend to keep an eye on this particular flight and I'm happy to forward
anything which looks like an upstream issue (it's only happened once so
far).

I'm equally happy to add some list or individual  other to the config
for places to spam with the results. Jim was happy to keep getting them
via xen-devel.

Ian.

Re: libvirt / libxl testing emails

[Daniel Veillard]

On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> Ian / Ian,
> 
> Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> about testing the libvirt git repo against libxl.  I mentioned that we
> already had some basic tests to do that, and he asked whether it would make
> sense to have the push gate e-mail the libvirt list.
> 
> I have no idea how the libvirt people feel about that, so I'll let DV make
> his case, and also let he Ians give their opinions.

  Actually Dan Berrange should be included. I think the nicest would
be a private ping and the information available online, I hink that
hw our existing rebuilder machines works but I would let Dan suggest the
best way at this point.

  thanks for getting back to me,

Daniel

-- 
Daniel Veillard      | Open Source and Standards, Red Hat
veillard@redhat.com  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | virtualization library  http://libvirt.org/

Re: libvirt / libxl testing emails

[Ian Campbell]

On Wed, 2014-06-04 at 00:00 +0800, Daniel Veillard wrote:
> On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> > Ian / Ian,
> > 
> > Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> > about testing the libvirt git repo against libxl.  I mentioned that we
> > already had some basic tests to do that, and he asked whether it would make
> > sense to have the push gate e-mail the libvirt list.
> > 
> > I have no idea how the libvirt people feel about that, so I'll let DV make
> > his case, and also let he Ians give their opinions.
> 
>   Actually Dan Berrange should be included. I think the nicest would
> be a private ping and the information available online,

Why private and not to the libvir list?

The information is available online via the test report (which don't
stay around forever, but for weeks at least).

I'm happy to do whatever you guys would prefer, just let me know.

Ian.

Re: libvirt / libxl testing emails

[Daniel Veillard]

On Fri, Jun 06, 2014 at 09:25:46AM +0100, Ian Campbell wrote:
> On Wed, 2014-06-04 at 00:00 +0800, Daniel Veillard wrote:
> > On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> > > Ian / Ian,
> > > 
> > > Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> > > about testing the libvirt git repo against libxl.  I mentioned that we
> > > already had some basic tests to do that, and he asked whether it would make
> > > sense to have the push gate e-mail the libvirt list.
> > > 
> > > I have no idea how the libvirt people feel about that, so I'll let DV make
> > > his case, and also let he Ians give their opinions.
> > 
> >   Actually Dan Berrange should be included. I think the nicest would
> > be a private ping and the information available online,
> 
> Why private and not to the libvir list?

  Well the point is that the libvir-list is already high traffic
and avoiding automated mails to it would help. Also the compile
box doesn't send mail directly to the list in case of breakage either,
so that would be coherent.

> The information is available online via the test report (which don't
> stay around forever, but for weeks at least).
> 
> I'm happy to do whatever you guys would prefer, just let me know.

  Do you have a pointer to where the test reports sit, that would be
helpful as a first step :-)

    thanks,

Daniel
-- 
Daniel Veillard      | Open Source and Standards, Red Hat
veillard@redhat.com  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | virtualization library  http://libvirt.org/

Re: libvirt / libxl testing emails

[Ian Campbell]

On Fri, 2014-06-06 at 17:32 +0800, Daniel Veillard wrote:
> On Fri, Jun 06, 2014 at 09:25:46AM +0100, Ian Campbell wrote:
> > On Wed, 2014-06-04 at 00:00 +0800, Daniel Veillard wrote:
> > > On Tue, Jun 03, 2014 at 03:16:49PM +0100, George Dunlap wrote:
> > > > Ian / Ian,
> > > > 
> > > > Daniel V. was lurking on a recent CentOS Virt SIG IRC meetings, and asked
> > > > about testing the libvirt git repo against libxl.  I mentioned that we
> > > > already had some basic tests to do that, and he asked whether it would make
> > > > sense to have the push gate e-mail the libvirt list.
> > > > 
> > > > I have no idea how the libvirt people feel about that, so I'll let DV make
> > > > his case, and also let he Ians give their opinions.
> > > 
> > >   Actually Dan Berrange should be included. I think the nicest would
> > > be a private ping and the information available online,
> > 
> > Why private and not to the libvir list?
> 
>   Well the point is that the libvir-list is already high traffic
> and avoiding automated mails to it would help.

Oh, I thought you were referring to my manual CC to libvirt that I have
been doing when an issue looked to be upstream related (exactly once so
far).

Yes, I certainly wouldn't recommend sending our automated test reports
to the list.

I'm happy to add an automated CC to whatever mailbox you guys want.

>  Also the compile
> box doesn't send mail directly to the list in case of breakage either,
> so that would be coherent.
> 
> > The information is available online via the test report (which don't
> > stay around forever, but for weeks at least).
> > 
> > I'm happy to do whatever you guys would prefer, just let me know.
> 
>   Do you have a pointer to where the test reports sit, that would be
> helpful as a first step :-)

There is a link in each test report e.g.
http://lists.xen.org/archives/html/xen-devel/2014-06/msg00776.html
links to
http://www.chiark.greenend.org.uk/~xensrcts/logs/26970/ note that the
logs are expired after a couple of weeks (this one is still live
though).

http://xenbits.xensource.com/gitweb/?p=osstest.git;a=blob;f=README;h=9a85549cb600fa5be91bcbae6605f78616db63da;hb=HEAD contains some terminology etc which might help you make sense of it.

Currently these are only build tests, because we've not taught our test
system about virsh yet, hence the "guest-start" step is marked as "never
passed".

By contrast
http://lists.xen.org/archives/html/xen-devel/2014-06/msg00557.html 
http://www.chiark.greenend.org.uk/~xensrcts/logs/26947/ is a failed test
(caused by a network glitch on our end)

http://lists.xen.org/archives/html/xen-devel/2014-05/msg00528.html has
some useful background one which test flights (flight == the FOO in the
"[FOO test]" but of the email subject) include libvirt tests and what
they are testing against what (summary: libvirt flight is new libvirt
against already tested xen and xen-* flights are new xen from that
branch against already an tested libvirt).

Ian.