osstest "short fast" tests of xen-unstable proposal

osstest "short fast" tests of xen-unstable proposal

[Ian Jackson]

osstest's test coverage is improving, which means that the tests keep
taking longer.  Even with ongoing expansion of the test facility, the
overall time between a bad push and getting results can be quite long
- currently perhaps 12-36h depending on circumstances.

It would be better if we could get quicker notification of build
breakages and other kinds of very basic breakage.

I propose:


Invent a new `xen-unstable-smoke' flight (in osstest terminology, a
`branch').

This would be a push gate for xen.git.  Its input would be
xen.git#staging.  Its output would be a new branch, xen.git#smoked.
The existing `xen-unstable' flights would take xen.git#smoked as
input.  So there would be a two-stage push gate,

 staging -[xen-unstable-smoke test]-> smoked -[xen-unstable test]-> master

This would apply to xen-unstable only, not to stable braches, nor to
any other codebase (eg, qemu or Linux).  It would aim to run every 2h.

Arrangements would be made to reuse the outputs of most recent builds
of qemuu and the currently favoured Linux branch.[5]

If any test failed, the flight would be automatically aborted and
report immediately [6].  Unlike most flights, tests would not be
`sticky' to failing hosts, nor (when succeding) prefer hosts which
they hadn't run on for a while [6], so they would take the first
available machine.


The new flights would contain only:
                                                 x86    ARM  [1]

  build-amd64                                    800s
  build-armhf                                           800s

  test-amd64-amd64-xl                           2300s
  test-armhf-armhf-xl                                  2500s

  test-amd64-amd64-xl-qemuu-debianhvm-i386 [2]  3200s

                                                -----  -----
                                                6300s  3300s


Total machine resources for each run would be about 2h of x86 capacity
and 1h of ARM capacity, with the infrastructure as currently
configured.  We currently have 19 x86 machines live out of a planned
24, and 5 ARM machines out of a planned 8 [3].

The ARM machines are much less heavily used because they have a much
smaller variety of tests (no HVM, no Windows, no RHEL, etc).

So thinking about x86, doing a test like this about every 2h would
take ~5% of our current capacity, which would be well worthwhile.
Doing it much more often would be tricky because of resource
allocation delays: we have to wait for another test to finish, and we
may need to reinstall a build box.[4]


Comments welcome.

Ian.


[1] These are execution times for this test in 58974, which was the
last xen-unstable pass.  They do not include host allocation - ie,
they do not include the time waiting for a host.

[2] This job does not currently exist, but it would be by analogy
with test-amd64-amd64-xl-qemuu-debianhvm-amd64.

[3] There have been some hardware problems with our ARM crate.

[4] Currently we just use general test boxes as build servers.  In the
next hardware order I will try to procure _four_ of some kind of box,
and dedicate two of them to builds.  This will improve matters because
they will be reinstalled less often (so avoiding delays for
reinstallation and also benefiting more from ccache).

[5] This will need improvements to osstest's garbage collector, or
those builds might be deleted while still being used.

[6] This feature does not yet exist in osstest but could be added.

Re: osstest "short fast" tests of xen-unstable proposal

[George Dunlap]

On Tue, Jun 30, 2015 at 6:46 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> osstest's test coverage is improving, which means that the tests keep
> taking longer.  Even with ongoing expansion of the test facility, the
> overall time between a bad push and getting results can be quite long
> - currently perhaps 12-36h depending on circumstances.
>
> It would be better if we could get quicker notification of build
> breakages and other kinds of very basic breakage.
>
> I propose:
>
>
> Invent a new `xen-unstable-smoke' flight (in osstest terminology, a
> `branch').
>
> This would be a push gate for xen.git.  Its input would be
> xen.git#staging.  Its output would be a new branch, xen.git#smoked.
> The existing `xen-unstable' flights would take xen.git#smoked as
> input.  So there would be a two-stage push gate,
>
>  staging -[xen-unstable-smoke test]-> smoked -[xen-unstable test]-> master
>
> This would apply to xen-unstable only, not to stable braches, nor to
> any other codebase (eg, qemu or Linux).  It would aim to run every 2h.
>
> Arrangements would be made to reuse the outputs of most recent builds
> of qemuu and the currently favoured Linux branch.[5]
>
> If any test failed, the flight would be automatically aborted and
> report immediately [6].  Unlike most flights, tests would not be
> `sticky' to failing hosts, nor (when succeding) prefer hosts which
> they hadn't run on for a while [6], so they would take the first
> available machine.

This sounds good to me.

 -George

Re: osstest "short fast" tests of xen-unstable proposal

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1539 bytes --]

On Thu, 2015-07-02 at 18:17 +0100, George Dunlap wrote:
> On Tue, Jun 30, 2015 at 6:46 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:

> > Invent a new `xen-unstable-smoke' flight (in osstest terminology, a
> > `branch').
> >
> > This would be a push gate for xen.git.  Its input would be
> > xen.git#staging.  Its output would be a new branch, xen.git#smoked.
> > The existing `xen-unstable' flights would take xen.git#smoked as
> > input.  So there would be a two-stage push gate,
> >
> >  staging -[xen-unstable-smoke test]-> smoked -[xen-unstable test]-> master
> >
> > This would apply to xen-unstable only, not to stable braches, nor to
> > any other codebase (eg, qemu or Linux).  It would aim to run every 2h.
> >
> > Arrangements would be made to reuse the outputs of most recent builds
> > of qemuu and the currently favoured Linux branch.[5]
> >
> > If any test failed, the flight would be automatically aborted and
> > report immediately [6].  Unlike most flights, tests would not be
> > `sticky' to failing hosts, nor (when succeding) prefer hosts which
> > they hadn't run on for a while [6], so they would take the first
> > available machine.
> 
> This sounds good to me.
> 
+1

This would all be really, really useful!

Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: osstest "short fast" tests of xen-unstable proposal

[Jan Beulich]

>>> On 30.06.15 at 19:46, <Ian.Jackson@eu.citrix.com> wrote:
> osstest's test coverage is improving, which means that the tests keep
> taking longer.  Even with ongoing expansion of the test facility, the
> overall time between a bad push and getting results can be quite long
> - currently perhaps 12-36h depending on circumstances.
> 
> It would be better if we could get quicker notification of build
> breakages and other kinds of very basic breakage.
> 
> I propose:
> 
> 
> Invent a new `xen-unstable-smoke' flight (in osstest terminology, a
> `branch').
> 
> This would be a push gate for xen.git.  Its input would be
> xen.git#staging.  Its output would be a new branch, xen.git#smoked.
> The existing `xen-unstable' flights would take xen.git#smoked as
> input.  So there would be a two-stage push gate,
> 
>  staging -[xen-unstable-smoke test]-> smoked -[xen-unstable test]-> master
> 
> This would apply to xen-unstable only, not to stable braches, nor to
> any other codebase (eg, qemu or Linux).  It would aim to run every 2h.
> 
> Arrangements would be made to reuse the outputs of most recent builds
> of qemuu and the currently favoured Linux branch.[5]
> 
> If any test failed, the flight would be automatically aborted and
> report immediately [6].  Unlike most flights, tests would not be
> `sticky' to failing hosts, nor (when succeding) prefer hosts which
> they hadn't run on for a while [6], so they would take the first
> available machine.
> 
> 
> The new flights would contain only:
>                                                  x86    ARM  [1]
> 
>   build-amd64                                    800s
>   build-armhf                                           800s
> 
>   test-amd64-amd64-xl                           2300s
>   test-armhf-armhf-xl                                  2500s
> 
>   test-amd64-amd64-xl-qemuu-debianhvm-i386 [2]  3200s
> 
>                                                 -----  -----
>                                                 6300s  3300s
> 
> 
> Total machine resources for each run would be about 2h of x86 capacity
> and 1h of ARM capacity, with the infrastructure as currently
> configured.  We currently have 19 x86 machines live out of a planned
> 24, and 5 ARM machines out of a planned 8 [3].
> 
> The ARM machines are much less heavily used because they have a much
> smaller variety of tests (no HVM, no Windows, no RHEL, etc).
> 
> So thinking about x86, doing a test like this about every 2h would
> take ~5% of our current capacity, which would be well worthwhile.
> Doing it much more often would be tricky because of resource
> allocation delays: we have to wait for another test to finish, and we
> may need to reinstall a build box.[4]
> 
> 
> Comments welcome.

Yes, please.

Thanks, Jan