Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Lars Kurth]

Hi everybody,

I wanted to follow up on George's e-mail (Xen 4.3 development update 
CODE FREEZE at 
http://lists.xen.org/archives/html/xen-devel/2013-04/msg02142.html). One 
way we can facilitate the goal of

  1. Have a bug-free 4.3 release

... is to build in some Test Days into the coming weeks. This worked 
well for Xen 4.2 RC2 as well as XCP 1.6. What we do need though is a 
good set of test instructions for Xen 4.3 : examples for last time round 
were
- http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions
- http://wiki.xen.org/wiki/XCP_1.6_test_instructions
We have a document day coming up next Monday, which would be an ideal 
day to put together Xen 4.3 RC test intrstructions. I remember we got a 
lot of test coverage for 4.2 RC2 as well as XCP 1.6.

Having a predictable schedule around RC's would also help. It appears to 
me that maybe one test day every other week starting from the first RC 
(planned for May 6th would work) ... we could do the test day 1-2 days 
after the planned RC's to provide for a buffer should an RC slip.

That would give us dates for Xen Test Days for May 8, May 22 (not 
clashing with Hackathon), June 5 (with RC's planned 2 days before).

Views? Should I start announcing the dates and promote the Test Days? If 
so, I need some commitment on getting good Test instructions as well as 
a good sense that RC's are possible for these dates. If so, I would like 
to announce these by the end of this week.

Regards
Lars

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[jacek burghardt]

[-- Attachment #1.1: Type: text/plain, Size: 2361 bytes --]

This sounds like great idea. What xen needs more clarification
on bugs reporting, I had experienced many bugs with xen unstable lately and
I don't know if anyone cares to fix the issues and if anyone is even
running testing unstable and if they experience same issues. HVM recently
stop providing network (at least with e1000 emulation   )unless xen drivers
are installed and  freenas does not boot up on xen and does not have
network connection


On Tue, Apr 23, 2013 at 8:29 AM, Lars Kurth <lars.kurth@xen.org> wrote:

> Hi everybody,
>
> I wanted to follow up on George's e-mail (Xen 4.3 development update CODE
> FREEZE at http://lists.xen.org/archives/**html/xen-devel/2013-04/**
> msg02142.html<http://lists.xen.org/archives/html/xen-devel/2013-04/msg02142.html>).
> One way we can facilitate the goal of
>
>  1. Have a bug-free 4.3 release
>
> ... is to build in some Test Days into the coming weeks. This worked well
> for Xen 4.2 RC2 as well as XCP 1.6. What we do need though is a good set of
> test instructions for Xen 4.3 : examples for last time round were
> - http://wiki.xen.org/wiki/Xen_**4.2_RC2_test_instructions<http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions>
> - http://wiki.xen.org/wiki/XCP_**1.6_test_instructions<http://wiki.xen.org/wiki/XCP_1.6_test_instructions>
> We have a document day coming up next Monday, which would be an ideal day
> to put together Xen 4.3 RC test intrstructions. I remember we got a lot of
> test coverage for 4.2 RC2 as well as XCP 1.6.
>
> Having a predictable schedule around RC's would also help. It appears to
> me that maybe one test day every other week starting from the first RC
> (planned for May 6th would work) ... we could do the test day 1-2 days
> after the planned RC's to provide for a buffer should an RC slip.
>
> That would give us dates for Xen Test Days for May 8, May 22 (not clashing
> with Hackathon), June 5 (with RC's planned 2 days before).
>
> Views? Should I start announcing the dates and promote the Test Days? If
> so, I need some commitment on getting good Test instructions as well as a
> good sense that RC's are possible for these dates. If so, I would like to
> announce these by the end of this week.
>
> Regards
> Lars
>
>
> ______________________________**_________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 2978 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Roger Pau Monné]

On 23/04/13 16:53, jacek burghardt wrote:
> This sounds like great idea. What xen needs more clarification
> on bugs reporting, I had experienced many bugs with xen unstable lately
> and I don't know if anyone cares to fix the issues and if anyone is even
> running testing unstable and if they experience same issues. HVM
> recently stop providing network (at least with e1000 emulation   )unless
> xen drivers are installed and  freenas does not boot up on xen and does
> not have network connection

The problem with FreeNAS not booting is already reported and Jan has a
patch for it (since FreeNAS is basically FreeBSD I guess the same patch
that Jan provided for FreeBSD will apply to FreeNAS):

http://marc.info/?l=xen-devel&m=136637574324172

Could you provide more info about the network problem? Which OS are you
using inside the guest? Is the emulated network interface correctly
attached to a bridge? Could you bisect the Xen tree in order to find the
commit that breaks emulated networking?

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Ian Campbell]

On Tue, 2013-04-23 at 15:53 +0100, jacek burghardt wrote:
> This sounds like great idea. What xen needs more clarification
> on bugs reporting, I had experienced many bugs with xen
> unstable lately and I don't know if anyone cares to fix the issues and
> if anyone is even running testing unstable and if they experience same
> issues. HVM recently stop providing network (at least with e1000
> emulation   )unless xen drivers are installed and  freenas does not
> boot up on xen and does not have network connection

Please see http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen for how
to report a bug against Xen. If you follow the suggestions there and
include all the useful details and logs instead of posting brief mails
with little detail you will likely find you get better traction with
your bug reports.

Ian.

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Wei Liu]

On Tue, Apr 23, 2013 at 03:53:01PM +0100, jacek burghardt wrote:
> This sounds like great idea. What xen needs more clarification on bugs reporting, I had experienced many bugs with xen unstable lately and I don't know if anyone cares to fix the issues and if anyone is even running testing unstable and if they experience same issues. HVM recently stop providing network (at least with e1000 emulation   )unless xen drivers are installed and  freenas does not boot up on xen and does not have network connection
> 
> 

Thanks for your help with testing Xen unstable. It is not that we don't
care about fixing problems, but sometimes we would like to see more
concrete logs / figures so that we can identify the genuine causes.

http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen could help we
interact more efficient.


Wei.

> On Tue, Apr 23, 2013 at 8:29 AM, Lars Kurth <lars.kurth@xen.org<mailto:lars.kurth@xen.org>> wrote:
> Hi everybody,
> 
> I wanted to follow up on George's e-mail (Xen 4.3 development update CODE FREEZE at http://lists.xen.org/archives/html/xen-devel/2013-04/msg02142.html). One way we can facilitate the goal of
> 
>  1. Have a bug-free 4.3 release
> 
> ... is to build in some Test Days into the coming weeks. This worked well for Xen 4.2 RC2 as well as XCP 1.6. What we do need though is a good set of test instructions for Xen 4.3 : examples for last time round were
> - http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions
> - http://wiki.xen.org/wiki/XCP_1.6_test_instructions
> We have a document day coming up next Monday, which would be an ideal day to put together Xen 4.3 RC test intrstructions. I remember we got a lot of test coverage for 4.2 RC2 as well as XCP 1.6.
> 
> Having a predictable schedule around RC's would also help. It appears to me that maybe one test day every other week starting from the first RC (planned for May 6th would work) ... we could do the test day 1-2 days after the planned RC's to provide for a buffer should an RC slip.
> 
> That would give us dates for Xen Test Days for May 8, May 22 (not clashing with Hackathon), June 5 (with RC's planned 2 days before).
> 
> Views? Should I start announcing the dates and promote the Test Days? If so, I need some commitment on getting good Test instructions as well as a good sense that RC's are possible for these dates. If so, I would like to announce these by the end of this week.
> 
> Regards
> Lars
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org<mailto:Xen-devel@lists.xen.org>
> http://lists.xen.org/xen-devel
> 

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Lars Kurth]

Thanks for the pointers on bug reports. I still would like to get input 
on planning the Test Days though and whether we should build these into 
the release cycle.
Lars

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Stefano Stabellini]

On Tue, 23 Apr 2013, Lars Kurth wrote:
> Hi everybody,
> 
> I wanted to follow up on George's e-mail (Xen 4.3 development update 
> CODE FREEZE at 
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg02142.html). One 
> way we can facilitate the goal of
> 
>   1. Have a bug-free 4.3 release
> 
> ... is to build in some Test Days into the coming weeks. This worked 
> well for Xen 4.2 RC2 as well as XCP 1.6. What we do need though is a 
> good set of test instructions for Xen 4.3 : examples for last time round 
> were
> - http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions
> - http://wiki.xen.org/wiki/XCP_1.6_test_instructions
> We have a document day coming up next Monday, which would be an ideal 
> day to put together Xen 4.3 RC test intrstructions. I remember we got a 
> lot of test coverage for 4.2 RC2 as well as XCP 1.6.
> 
> Having a predictable schedule around RC's would also help. It appears to 
> me that maybe one test day every other week starting from the first RC 
> (planned for May 6th would work) ... we could do the test day 1-2 days 
> after the planned RC's to provide for a buffer should an RC slip.

That sounds like a good plan to me


> That would give us dates for Xen Test Days for May 8, May 22 (not 
> clashing with Hackathon), June 5 (with RC's planned 2 days before).

Fine by me


> Views? Should I start announcing the dates and promote the Test Days? If 
> so, I need some commitment on getting good Test instructions as well as 
> a good sense that RC's are possible for these dates. If so, I would like 
> to announce these by the end of this week.

Go ahead

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2357 bytes --]

On mar, 2013-04-23 at 15:29 +0100, Lars Kurth wrote:
> Hi everybody,
> 
> I wanted to follow up on George's e-mail (Xen 4.3 development update 
> CODE FREEZE at 
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg02142.html). One 
> way we can facilitate the goal of
> 
>   1. Have a bug-free 4.3 release
> 
> ... is to build in some Test Days into the coming weeks. This worked 
> well for Xen 4.2 RC2 as well as XCP 1.6. 
>
+1 ... I really really like the idea of test days!

> What we do need though is a 
> good set of test instructions for Xen 4.3 : examples for last time round 
> were
> - http://wiki.xen.org/wiki/Xen_4.2_RC2_test_instructions
> - http://wiki.xen.org/wiki/XCP_1.6_test_instructions
> We have a document day coming up next Monday, which would be an ideal 
> day to put together Xen 4.3 RC test intrstructions. I remember we got a 
> lot of test coverage for 4.2 RC2 as well as XCP 1.6.
> 
> Having a predictable schedule around RC's would also help. It appears to 
> me that maybe one test day every other week starting from the first RC 
> (planned for May 6th would work) ... we could do the test day 1-2 days 
> after the planned RC's to provide for a buffer should an RC slip.
> 
> That would give us dates for Xen Test Days for May 8, May 22 (not 
> clashing with Hackathon), June 5 (with RC's planned 2 days before).
> 
Sounds cool.

> Views? Should I start announcing the dates and promote the Test Days?
>
Yes, I think we should. For what I've seen from the 'Fedora Test Days',
the success of the whole thing depends on having people around and
actually following the instructions and running the tests, even if
everyone just do that for a small subset of them!

>  If 
> so, I need some commitment on getting good Test instructions as well as 
> a good sense that RC's are possible for these dates. If so, I would like 
> to announce these by the end of this week.
> 
I'm going to be on leave next Monday (during DocsDay), but for
everything else you need from May 2nd on, count me. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Ian Campbell]

On Tue, 2013-04-23 at 15:29 +0100, Lars Kurth wrote:
> 
> Views? Should I start announcing the dates and promote the Test Days? 

All sounds good, I think you should go ahead.

Ian.

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Lars Kurth]

[-- Attachment #1.1: Type: text/plain, Size: 137 bytes --]

>> Views? Should I start announcing the dates and promote the Test Days?
> All sounds good, I think you should go ahead.
Any objections?

[-- Attachment #1.2: Type: text/html, Size: 526 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[Lars Kurth]

[-- Attachment #1.1: Type: text/plain, Size: 428 bytes --]

On 25/04/2013 09:25, Lars Kurth wrote:
> >> Views? Should I start announcing the dates and promote the Test Days?
> > All sounds good, I think you should go ahead.
> Any objections?
>
Appears there are none. I created
- http://wiki.xen.org/wiki/Xen_4.3_RC1_test_instructions which needs to 
be popolated
- Remember Mondy is Docs Day - A perfect opportunity to do this
- Also updated http://wiki.xen.org/wiki/Xen_Test_Days

Lars

[-- Attachment #1.2: Type: text/html, Size: 1394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Building Xen Test Days into the conclusion of the Xen 4.3 release cycle

[George Dunlap]

On Thu, Apr 25, 2013 at 12:18 PM, Lars Kurth <lars.kurth@xen.org> wrote:
> On 25/04/2013 09:25, Lars Kurth wrote:
>
>>> Views? Should I start announcing the dates and promote the Test Days?
>> All sounds good, I think you should go ahead.
> Any objections?
>
> Appears there are none. I created
> - http://wiki.xen.org/wiki/Xen_4.3_RC1_test_instructions which needs to be
> popolated
> - Remember Mondy is Docs Day - A perfect opportunity to do this
> - Also updated http://wiki.xen.org/wiki/Xen_Test_Days

I was thinking it would be good to think about a "test matrix".  Below
is a sort of summary of what I came up with -- I'd appreciate if
people could add things I'm missing.

In general I was thinking "variables" as a sort of "state of the
system" V, and then we want to test specific functionality f given
state V.

 -George

Variables
 - Host Hardware
   - CPU (Intel, AMD)
   - NUMA topology
   - Amount of memory
   - IOMMU, &c
 - Host config
   - Domain 0 distro
     { Debian, Fedora, Ubuntu, Arch, Alpine, NetBSD }
   - Domain 0 kernel
     { distro, mainline most recent, mainline stable }
   - Network driver domain
   - Networking (bridging / OVS / &c)
   - With / without default XSM policy
 - Guest config
   - Guest OS
   - PV / PVHVM (for Linux / NetBSD guests)
   - qemu version ( qemu-xen-traditional / qemu-xen / straight qemu )
   - Stubdoms
   - Emualted / PV devices (disk, network)
   - USB pass-through

Functionality
 - Build / install
 - Basic
   create, list, shutdown, reboot, destroy, suspend/resume, migrate,
pause, unpause, console
 - Extended basic
   cd-insert, cd-eject, button-press, vcpu-pin, vcpu-set, domid,
domname, rename, trigger, sysrq,
   info, dmesg, top, config-update, info
 - Guest devices
   - PCI pass-through
   - Network / block attach
   - Host usb (config file only for 4.3)
   - qxl spice
 - Memory
   - Ballooning
     PoD, mem-max, mem-set
   - Page sharing
   - Hypervisor swap
 - Scheduling parameters
 - NUMA
 - debug-keys
 - vtpm
 - tmem
 - cpupools
 - remus
 - alternate block / network scripts
 - Flask

Types of testing
 - Smoke-test
   Just do something quickly to see if it works
 - Normal functional test
   Try to use it the way you expect it to work
 - Stress-testing
   Automated heavy use over long period of time
 - Edge-case testing
   Try to use it in ways you don't expect it to be used
 - Performance
   Test it specifically measuring the performance