Re: [xen-unstable test] 24250: tolerable FAIL

[Anthony PERARD <anthony.perard@citrix.com>]

On Wed, Jan 08, 2014 at 11:24:50AM +0000, Ian Campbell wrote:
> On Wed, 2014-01-08 at 11:12 +0000, Ian Campbell wrote:
> > On Wed, 2014-01-08 at 10:59 +0000, Jan Beulich wrote:
> > > >>> On 08.01.14 at 11:49, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > > > On Tue, 2014-01-07 at 12:29 +0000, Jan Beulich wrote:
> > > >> >>> On 06.01.14 at 10:36, xen.org <ian.jackson@eu.citrix.com> wrote:
> > > >> > flight 24250 xen-unstable real [real]
> > > >> > http://www.chiark.greenend.org.uk/~xensrcts/logs/24250/ 
> > > >> > 
> > > >> > Failures :-/ but no regressions.
> > > >> > 
> > > >> > Regressions which are regarded as allowable (not blocking):
> > > >> >  test-armhf-armhf-xl           7 debian-install               fail   like 
> > > > 24146
> > > >> >  test-amd64-i386-xl-winxpsp3-vcpus1  7 windows-install          fail like 
> > > > 23938
> > > >> >  test-amd64-i386-xl-win7-amd64  7 windows-install              fail  like 
> > > > 24146
> > > >> 
> > > >> These windows-install failures have been pretty persistent for
> > > >> the last month or two. I've been looking at the logs from the
> > > >> hypervisor side a number of times without spotting anything. It'd
> > > >> be nice to know whether anyone also did so from the tools and
> > > >> qemu sides... In any event we will need to do something about
> > > >> this before 4.4 goes out.
> > > > 
> > > > http://www.chiark.greenend.org.uk/~xensrcts/logs/24250/test-amd64-i386-xl-wi 
> > > > n7-amd64/win.guest.osstest--vnc.jpeg
> > > > 
> > > > says that Windows experienced an unexpected error.
> > > > 
> > > > http://www.chiark.greenend.org.uk/~xensrcts/logs/24250/test-amd64-i386-xl-wi 
> > > > nxpsp3-vcpus1/win.guest.osstest--vnc.jpeg
> > > > 
> > > > is a blue screen "BAD_POOL_CALLER".
> > 
> > FWIW It's code 0x7 which
> > http://msdn.microsoft.com/en-us/library/windows/hardware/ff560185%28v=vs.85%29.aspx
> > says is "The current thread attempted to free the pool, which was
> > already freed.".
> > 
> > The Internet(tm) seems to think this is often a driver issue. Not that
> > this helps us much!
> > 
> > > > I think this is unlikely to be a toolstack thing, but as to whether it
> > > > is a Xen or a Windows issue I wouldn't like to say. I had a look through
> > > > the toolstack logs anyway and didn't see anything untoward.
> > > 
> > > Right, neither did I. I was particularly thinking of qemu though,
> > > since I think these pretty persistent failures started around the
> > > time the qemu tree upgrade was done. Of course this could just
> > > be coincidence with a hypervisor side change having bad effects.
> > 
> > If there is a correlation then it would be interesting to investigate --
> > Anthony/Stefano could you guys have a look please.
> 
> The earliest instance I could see with logs was 22371 (10/12/2013).
> 
> 21288 (30/10/2013) has a windows-install failure but the logs have
> expired so I cannot tell if it was the same failure.
> 
> I didn't look at the vnc for every failure I spotted, there were a lot
> like these, and a lot where Windows was just sat at its login screen
> (quite long standing issue I think? Thought not to be us?). 
> 
> There were a smattering of other failures too e.g.:
> http://www.chiark.greenend.org.uk/~xensrcts/logs/22466/test-amd64-i386-xl-win7-amd64/win.guest.osstest--vnc.jpeg
> http://www.chiark.greenend.org.uk/~xensrcts/logs/22455/test-amd64-i386-xl-win7-amd64/win.guest.osstest--vnc.jpeg

There is also this one:
http://www.chiark.greenend.org.uk/~xensrcts/logs/23724/test-amd64-amd64-xl-qemuu-winxpsp3/win.guest.osstest--vnc.jpeg
an issue with ntfs.sys.

Or this one:
http://www.chiark.greenend.org.uk/~xensrcts/logs/23035/test-amd64-i386-xl-winxpsp3-vcpus1/win.guest.osstest--vnc.jpeg
which say "a device driver has pool"
and google respond:
http://www.faultwire.com/solutions-fatal_error/A-device-driver-has-pool-0x000000C5-*1198.html
"A device driver has a bug that attempted to access memory either
nonexistent memory or memory it is not allowed to access."

So, maybe a emulated disk issue or memory issue ?

I'll try to reproduce the issue.

> (bearing in mind that I didn't check all of them, if there was an easy
> way to data mine the screen shots for all the failures of this test into
> a directory it might be easier to scan)

-- 
Anthony PERARD