Re: Fresh install of Windows XP hangs early in boot?

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

 > I experienced random hangs during the install (stracing kvm shows no
 > system calls, and it appears to be spinning at 100% CPU), but eventually
 > I got an install that ran all the way to completion.  However, that
 > image seems to hang every time shortly after boot starts.  I see the
 > Windows splash screen, the little blue dots move for a few seconds, and
 > then the guest hangs in the same way -- no system calls, 100% CPU.

FWIW, when I ltrace the stuck kvm process, I get an endless string of

    memcpy(0x7fffb09355f0, "\224\213'\206", 4)

(the value changes from run to run, but stays constant within a run)

Unfortunately Debian packages don't seem to be built with debugging
symbols, so gdb doesn't show a very enlightening backtrace.  I'll try to
get more info.

 - R.

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

I built with debugging symbols, and this seems to be an issue with SCSI
disk emulation.  The traceback is:

#0  0x00007fc086d7dd10 in memcpy () from /lib/libc.so.6
#1  0x00000000004a319b in cpu_physical_memory_rw (addr=108661608,
    buf=0x7fff904ca190 "\224['\206\210\030z\006I�A", len=4, is_write=0)
    at /users/rdreier/kvm-deb.git/qemu/exec.c:2847
#2  0x000000000041f0c2 in lsi_execute_script (s=0x2ef7a30) at ../cpu-all.h:924
#3  0x000000000049bd91 in qcow_aio_read_cb (opaque=0x3018d70, ret=0) at block-qcow2.c:840
#4  0x000000000041cba0 in qemu_aio_poll () at /users/rdreier/kvm-deb.git/qemu/block-raw-posix.c:513
#5  0x000000000040b38a in main_loop_wait (timeout=<value optimized out>)
    at /users/rdreier/kvm-deb.git/qemu/vl.c:7777
#6  0x00000000004f607a in kvm_main_loop () at /users/rdreier/kvm-deb.git/qemu/qemu-kvm.c:587
#7  0x0000000000412b46 in main (argc=<value optimized out>, argv=0x7fff904cb0c8)
    at /users/rdreier/kvm-deb.git/qemu/vl.c:7811

and no progress ever seems to be made (the same address is read over and
over)

I'm trying again with IDE instead of SCSI disks.  But I would like to
help debug the SCSI emulation... will look at it further later, and I'm
happy to provide any info someone else could use.

 - R.

Re: Fresh install of Windows XP hangs early in boot?

[Alberto Treviño]

On Tuesday 29 July 2008 01:39:53 pm you wrote:
> I built with debugging symbols, and this seems to be an issue with SCSI
> disk emulation.

The problem might be related to this bug report:

http://sourceforge.net/tracker/index.php?func=detail&aid=1895893&group_id=180599&atid=893831

Marcelo Tosatti worked hard on getting the bug fixed, but the patch was 
rejected by the Qemu team because it was not a "proper" fix.  Marcelo has 
been trying to figure out what a proper fix should look like and rework the 
patch.  I don't know what Marcelo's progress has been on the issue.  The 
last working patches Marcelo and others put out which fixes the lock-ups can 
be found here:

http://slim.mypals.org/slim/files/kvm/kvm-64-scsi.patch
http://slim.mypals.org/slim/files/kvm/lsi-busy-loop-3a.patch

You may try installing those patches on KVM (they can still be applied to 
KVM-72) and see if they fix your problems.

Hope this helps.

Re: Fresh install of Windows XP hangs early in boot?

[Marcelo Tosatti]

On Tue, Jul 29, 2008 at 12:39:53PM -0700, Roland Dreier wrote:
> I built with debugging symbols, and this seems to be an issue with SCSI
> disk emulation.  The traceback is:
> 
> #0  0x00007fc086d7dd10 in memcpy () from /lib/libc.so.6
> #1  0x00000000004a319b in cpu_physical_memory_rw (addr=108661608,
>     buf=0x7fff904ca190 "\224['\206\210\030z\006I�A", len=4, is_write=0)
>     at /users/rdreier/kvm-deb.git/qemu/exec.c:2847
> #2  0x000000000041f0c2 in lsi_execute_script (s=0x2ef7a30) at ../cpu-all.h:924
> #3  0x000000000049bd91 in qcow_aio_read_cb (opaque=0x3018d70, ret=0) at block-qcow2.c:840
> #4  0x000000000041cba0 in qemu_aio_poll () at /users/rdreier/kvm-deb.git/qemu/block-raw-posix.c:513
> #5  0x000000000040b38a in main_loop_wait (timeout=<value optimized out>)
>     at /users/rdreier/kvm-deb.git/qemu/vl.c:7777
> #6  0x00000000004f607a in kvm_main_loop () at /users/rdreier/kvm-deb.git/qemu/qemu-kvm.c:587
> #7  0x0000000000412b46 in main (argc=<value optimized out>, argv=0x7fff904cb0c8)
>     at /users/rdreier/kvm-deb.git/qemu/vl.c:7811
> 
> and no progress ever seems to be made (the same address is read over and
> over)
> 
> I'm trying again with IDE instead of SCSI disks.  But I would like to
> help debug the SCSI emulation... will look at it further later, and I'm
> happy to provide any info someone else could use.

Hi Roland,

Known problem:
http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html

Windows driver has scripts code which busy loops on main memory. Since
that executes in vcpu context, the cpu won't ever get a chance to change
the variable which the SCSI emulation is looping at.

The proposed hack, however, is half-assed (see Paul Brook's comments on
that thread).

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

 > Known problem:
 > http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html

 > Windows driver has scripts code which busy loops on main memory. Since
 > that executes in vcpu context, the cpu won't ever get a chance to change
 > the variable which the SCSI emulation is looping at.

Thanks Marcelo (and Alberto) for pointing me at this.

 > The proposed hack, however, is half-assed (see Paul Brook's comments on
 > that thread).

I'd be willing to take a stab at a better fix.  Is there any
documentation about the lsi microcode engine anywhere?  Would the right
fix to be to detect the busy loop on CPU memory and return from
lsi_execute_script() in that case?

BTW I tried using if=ide to install Windows XP and got a blue screen
during the installer.  What are people doing to run XP in a kvm guest?

 - R.

Re: Fresh install of Windows XP hangs early in boot?

[Marcelo Tosatti]

On Tue, Jul 29, 2008 at 03:00:06PM -0700, Roland Dreier wrote:
>  > Known problem:
>  > http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html
> 
>  > Windows driver has scripts code which busy loops on main memory. Since
>  > that executes in vcpu context, the cpu won't ever get a chance to change
>  > the variable which the SCSI emulation is looping at.
> 
> Thanks Marcelo (and Alberto) for pointing me at this.
> 
>  > The proposed hack, however, is half-assed (see Paul Brook's comments on
>  > that thread).
> 
> I'd be willing to take a stab at a better fix.  

Awesome! That would be very welcome. It is quite important for
better-than-IDE performance with Windows guests.

> Is there any documentation about the lsi microcode engine anywhere?

http://www.lsi.com/files/docs/techdocs/storage_stand_prod/SCSIControllers/56_3.pdf

> Would the right fix to be to detect the busy loop on CPU memory and
> return from lsi_execute_script() in that case?

Detect the busy loop, interrupt script execution but guaranteeing that
it will continue (by starting a timer to restart execution some time in
the future, for example).

The tricky part is that in between, SCRIPTS execution could have been
restarted by the guest via PIO/MMIO, or by an I/O completion event, so 
proper synchronization is needed.

The hack I did relied on the driver to restart execution by Signal
Process command (SIGP), but its possible that such command would never
be sent (as Paul Brook mentions in the thread).

> BTW I tried using if=ide to install Windows XP and got a blue screen
> during the installer.  What are people doing to run XP in a kvm guest?

Are you using a recent version of kvm-userspace/kernel modules? Please
save the blue screen and mail it to the list or fill a bug.

Thanks

Re: Fresh install of Windows XP hangs early in boot?

[Marcelo Tosatti]

On Tue, Jul 29, 2008 at 07:57:34PM -0300, Marcelo Tosatti wrote:
> On Tue, Jul 29, 2008 at 03:00:06PM -0700, Roland Dreier wrote:
> >  > Known problem:
> >  > http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html
> > 
> >  > Windows driver has scripts code which busy loops on main memory. Since
> >  > that executes in vcpu context, the cpu won't ever get a chance to change
> >  > the variable which the SCSI emulation is looping at.
> > 
> > Thanks Marcelo (and Alberto) for pointing me at this.
> > 
> >  > The proposed hack, however, is half-assed (see Paul Brook's comments on
> >  > that thread).
> > 
> > I'd be willing to take a stab at a better fix.  
> 
> Awesome! That would be very welcome. It is quite important for
> better-than-IDE performance with Windows guests.
> 
> > Is there any documentation about the lsi microcode engine anywhere?
> 
> http://www.lsi.com/files/docs/techdocs/storage_stand_prod/SCSIControllers/56_3.pdf

http://www.lsi.com/files/docs/techdocs/storage_stand_prod/SCSIControllers/53c896_3.pdf

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

 > > BTW I tried using if=ide to install Windows XP and got a blue screen
 . > during the installer.  What are people doing to run XP in a kvm guest?

Funnily enough installing XP SP2 with if=ide worked fine on an Intel
host.  I notice that I left off -std-vga on the working install too (I
used it on the AMD host that blue-screened).

Anyway, just another data point.  Let me know if any other data would help.

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

 > > BTW I tried using if=ide to install Windows XP and got a blue screen
 > > during the installer.  What are people doing to run XP in a kvm guest?

 > Are you using a recent version of kvm-userspace/kernel modules? Please
 > save the blue screen and mail it to the list or fill a bug.

Pretty recent... this is with kvm modules from vanilla mainline
2.6.27-rc1 and kvm-72 userspace from Debian.  Host is a 64-bit kernel
running on AMD CPU (no NPT).

Here's the bluescreen -- it seems the same as the last install, so
pretty reproducible:


[-- Attachment #2: xp-bluescreen.png --]
[-- Type: image/png, Size: 12879 bytes --]

Re: Fresh install of Windows XP hangs early in boot?

[Alberto Treviño]

On Tuesday 29 July 2008 04:00:06 pm Roland Dreier wrote:
> I'd be willing to take a stab at a better fix.

Thanks, Roland for volunteering.  I has been nearly 10 years since I worked 
on low-level hardware programming and I figured it would take me too long 
just to become familiar with SCRIPTS, then try to figure out the Windows 
"black box".  That's why I haven't taken a stab at it.

However, I'd be more than happy to test any patches you may have.  I have a 
couple of development environments where I could play around and test any 
fixes.

Re: Fresh install of Windows XP hangs early in boot?

[Chris Lalancette]

Roland Dreier wrote:
> BTW I tried using if=ide to install Windows XP and got a blue screen
> during the installer.  What are people doing to run XP in a kvm guest?

Hm, your comment later on makes me think you tried this on AMD.  If so, I have
also run into a similar problem with Windows guests under AMD.  After installing
WinDbg, it told me that it was a "Paging Request in Non-Paged memory" related to
the Video memory area.  Does yours look similar to that?  I have not had time to
track it further than that, though.

Chris Lalancette

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

 > Hm, your comment later on makes me think you tried this on AMD.  If so, I have
 > also run into a similar problem with Windows guests under AMD.  After installing
 > WinDbg, it told me that it was a "Paging Request in Non-Paged memory" related to
 > the Video memory area.  Does yours look similar to that?  I have not had time to
 > track it further than that, though.

Yes, I got a bluescreen on an AMD host but not an Intel host.  And the
bluescreen (I posted earlier) said "PAGE_FAULT_IN_NONPAGED_AREA"

 - R.

Re: Fresh install of Windows XP hangs early in boot?

[Avi Kivity]

Roland Dreier wrote:
>  > Hm, your comment later on makes me think you tried this on AMD.  If so, I have
>  > also run into a similar problem with Windows guests under AMD.  After installing
>  > WinDbg, it told me that it was a "Paging Request in Non-Paged memory" related to
>  > the Video memory area.  Does yours look similar to that?  I have not had time to
>  > track it further than that, though.
>
> Yes, I got a bluescreen on an AMD host but not an Intel host.  And the
> bluescreen (I posted earlier) said "PAGE_FAULT_IN_NONPAGED_AREA"
>
>   

I've had bad luck with outdated BIOSes on AMD.  If you have a Barcelona, 
try updating your BIOS.

-- 
error compiling committee.c: too many arguments to function

Re: Fresh install of Windows XP hangs early in boot?

[Roland Dreier]

 > Known problem:
 > http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html

I tried this hack (and actually made the magic insns number 500), and
doing an XP install I got

    lsi_scsi: error: Reselect with pending DMA

do you have any feeling if this is because the script execution got
stopped too soon?  Or is this likely a further issue?

 - R.

Re: Fresh install of Windows XP hangs early in boot?

[Marcelo Tosatti]

On Tue, Jul 29, 2008 at 10:27:14PM -0700, Roland Dreier wrote:
>  > Known problem:
>  > http://www.nabble.com/LSI:-avoid-infinite-loops-p17116605.html
> 
> I tried this hack (and actually made the magic insns number 500), and
> doing an XP install I got
> 
>     lsi_scsi: error: Reselect with pending DMA
> 
> do you have any feeling if this is because the script execution got
> stopped too soon?  Or is this likely a further issue?

Perhaps the context that was stopped due to > 500 instructions was
handling a DMA operation that didnt finish.

Since the hack won't restart execution by itself, the driver executed
a command (probably SIGP) to restart, leaving the DMA operation
unfinished.