Using KVM for Windows kernel debugging

Using KVM for Windows kernel debugging

[Tom Parkin]

Hi list,

I'm currently working on a project which involves Windows kernel
debugging.  Typically this is done using two PCs: one to run the code,
and the other to run the debugger.  The debug and target PCs are then
connected via serial, USB or 1394.

I'm trying to get the above configured using two KVM-based virtual
machines for the Windows hosts.  My research[1] suggests that this
should be possible, but so far I've not had much luck.  So far, I've
linked the two KVM hosts together using a virtualised serial
connection (that is: using the "-serial pipe" argument to kvm-qemu
with two pipes and socat acting as the interconnect[2]), and using
socat's debugging options I can see data passing between the hosts,
but the windows kernel debugger never achieves a connection.

Has anyone on the list got any experience of this kind of setup, or
any suggestions that might help ?

Thanks,
Tom

[1].  There are a few webpages which deal with the general principle,
even if they're not KVM-specific:

http://blogs.msdn.com/virtual_pc_guy/archive/2005/10/20/482413.aspx
http://blog.electric-cloud.com/2009/08/05/how-to-set-up-kernel-debugging-for-windows-in-vmware-esx/

[2].  To be precise,

socat config:

  $ socat PIPE:wdbg1 PIPE:wdbg2

KVM1:

  $ kvm-qemu -hda <image-1.img> -serial pipe:wdbg1

KVM2:

  $ kvm-qemu -hda <image-2.img> -serial pipe:wdbg2

-- 
Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

2009/8/17 Tom Parkin <tom.parkin@gmail.com>:
> Has anyone on the list got any experience of this kind of setup, or
> any suggestions that might help ?

Sorry for the spam, but I forgot to mention that I found this mail in
the archives which looks pertinent:

http://article.gmane.org/gmane.comp.emulators.kvm.devel/20770/match=windows

Sadly, the wiki link is no longer valid:

http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug

If anyone could tell me where that information now resides I'd appreciate it.

Thanks again !
Tom

Re: Using KVM for Windows kernel debugging

[Avi Kivity]

(copying smintz)

On 08/17/2009 06:16 PM, Tom Parkin wrote:
> 2009/8/17 Tom Parkin<tom.parkin@gmail.com>:
>    
>> Has anyone on the list got any experience of this kind of setup, or
>> any suggestions that might help ?
>>      
> Sorry for the spam, but I forgot to mention that I found this mail in
> the archives which looks pertinent:
>
> http://article.gmane.org/gmane.comp.emulators.kvm.devel/20770/match=windows
>
> Sadly, the wiki link is no longer valid:
>
> http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug
>
> If anyone could tell me where that information now resides I'd appreciate it.
>    

This wiki is not in http://www.linux-kvm.org.  Shahar, have we lost that 
page?

-- 
error compiling committee.c: too many arguments to function

RE: Using KVM for Windows kernel debugging

[Yan Vugenfirer]

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
> Behalf Of Tom Parkin
> Sent: Monday, August 17, 2009 6:16 PM
> To: kvm@vger.kernel.org
> Cc: tom.parkin@gmail.com
> Subject: Re: Using KVM for Windows kernel debugging
> 
> 2009/8/17 Tom Parkin <tom.parkin@gmail.com>:
> > Has anyone on the list got any experience of this kind of setup, or
> > any suggestions that might help ?
> 
> Sorry for the spam, but I forgot to mention that I found this mail in
> the archives which looks pertinent:
> 
> http://article.gmane.org/gmane.comp.emulators.kvm.devel/20770/match=win
> dows
> 
> Sadly, the wiki link is no longer valid:
> 
> http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug
> 
> If anyone could tell me where that information now resides I'd
> appreciate it.
[YV] I recreated the page:
http://www.linux-kvm.org/page/WindowsGuestDrivers/GuestDebugging

> 
> Thanks again !
> Tom
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

Thanks so much for that, Yan, it looks exactly like what I need.  I'll
give it a try when I'm back in the office.

Best regards,
Tom

On 18/08/2009, Yan Vugenfirer <yvugenfi@redhat.com> wrote:
>
>
>> -----Original Message-----
>> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
>> Behalf Of Tom Parkin
>> Sent: Monday, August 17, 2009 6:16 PM
>> To: kvm@vger.kernel.org
>> Cc: tom.parkin@gmail.com
>> Subject: Re: Using KVM for Windows kernel debugging
>>
>> 2009/8/17 Tom Parkin <tom.parkin@gmail.com>:
>> > Has anyone on the list got any experience of this kind of setup, or
>> > any suggestions that might help ?
>>
>> Sorry for the spam, but I forgot to mention that I found this mail in
>> the archives which looks pertinent:
>>
>> http://article.gmane.org/gmane.comp.emulators.kvm.devel/20770/match=win
>> dows
>>
>> Sadly, the wiki link is no longer valid:
>>
>> http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug
>>
>> If anyone could tell me where that information now resides I'd
>> appreciate it.
> [YV] I recreated the page:
> http://www.linux-kvm.org/page/WindowsGuestDrivers/GuestDebugging
>
>>
>> Thanks again !
>> Tom
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
Sent from my mobile device

Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

2009/8/17 Tom Parkin <tom.parkin@gmail.com>:
> Thanks so much for that, Yan, it looks exactly like what I need.  I'll
> give it a try when I'm back in the office.

Having given it a try, I'm having some troubles which I hope someone
may be able to assist with ?

Here's my configuration :

I have two Windows XP hosts running in two virtual machines[0].  One
is set up as the "debugee" to export debugging information via. COM1,
the other is set up as the "debugger" with the WinDBG kernel debugger
installed.

I have followed the instructions on the Wiki[1] for creating a virtual
serial connection between the two VMs, and I am able to send messages
between the two VMs using Hyperterm.

However, I am unable to successfully establish a connection between
the WinDBG debugger process and the debugee machine.  The best I've
managed so far is as follows :

  o Boot the debugger VM and start WinDBG
  o Boot the debugee VM
  o The debugee boots to the Windows bootloader screen.  Immediately
after that it appears to hang with a black screen, and it starts
chewing CPU
  o Wait for a short time (~1-2min), after which the WinDBG process
crashes on the debugger VM
  o Restart WinDBG and wait again for a short time (~1-2min).  Again,
WinDBG crashes
  o Restart WinDBG a third time.  This time the debugger window shows
"Kernel debugger connection established", although the window status
bar still shows "Debugee not connected"
  o Wait for some time (~5min), during which some further messages
come up in the debugger.  Eventually it seems to settle into a loop of
"GetContextState failed" with the occasional "Unable to read KTHREAD
address".

And that appears to be that.  I've left it to run for up to ~15 min,
during which time the debugee VM window never comes out of the
apparent black screen hang, and the debugee kvm process continues to
chew CPU, pretty much pegging one of my cores at 100%.

My questions:

   +  The Wiki mentions a patch to the kvm-qemu sources[2].  Looking
at the git tree it seems this change may be merged, so possibly this
patch isn't required any more.  Can anyone confirm this ?
   +  Does anyone have a working Windows guest debugging setup working
?  Could you share the details ?

Of course, any suggestions on how to debug the entire configuration
would be gratefully received !

Thanks,
Tom

[0].  I'm running ubuntu 9.04 with the distro-provided kvm package
version "1:84+dfsg-0ubuntu12.3".  I'm at somewhat of a loss to relate
this to actual kvm-qemu releases...  My kernel version is
2.6.28-14-generic.  My cpu is a AMD Turion(tm)X2 Ultra DualCore Mobile
ZM-86, and I'm running kvm_amd with the option "npt=0" to avoid kernel
oopses when starting VM images.

[1].  As provided by Yan previously;

http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug

[2].  The link in the Wiki is for a private IP (10.0.0.1) but I think
the patch is probably the same as the one referenced here:

http://www.damogran.de/blog/archives/14-WinDbg-and-QEMU.html

-- 
Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

RE: Using KVM for Windows kernel debugging

[Yan Vugenfirer]

In case you don’t need to debug the boot process you could wait for the
Windows to  start and only then start Windbg configure the communication
options and hit break (ctrl+break) to initiate debug session.

Symbols could be downloaded from:
http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx


Best regards,
Yan.

From: Vadim Rozenfeld [mailto:vrozenfe@redhat.com]
Sent: Tuesday, August 18, 2009 2:02 PM
To: Tom Parkin
Cc: Yan Vugenfirer; kvm@vger.kernel.org
Subject: Re: Using KVM for Windows kernel debugging

On 08/18/2009 01:52 PM, Tom Parkin wrote:
2009/8/17 Tom Parkin <tom.parkin@gmail.com>:

Thanks so much for that, Yan, it looks exactly like what I need.  I'll
give it a try when I'm back in the office.


Having given it a try, I'm having some troubles which I hope someone
may be able to assist with ?

Here's my configuration :

I have two Windows XP hosts running in two virtual machines[0].  One
is set up as the "debugee" to export debugging information via. COM1,
the other is set up as the "debugger" with the WinDBG kernel debugger
installed.

I have followed the instructions on the Wiki[1] for creating a virtual
serial connection between the two VMs, and I am able to send messages
between the two VMs using Hyperterm.

However, I am unable to successfully establish a connection between
the WinDBG debugger process and the debugee machine.  The best I've
managed so far is as follows :

  o Boot the debugger VM and start WinDBG
  o Boot the debugee VM
  o The debugee boots to the Windows bootloader screen.  Immediately
after that it appears to hang with a black screen, and it starts
chewing CPU
  o Wait for a short time (~1-2min), after which the WinDBG process
crashes on the debugger VM
  o Restart WinDBG and wait again for a short time (~1-2min).  Again,
WinDBG crashes
  o Restart WinDBG a third time.  This time the debugger window shows
"Kernel debugger connection established", although the window status
bar still shows "Debugee not connected"

Try to download symbols first.

  o Wait for some time (~5min), during which some further messages
come up in the debugger.  Eventually it seems to settle into a loop of
"GetContextState failed" with the occasional "Unable to read KTHREAD
address".

Host (WinDbg) and target are running out-of-sync.


And that appears to be that.  I've left it to run for up to ~15 min,
during which time the debugee VM window never comes out of the
apparent black screen hang, and the debugee kvm process continues to
chew CPU, pretty much pegging one of my cores at 100%.

My questions:

   +  The Wiki mentions a patch to the kvm-qemu sources[2].  Looking
at the git tree it seems this change may be merged, so possibly this
patch isn't required any more.  Can anyone confirm this ?
   +  Does anyone have a working Windows guest debugging setup working
?  Could you share the details ?

Of course, any suggestions on how to debug the entire configuration
would be gratefully received !

Thanks,
Tom

[0].  I'm running ubuntu 9.04 with the distro-provided kvm package
version "1:84+dfsg-0ubuntu12.3".  I'm at somewhat of a loss to relate
this to actual kvm-qemu releases...  My kernel version is
2.6.28-14-generic.  My cpu is a AMD Turion(tm)X2 Ultra DualCore Mobile
ZM-86, and I'm running kvm_amd with the option "npt=0" to avoid kernel
oopses when starting VM images.

[1].  As provided by Yan previously;

http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug

[2].  The link in the Wiki is for a private IP (10.0.0.1) but I think
the patch is probably the same as the one referenced here:

http://www.damogran.de/blog/archives/14-WinDbg-and-QEMU.html

Re: Using KVM for Windows kernel debugging

[Vadim Rozenfeld]

On 08/18/2009 01:52 PM, Tom Parkin wrote:
> 2009/8/17 Tom Parkin<tom.parkin@gmail.com>:
>    
>> Thanks so much for that, Yan, it looks exactly like what I need.  I'll
>> give it a try when I'm back in the office.
>>      
>
> Having given it a try, I'm having some troubles which I hope someone
> may be able to assist with ?
>
> Here's my configuration :
>
> I have two Windows XP hosts running in two virtual machines[0].  One
> is set up as the "debugee" to export debugging information via. COM1,
> the other is set up as the "debugger" with the WinDBG kernel debugger
> installed.
>
> I have followed the instructions on the Wiki[1] for creating a virtual
> serial connection between the two VMs, and I am able to send messages
> between the two VMs using Hyperterm.
>
> However, I am unable to successfully establish a connection between
> the WinDBG debugger process and the debugee machine.  The best I've
> managed so far is as follows :
>
>    o Boot the debugger VM and start WinDBG
>    o Boot the debugee VM
>    o The debugee boots to the Windows bootloader screen.  Immediately
> after that it appears to hang with a black screen, and it starts
> chewing CPU
>    o Wait for a short time (~1-2min), after which the WinDBG process
> crashes on the debugger VM
>    o Restart WinDBG and wait again for a short time (~1-2min).  Again,
> WinDBG crashes
>    o Restart WinDBG a third time.  This time the debugger window shows
> "Kernel debugger connection established", although the window status
> bar still shows "Debugee not connected"
>    
Try to get MS symbols first.
>    o Wait for some time (~5min), during which some further messages
> come up in the debugger.  Eventually it seems to settle into a loop of
> "GetContextState failed" with the occasional "Unable to read KTHREAD
> address".
>    
Could be a timing issue. Probably host (WinDbg) and target are running 
out-of-sync.
Try to add /break switch to boot.ini or bcdedit.
> And that appears to be that.  I've left it to run for up to ~15 min,
> during which time the debugee VM window never comes out of the
> apparent black screen hang, and the debugee kvm process continues to
> chew CPU, pretty much pegging one of my cores at 100%.
>
> My questions:
>
>     +  The Wiki mentions a patch to the kvm-qemu sources[2].  Looking
> at the git tree it seems this change may be merged, so possibly this
> patch isn't required any more.  Can anyone confirm this ?
>     +  Does anyone have a working Windows guest debugging setup working
> ?  Could you share the details ?
>
> Of course, any suggestions on how to debug the entire configuration
> would be gratefully received !
>
> Thanks,
> Tom
>
> [0].  I'm running ubuntu 9.04 with the distro-provided kvm package
> version "1:84+dfsg-0ubuntu12.3".  I'm at somewhat of a loss to relate
> this to actual kvm-qemu releases...  My kernel version is
> 2.6.28-14-generic.  My cpu is a AMD Turion(tm)X2 Ultra DualCore Mobile
> ZM-86, and I'm running kvm_amd with the option "npt=0" to avoid kernel
> oopses when starting VM images.
>
> [1].  As provided by Yan previously;
>
> http://kvm.qumranet.com/kvmwiki/WindowsGuestDebug
>
> [2].  The link in the Wiki is for a private IP (10.0.0.1) but I think
> the patch is probably the same as the one referenced here:
>
> http://www.damogran.de/blog/archives/14-WinDbg-and-QEMU.html
>
>    

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

Hi Vadim,

2009/8/18 Vadim Rozenfeld <vrozenfe@redhat.com>:
> Try to download symbols first.

Thanks for the tip, it gets me a bit closer -- although still not
fully up and running !

With the symbols installed, and the windbg symbol path set, the windbg
process doesn't exit, and does print "Connected to Windows XP ..."
(although the window status bar still reports "Debuggee not
connected").

However, other than that I have similar behavior as previously
reported -- the target VM hangs on a black screen chewing CPU, and in
time the windbg process starts reporting "GetContextState failed...".

Furthermore, I tried using kd rather than windbg to attempt to connect
to the target machine:

kd.exe -y <my-symbols> -k <comport spec>

I get different errors from kd:

"Unable to read head of debugger data list, Win32 error 0n56"

Tom
-- 
Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

Hi Yan,

2009/8/18 Yan Vugenfirer <yvugenfi@redhat.com>:
> In case you don’t need to debug the boot process you could wait for the
> Windows to  start and only then start Windbg configure the communication
> options and hit break (ctrl+break) to initiate debug session.

That would be fine; I have no requirement to debug the boot process
currently.  I'm actually trying to debug a driver segfault which
causes a bluescreen on load, and I can control when the driver is
loaded using userspace applications.

However, when I tried the above, I get:

   Failed to open \\.\com1
   Kernel debugger failed initialisation, Win32 error 0n5 "Access is denied."

However, I have administrative rights on the machine.  Just in case
the error actually relates to the target machine, I double checked
with kdbgctl.exe that kernel debugging is allowed on the target, and
it reports that full access is permitted.

Tom
-- 
Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

Re: Using KVM for Windows kernel debugging

[Vadim Rozenfeld]

On 08/18/2009 04:54 PM, Tom Parkin wrote:
> Hi Vadim,
>
> 2009/8/18 Vadim Rozenfeld<vrozenfe@redhat.com>:
>    
>> Try to download symbols first.
>>      
>
> Thanks for the tip, it gets me a bit closer -- although still not
> fully up and running !
>
> With the symbols installed, and the windbg symbol path set, the windbg
> process doesn't exit, and does print "Connected to Windows XP ..."
> (although the window status bar still reports "Debuggee not
> connected").
>    
did you try /break switch in boot.ini?
> However, other than that I have similar behavior as previously
> reported -- the target VM hangs on a black screen chewing CPU, and in
> time the windbg process starts reporting "GetContextState failed...".
>
> Furthermore, I tried using kd rather than windbg to attempt to connect
> to the target machine:
>    
kd and WinDbg use the same debug engine the only one difference is in 
user interface. However, kd is much faster.
> kd.exe -y<my-symbols>  -k<comport spec>
>
> I get different errors from kd:
>
> "Unable to read head of debugger data list, Win32 error 0n56"
>
>    
What is your guest OS? Are you sure you have correct symbols?

If you don't need live debugging, you can set up your system to generate 
crash dump file and use it
for post-mortem debugging session.

Regards,
Vadim
> Tom
>    

Re: Using KVM for Windows kernel debugging

[Tom Parkin]

2009/8/18 Vadim Rozenfeld <vrozenfe@redhat.com>:
> did you try /break switch in boot.ini?

It's a new one to me.  I've just given it a go and found no change in
behavior, from which I deduce that the target machine isn't getting as
far even as the HAL init (although this might be an incorrect reading
of the /break option).

> kd and WinDbg use the same debug engine the only one difference is in user
> interface. However, kd is much faster.

That's worthwhile to know, thanks :-)

I was mainly trying kd to see whether it reported any different/extra
information on stderr that windbg was silently swallowing.

> What is your guest OS? Are you sure you got right symbols?

My guest OS is Windows XP SP3.  I have downloaded
WindowsXP-KB936929-SP3-x86-symbols-full-ENU.exe from the Microsoft
website, which I think should be correct.

I'll keep trying to get the connection up and running.  In the
meantime, a kernel memory dump combined with after-the-fact debugging
should be enough to fix the errors I have currently.

Thanks again for your suggestions and advice,
Tom
-- 
Tom Parkin
www.thhp.org.uk
The worst moment for the atheist is when he is really thankful and has
nobody to thank /Rossetti/

Re: Using KVM for Windows kernel debugging

[duck]

I have never got this to work reliably. Occasionally I can get as far as 
making a debugger connection at boot-time, IIRC, but have never managed to 
use the deugger at all. You always seem to end up in some 
debugger-debuggee deadlock.

I suspect that the serial link simulation is imperfect enough (lost 
interrupts and thus lost characters?) that you are out of luck for 
serial-link-based remote debugging, even though userland serial comms, 
e.g. using Hyperterm, seem to work fine.

I was able to get debugging working, a year or two ago, under QEMU using 
the MIN_CYCLE_COUNT patch (I made a command line option called '-mcc' to 
allow this to be tuned at run-time) but IIRC this patch is irrelevant when 
KVM is used.

Can anyone suggest a KVM patch which might help, even if it slows the 
debuggee VM down?
 
Pity SoftICE got killed off :-) 

You could try http://www.sysersoft.com/ for a one-PC Windows kernel 
debugging solution. Seems to be a small Beijing software house which just 
sells on-line. Their product is not free, though there is a free trial. 
(I haven't tried it under KVM, though they advertise that it works 
under VxWxxx [censored]).
 

Re: Using KVM for Windows kernel debugging

[Vadim Rozenfeld]

On 08/19/2009 01:31 AM, duck wrote:
> I have never got this to work reliably. Occasionally I can get as far as
> making a debugger connection at boot-time, IIRC, but have never managed to
> use the deugger at all. You always seem to end up in some
> debugger-debuggee deadlock.
>    
It works.
However, getting into debug session is real pain in the neck.
Especially for the first time.
> I suspect that the serial link simulation is imperfect enough (lost
> interrupts and thus lost characters?) that you are out of luck for
> serial-link-based remote debugging, even though userland serial comms,
> e.g. using Hyperterm, seem to work fine.
>
> I was able to get debugging working, a year or two ago, under QEMU using
> the MIN_CYCLE_COUNT patch (I made a command line option called '-mcc' to
> allow this to be tuned at run-time) but IIRC this patch is irrelevant when
> KVM is used.
>
> Can anyone suggest a KVM patch which might help, even if it slows the
> debuggee VM down?
>
> Pity SoftICE got killed off :-)
>
> You could try http://www.sysersoft.com/ for a one-PC Windows kernel
> debugging solution. Seems to be a small Beijing software house which just
> sells on-line. Their product is not free, though there is a free trial.
> (I haven't tried it under KVM, though they advertise that it works
> under VxWxxx [censored]).
>    
Maybe we can use vm kd extensions project 
(http://www.nynaeve.net/?page_id=168)
as a starting point and fit it into kvm. AFAIK, VmWare and Virtual Box 
have done it already.
>    
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>