debugging windows guests

debugging windows guests

[Raindog]

Hello,

I am researching KVM as a malware analysis platform and had some 
questions about debugging the guest OS. In my case I intend to use 
windows guests. So my questsions are as follows:

Questions:

1. What instrumentation facilities are their available?

2. Is it possible to extend the debugging interface so that debugging is 
more transparent to the guest OS? IE: there is still a limit of 4 HW 
breakpoints (which makes me wonder why a LIST is used for them...)

3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU 
at a low level, for example, for writing custom tracers, etc. Is there 
one? Or is there something similar?


Bugs:

1. I hit a bug w/ instruction logging using a RAM based temp folder. If 
I ran w/ the following command line:
(Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))

qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm

It would successfully log to the tmp log file, but obviously, KVM would 
be disabled.

If I use sudo, it won't log to the file, is this a known issue?

2. -enable-nesting on AMD hardware using a xen guest OS causes xen to 
GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based 
guests?

RE: debugging windows guests

[Yan Vugenfirer]

> -----Original Message-----
> From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] On
> Behalf Of Raindog
> Sent: Tuesday, December 15, 2009 2:25 AM
> To: kvm@vger.kernel.org
> Subject: debugging windows guests
> 
> Hello,
> 
> I am researching KVM as a malware analysis platform and had some
> questions about debugging the guest OS. In my case I intend to use
> windows guests. So my questsions are as follows:
> 
> Questions:
> 
> 1. What instrumentation facilities are their available?

[YV] http://www.linux-kvm.org/page/WindowsGuestDrivers/GuestDebugging

> 
> 2. Is it possible to extend the debugging interface so that debugging
> is
> more transparent to the guest OS? IE: there is still a limit of 4 HW
> breakpoints (which makes me wonder why a LIST is used for them...)
> 
> 3. I'm not finding any published API for interfacing with
> KVM/KQEMU/QEMU
> at a low level, for example, for writing custom tracers, etc. Is there
> one? Or is there something similar?
> 
> 
> Bugs:
> 
> 1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> I ran w/ the following command line:
> (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> 
> qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
> 
> It would successfully log to the tmp log file, but obviously, KVM would
> be disabled.
> 
> If I use sudo, it won't log to the file, is this a known issue?
> 
> 2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> guests?
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: debugging windows guests

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2610 bytes --]

Raindog wrote:
> Hello,
> 
> I am researching KVM as a malware analysis platform and had some
> questions about debugging the guest OS. In my case I intend to use
> windows guests. So my questsions are as follows:
> 
> Questions:
> 
> 1. What instrumentation facilities are their available?
> 
> 2. Is it possible to extend the debugging interface so that debugging is
> more transparent to the guest OS? IE: there is still a limit of 4 HW
> breakpoints (which makes me wonder why a LIST is used for them...)

In accelerated KVM mode, the x86 architecture restricts us to 4 break-
or watchpoints that can be active at the same time. If you switch to
emulation mode, there are no such limits. Actually, I just made use of
this for debugging a subtle stack corruption in a guest, and I had more
than 70 watchpoints active at the same time. It's just "slightly" slower
than KVM...

> 
> 3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
> at a low level, for example, for writing custom tracers, etc. Is there
> one? Or is there something similar?

KVM provides tracepoints for the Linux ftrace framework, see related
documentation of the kernel. If you extend your guest to issue certain
events that the hypervisor sees and traces (e.g. writes to pseudo I/O
ports), you can also trace things inside the guest that are otherwise
invisible to the host. I once hacked up an ad-hoc tracing by means of
hypercalls (required some kvm patching). That also worked from guest
userspace - and revealed that even more hypercalls could be called that
way (that's fixed in KVM now).

> 
> 
> Bugs:
> 
> 1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> I ran w/ the following command line:
> (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> 
> qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm

-d only works in emulation mode as it relies on dynamic code translation
(TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
upstream QEMU, it's the other way around).

> 
> It would successfully log to the tmp log file, but obviously, KVM would
> be disabled.
> 
> If I use sudo, it won't log to the file, is this a known issue?
> 
> 2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> guests?

If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot
of nested SVM fixes went in recently. If it still fails, put Alex (Graf)
and Joerg (Roedel) on CC.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: debugging windows guests

[Raindog]

On 12/15/2009 3:39 PM, Jan Kiszka wrote:
> Raindog wrote:
> >  Hello,
> >
> >  I am researching KVM as a malware analysis platform and had some
> >  questions about debugging the guest OS. In my case I intend to use
> >  windows guests. So my questsions are as follows:
> >
> >  Questions:
> >
> >  1. What instrumentation facilities are their available?
> >
> >  2. Is it possible to extend the debugging interface so that debugging is
> >  more transparent to the guest OS? IE: there is still a limit of 4 HW
> >  breakpoints (which makes me wonder why a LIST is used for them...)
>
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
>    

Ok, is there anything special that needs to be done to enable additional 
watchpoints as they are being called? How are these set btw? Is it 
accomplished transparently through gdb? IE: if you set a watchpoint at a 
specific address, under emulation mode, they simulate HW bps in that no 
code is modified via the injection of an int 3?

> >
> >  3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
> >  at a low level, for example, for writing custom tracers, etc. Is there
> >  one? Or is there something similar?
>
> KVM provides tracepoints for the Linux ftrace framework, see related
> documentation of the kernel.

I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt 
but that can hardly be accused of being called documentation. I don't 
think something like this: 
http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is 
unreasonable.

> If you extend your guest
Windows is by design not extensible.

>   to issue certain
> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
> ports), you can also trace things inside the guest that are otherwise
> invisible to the host. I once hacked up an ad-hoc tracing by means of
> hypercalls (required some kvm patching). That also worked from guest
> userspace - and revealed that even more hypercalls could be called that
> way (that's fixed in KVM now).
>
> >
> >
> >  Bugs:
> >
> >  1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> >  I ran w/ the following command line:
> >  (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> >
> >  qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>
> -d only works in emulation mode as it relies on dynamic code translation
> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
> upstream QEMU, it's the other way around).
>    

Hence why running w/out admin rights enables the logging. I was confused 
because the logfile is still created w/ the kvm module disabled.

> >
> >  It would successfully log to the tmp log file, but obviously, KVM would
> >  be disabled.
> >
> >  If I use sudo, it won't log to the file, is this a known issue?
> >
> >  2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> >  GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> >  guests?
>
> If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot
> of nested SVM fixes went in recently. If it still fails, put Alex (Graf)
> and Joerg (Roedel) on CC.
>    

Upgrading now...

> Jan
>
>    

Re: debugging windows guests

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3888 bytes --]

Raindog wrote:
> On 12/15/2009 3:39 PM, Jan Kiszka wrote:
>> Raindog wrote:
>> >  Hello,
>> >
>> >  I am researching KVM as a malware analysis platform and had some
>> >  questions about debugging the guest OS. In my case I intend to use
>> >  windows guests. So my questsions are as follows:
>> >
>> >  Questions:
>> >
>> >  1. What instrumentation facilities are their available?
>> >
>> >  2. Is it possible to extend the debugging interface so that
>> debugging is
>> >  more transparent to the guest OS? IE: there is still a limit of 4 HW
>> >  breakpoints (which makes me wonder why a LIST is used for them...)
>>
>> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
>> or watchpoints that can be active at the same time. If you switch to
>> emulation mode, there are no such limits. Actually, I just made use of
>> this for debugging a subtle stack corruption in a guest, and I had more
>> than 70 watchpoints active at the same time. It's just "slightly" slower
>> than KVM...
>>    
> 
> Ok, is there anything special that needs to be done to enable additional
> watchpoints as they are being called? How are these set btw? Is it
> accomplished transparently through gdb? IE: if you set a watchpoint at a
> specific address, under emulation mode, they simulate HW bps in that no
> code is modified via the injection of an int 3?

Yes, break- and watchpoints are transparent to the guest in emulation
mode. In KVM mode, hardware breakpoints do not require int 3, but they
are limited and the guest my notice that its own breakpoints have no
effect as long as the host injected some.

> 
>> >
>> >  3. I'm not finding any published API for interfacing with
>> KVM/KQEMU/QEMU
>> >  at a low level, for example, for writing custom tracers, etc. Is there
>> >  one? Or is there something similar?
>>
>> KVM provides tracepoints for the Linux ftrace framework, see related
>> documentation of the kernel.
> 
> I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt
> but that can hardly be accused of being called documentation. I don't
> think something like this:
> http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is
> unreasonable.

2.6.27 is too old anyway. There should be at least one LWN.net article
on this, and also quite a few presentations and paper, just ask your
favorite search engine.

> 
>> If you extend your guest
> Windows is by design not extensible.

Depends on where and how you want to hook into it. Of course, its kernel
is out of reach. But if you are interested e.g. in marking specific I/O
requests, you could write your own driver and hook into the stack.

> 
>>   to issue certain
>> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
>> ports), you can also trace things inside the guest that are otherwise
>> invisible to the host. I once hacked up an ad-hoc tracing by means of
>> hypercalls (required some kvm patching). That also worked from guest
>> userspace - and revealed that even more hypercalls could be called that
>> way (that's fixed in KVM now).
>>
>> >
>> >
>> >  Bugs:
>> >
>> >  1. I hit a bug w/ instruction logging using a RAM based temp
>> folder. If
>> >  I ran w/ the following command line:
>> >  (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
>> >
>> >  qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>>
>> -d only works in emulation mode as it relies on dynamic code translation
>> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
>> upstream QEMU, it's the other way around).
>>    
> 
> Hence why running w/out admin rights enables the logging. I was confused
> because the logfile is still created w/ the kvm module disabled.

It's created but remains empty for obvious reasons in KVM mode.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: debugging windows guests

[Vadim Rozenfeld]

On Wed, 2009-12-16 at 00:39 +0100, Jan Kiszka wrote:
> Raindog wrote:
> > Hello,
> > 
> > I am researching KVM as a malware analysis platform and had some
> > questions about debugging the guest OS. In my case I intend to use
> > windows guests. So my questsions are as follows:
> > 
> > Questions:
> > 
> > 1. What instrumentation facilities are their available?
> > 
> > 2. Is it possible to extend the debugging interface so that debugging is
> > more transparent to the guest OS? IE: there is still a limit of 4 HW
> > breakpoints (which makes me wonder why a LIST is used for them...)
> 
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
> 
> > 
> > 3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
> > at a low level, for example, for writing custom tracers, etc. Is there
> > one? Or is there something similar?
> 
> KVM provides tracepoints for the Linux ftrace framework, see related
> documentation of the kernel. If you extend your guest to issue certain
> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
> ports), you can also trace things inside the guest that are otherwise
> invisible to the host.
You can WRITE_PORT_BUFFER_UCHAR to com1/com2 port when you are in kernel
mode. 
>  I once hacked up an ad-hoc tracing by means of
> hypercalls (required some kvm patching). That also worked from guest
> userspace - and revealed that even more hypercalls could be called that
> way (that's fixed in KVM now).
> 
> > 
> > 
> > Bugs:
> > 
> > 1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> > I ran w/ the following command line:
> > (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> > 
> > qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
> 
> -d only works in emulation mode as it relies on dynamic code translation
> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
> upstream QEMU, it's the other way around).
> 
> > 
> > It would successfully log to the tmp log file, but obviously, KVM would
> > be disabled.
> > 
> > If I use sudo, it won't log to the file, is this a known issue?
> > 
> > 2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> > GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> > guests?
> 
> If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot
> of nested SVM fixes went in recently. If it still fails, put Alex (Graf)
> and Joerg (Roedel) on CC.
> 
> Jan
> 

Re: debugging windows guests

[Alexander Graf]

Am 16.12.2009 um 09:14 schrieb Vadim Rozenfeld <vrozenfe@redhat.com>:

> On Wed, 2009-12-16 at 00:39 +0100, Jan Kiszka wrote:
>> Raindog wrote:
>>> Hello,
>>>
>>> I am researching KVM as a malware analysis platform and had some
>>> questions about debugging the guest OS. In my case I intend to use
>>> windows guests. So my questsions are as follows:
>>>
>>> Questions:
>>>
>>> 1. What instrumentation facilities are their available?
>>>
>>> 2. Is it possible to extend the debugging interface so that  
>>> debugging is
>>> more transparent to the guest OS? IE: there is still a limit of 4 HW
>>> breakpoints (which makes me wonder why a LIST is used for them...)
>>
>> In accelerated KVM mode, the x86 architecture restricts us to 4  
>> break-
>> or watchpoints that can be active at the same time. If you switch to
>> emulation mode, there are no such limits. Actually, I just made use  
>> of
>> this for debugging a subtle stack corruption in a guest, and I had  
>> more
>> than 70 watchpoints active at the same time. It's just "slightly"  
>> slower
>> than KVM...
>>
>>>
>>> 3. I'm not finding any published API for interfacing with KVM/ 
>>> KQEMU/QEMU
>>> at a low level, for example, for writing custom tracers, etc. Is  
>>> there
>>> one? Or is there something similar?
>>
>> KVM provides tracepoints for the Linux ftrace framework, see related
>> documentation of the kernel. If you extend your guest to issue  
>> certain
>> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
>> ports), you can also trace things inside the guest that are otherwise
>> invisible to the host.
> You can WRITE_PORT_BUFFER_UCHAR to com1/com2 port when you are in  
> kernel
> mode.
>> I once hacked up an ad-hoc tracing by means of
>> hypercalls (required some kvm patching). That also worked from guest
>> userspace - and revealed that even more hypercalls could be called  
>> that
>> way (that's fixed in KVM now).
>>
>>>
>>>
>>> Bugs:
>>>
>>> 1. I hit a bug w/ instruction logging using a RAM based temp  
>>> folder. If
>>> I ran w/ the following command line:
>>> (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
>>>
>>> qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>>
>> -d only works in emulation mode as it relies on dynamic code  
>> translation
>> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
>> upstream QEMU, it's the other way around).
>>
>>>
>>> It would successfully log to the tmp log file, but obviously, KVM  
>>> would
>>> be disabled.
>>>
>>> If I use sudo, it won't log to the file, is this a known issue?
>>>
>>> 2. -enable-nesting on AMD hardware using a xen guest OS causes xen  
>>> to
>>> GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen  
>>> based
>>> guests?
>>
>> If your host kernel or kvm-kmod is not 2.6.32 based, update first.  
>> A lot
>> of nested SVM fixes went in recently. If it still fails, put Alex  
>> (Graf)
>> and Joerg (Roedel) on CC.

Also make sure you pass nested=1 to kvm-amd.ko.

Xen definitely worked for me, so you're probably just missing one of  
the many magic bits :-).

Alex

Re: debugging windows guests

[Raindog]

On 12/15/2009 3:39 PM, Jan Kiszka wrote:
> Raindog wrote:
> >  Hello,
> >
> >  I am researching KVM as a malware analysis platform and had some
> >  questions about debugging the guest OS. In my case I intend to use
> >  windows guests. So my questsions are as follows:
> >
> >  Questions:
> >
> >  1. What instrumentation facilities are their available?
> >
> >  2. Is it possible to extend the debugging interface so that debugging is
> >  more transparent to the guest OS? IE: there is still a limit of 4 HW
> >  breakpoints (which makes me wonder why a LIST is used for them...)
>
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
>    

Are there any advantages over stock qemu if using kvm w/out the kernel 
module?

Re: debugging windows guests

[Avi Kivity]

On 12/17/2009 12:06 AM, Raindog wrote:
>
> Are there any advantages over stock qemu if using kvm w/out the kernel 
> module?

No.  qemu-kvm is not tested without kvm, so there may be disadvantages.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: debugging windows guests

[Raindog]

On 12/16/2009 9:36 PM, Avi Kivity wrote:
> On 12/17/2009 12:06 AM, Raindog wrote:
>>
>> Are there any advantages over stock qemu if using kvm w/out the 
>> kernel module?
>
> No.  qemu-kvm is not tested without kvm, so there may be disadvantages.
>

Does that then imply that svm emulation (--enable-nesting) is not well 
tested either?

Re: debugging windows guests

[Avi Kivity]

On 12/17/2009 09:05 AM, Raindog wrote:
> On 12/16/2009 9:36 PM, Avi Kivity wrote:
>> On 12/17/2009 12:06 AM, Raindog wrote:
>>>
>>> Are there any advantages over stock qemu if using kvm w/out the 
>>> kernel module?
>>
>> No.  qemu-kvm is not tested without kvm, so there may be disadvantages.
>>
>
> Does that then imply that svm emulation (--enable-nesting) is not well 
> tested either?

It's an unrelated feature (but I wouldn't say it is heavily tested).

-- 
error compiling committee.c: too many arguments to function