* Re: debugging windows guests
2009-12-15 23:39 ` Jan Kiszka
@ 2009-12-16 4:07 ` Raindog
2009-12-16 8:12 ` Jan Kiszka
2009-12-16 8:14 ` Vadim Rozenfeld
2009-12-16 22:06 ` Raindog
2 siblings, 1 reply; 11+ messages in thread
From: Raindog @ 2009-12-16 4:07 UTC (permalink / raw)
To: kvm
On 12/15/2009 3:39 PM, Jan Kiszka wrote:
> Raindog wrote:
> > Hello,
> >
> > I am researching KVM as a malware analysis platform and had some
> > questions about debugging the guest OS. In my case I intend to use
> > windows guests. So my questsions are as follows:
> >
> > Questions:
> >
> > 1. What instrumentation facilities are their available?
> >
> > 2. Is it possible to extend the debugging interface so that debugging is
> > more transparent to the guest OS? IE: there is still a limit of 4 HW
> > breakpoints (which makes me wonder why a LIST is used for them...)
>
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
>
Ok, is there anything special that needs to be done to enable additional
watchpoints as they are being called? How are these set btw? Is it
accomplished transparently through gdb? IE: if you set a watchpoint at a
specific address, under emulation mode, they simulate HW bps in that no
code is modified via the injection of an int 3?
> >
> > 3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
> > at a low level, for example, for writing custom tracers, etc. Is there
> > one? Or is there something similar?
>
> KVM provides tracepoints for the Linux ftrace framework, see related
> documentation of the kernel.
I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt
but that can hardly be accused of being called documentation. I don't
think something like this:
http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is
unreasonable.
> If you extend your guest
Windows is by design not extensible.
> to issue certain
> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
> ports), you can also trace things inside the guest that are otherwise
> invisible to the host. I once hacked up an ad-hoc tracing by means of
> hypercalls (required some kvm patching). That also worked from guest
> userspace - and revealed that even more hypercalls could be called that
> way (that's fixed in KVM now).
>
> >
> >
> > Bugs:
> >
> > 1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> > I ran w/ the following command line:
> > (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> >
> > qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>
> -d only works in emulation mode as it relies on dynamic code translation
> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
> upstream QEMU, it's the other way around).
>
Hence why running w/out admin rights enables the logging. I was confused
because the logfile is still created w/ the kvm module disabled.
> >
> > It would successfully log to the tmp log file, but obviously, KVM would
> > be disabled.
> >
> > If I use sudo, it won't log to the file, is this a known issue?
> >
> > 2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> > GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> > guests?
>
> If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot
> of nested SVM fixes went in recently. If it still fails, put Alex (Graf)
> and Joerg (Roedel) on CC.
>
Upgrading now...
> Jan
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-16 4:07 ` Raindog
@ 2009-12-16 8:12 ` Jan Kiszka
0 siblings, 0 replies; 11+ messages in thread
From: Jan Kiszka @ 2009-12-16 8:12 UTC (permalink / raw)
To: Raindog; +Cc: kvm
[-- Attachment #1: Type: text/plain, Size: 3888 bytes --]
Raindog wrote:
> On 12/15/2009 3:39 PM, Jan Kiszka wrote:
>> Raindog wrote:
>> > Hello,
>> >
>> > I am researching KVM as a malware analysis platform and had some
>> > questions about debugging the guest OS. In my case I intend to use
>> > windows guests. So my questsions are as follows:
>> >
>> > Questions:
>> >
>> > 1. What instrumentation facilities are their available?
>> >
>> > 2. Is it possible to extend the debugging interface so that
>> debugging is
>> > more transparent to the guest OS? IE: there is still a limit of 4 HW
>> > breakpoints (which makes me wonder why a LIST is used for them...)
>>
>> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
>> or watchpoints that can be active at the same time. If you switch to
>> emulation mode, there are no such limits. Actually, I just made use of
>> this for debugging a subtle stack corruption in a guest, and I had more
>> than 70 watchpoints active at the same time. It's just "slightly" slower
>> than KVM...
>>
>
> Ok, is there anything special that needs to be done to enable additional
> watchpoints as they are being called? How are these set btw? Is it
> accomplished transparently through gdb? IE: if you set a watchpoint at a
> specific address, under emulation mode, they simulate HW bps in that no
> code is modified via the injection of an int 3?
Yes, break- and watchpoints are transparent to the guest in emulation
mode. In KVM mode, hardware breakpoints do not require int 3, but they
are limited and the guest my notice that its own breakpoints have no
effect as long as the host injected some.
>
>> >
>> > 3. I'm not finding any published API for interfacing with
>> KVM/KQEMU/QEMU
>> > at a low level, for example, for writing custom tracers, etc. Is there
>> > one? Or is there something similar?
>>
>> KVM provides tracepoints for the Linux ftrace framework, see related
>> documentation of the kernel.
>
> I found this http://lxr.linux.no/#linux+v2.6.27/Documentation/ftrace.txt
> but that can hardly be accused of being called documentation. I don't
> think something like this:
> http://www.pintool.org/tutorials/asplos08/slides/PinTutorial.pdf is
> unreasonable.
2.6.27 is too old anyway. There should be at least one LWN.net article
on this, and also quite a few presentations and paper, just ask your
favorite search engine.
>
>> If you extend your guest
> Windows is by design not extensible.
Depends on where and how you want to hook into it. Of course, its kernel
is out of reach. But if you are interested e.g. in marking specific I/O
requests, you could write your own driver and hook into the stack.
>
>> to issue certain
>> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
>> ports), you can also trace things inside the guest that are otherwise
>> invisible to the host. I once hacked up an ad-hoc tracing by means of
>> hypercalls (required some kvm patching). That also worked from guest
>> userspace - and revealed that even more hypercalls could be called that
>> way (that's fixed in KVM now).
>>
>> >
>> >
>> > Bugs:
>> >
>> > 1. I hit a bug w/ instruction logging using a RAM based temp
>> folder. If
>> > I ran w/ the following command line:
>> > (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
>> >
>> > qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>>
>> -d only works in emulation mode as it relies on dynamic code translation
>> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
>> upstream QEMU, it's the other way around).
>>
>
> Hence why running w/out admin rights enables the logging. I was confused
> because the logfile is still created w/ the kvm module disabled.
It's created but remains empty for obvious reasons in KVM mode.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-15 23:39 ` Jan Kiszka
2009-12-16 4:07 ` Raindog
@ 2009-12-16 8:14 ` Vadim Rozenfeld
2009-12-16 9:38 ` Alexander Graf
2009-12-16 22:06 ` Raindog
2 siblings, 1 reply; 11+ messages in thread
From: Vadim Rozenfeld @ 2009-12-16 8:14 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Raindog, kvm
On Wed, 2009-12-16 at 00:39 +0100, Jan Kiszka wrote:
> Raindog wrote:
> > Hello,
> >
> > I am researching KVM as a malware analysis platform and had some
> > questions about debugging the guest OS. In my case I intend to use
> > windows guests. So my questsions are as follows:
> >
> > Questions:
> >
> > 1. What instrumentation facilities are their available?
> >
> > 2. Is it possible to extend the debugging interface so that debugging is
> > more transparent to the guest OS? IE: there is still a limit of 4 HW
> > breakpoints (which makes me wonder why a LIST is used for them...)
>
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
>
> >
> > 3. I'm not finding any published API for interfacing with KVM/KQEMU/QEMU
> > at a low level, for example, for writing custom tracers, etc. Is there
> > one? Or is there something similar?
>
> KVM provides tracepoints for the Linux ftrace framework, see related
> documentation of the kernel. If you extend your guest to issue certain
> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
> ports), you can also trace things inside the guest that are otherwise
> invisible to the host.
You can WRITE_PORT_BUFFER_UCHAR to com1/com2 port when you are in kernel
mode.
> I once hacked up an ad-hoc tracing by means of
> hypercalls (required some kvm patching). That also worked from guest
> userspace - and revealed that even more hypercalls could be called that
> way (that's fixed in KVM now).
>
> >
> >
> > Bugs:
> >
> > 1. I hit a bug w/ instruction logging using a RAM based temp folder. If
> > I ran w/ the following command line:
> > (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
> >
> > qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>
> -d only works in emulation mode as it relies on dynamic code translation
> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
> upstream QEMU, it's the other way around).
>
> >
> > It would successfully log to the tmp log file, but obviously, KVM would
> > be disabled.
> >
> > If I use sudo, it won't log to the file, is this a known issue?
> >
> > 2. -enable-nesting on AMD hardware using a xen guest OS causes xen to
> > GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen based
> > guests?
>
> If your host kernel or kvm-kmod is not 2.6.32 based, update first. A lot
> of nested SVM fixes went in recently. If it still fails, put Alex (Graf)
> and Joerg (Roedel) on CC.
>
> Jan
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-16 8:14 ` Vadim Rozenfeld
@ 2009-12-16 9:38 ` Alexander Graf
0 siblings, 0 replies; 11+ messages in thread
From: Alexander Graf @ 2009-12-16 9:38 UTC (permalink / raw)
To: Vadim Rozenfeld; +Cc: Jan Kiszka, Raindog, kvm@vger.kernel.org
Am 16.12.2009 um 09:14 schrieb Vadim Rozenfeld <vrozenfe@redhat.com>:
> On Wed, 2009-12-16 at 00:39 +0100, Jan Kiszka wrote:
>> Raindog wrote:
>>> Hello,
>>>
>>> I am researching KVM as a malware analysis platform and had some
>>> questions about debugging the guest OS. In my case I intend to use
>>> windows guests. So my questsions are as follows:
>>>
>>> Questions:
>>>
>>> 1. What instrumentation facilities are their available?
>>>
>>> 2. Is it possible to extend the debugging interface so that
>>> debugging is
>>> more transparent to the guest OS? IE: there is still a limit of 4 HW
>>> breakpoints (which makes me wonder why a LIST is used for them...)
>>
>> In accelerated KVM mode, the x86 architecture restricts us to 4
>> break-
>> or watchpoints that can be active at the same time. If you switch to
>> emulation mode, there are no such limits. Actually, I just made use
>> of
>> this for debugging a subtle stack corruption in a guest, and I had
>> more
>> than 70 watchpoints active at the same time. It's just "slightly"
>> slower
>> than KVM...
>>
>>>
>>> 3. I'm not finding any published API for interfacing with KVM/
>>> KQEMU/QEMU
>>> at a low level, for example, for writing custom tracers, etc. Is
>>> there
>>> one? Or is there something similar?
>>
>> KVM provides tracepoints for the Linux ftrace framework, see related
>> documentation of the kernel. If you extend your guest to issue
>> certain
>> events that the hypervisor sees and traces (e.g. writes to pseudo I/O
>> ports), you can also trace things inside the guest that are otherwise
>> invisible to the host.
> You can WRITE_PORT_BUFFER_UCHAR to com1/com2 port when you are in
> kernel
> mode.
>> I once hacked up an ad-hoc tracing by means of
>> hypercalls (required some kvm patching). That also worked from guest
>> userspace - and revealed that even more hypercalls could be called
>> that
>> way (that's fixed in KVM now).
>>
>>>
>>>
>>> Bugs:
>>>
>>> 1. I hit a bug w/ instruction logging using a RAM based temp
>>> folder. If
>>> I ran w/ the following command line:
>>> (Version info: QEMU PC emulator version 0.10.50 (qemu-kvm-devel-88))
>>>
>>> qemu-system-x86_64 -hda debian.img -enable-nesting -d in_asm
>>
>> -d only works in emulation mode as it relies on dynamic code
>> translation
>> (TCG). For qemu-kvm, you need to switch to emulation via -no-kvm (for
>> upstream QEMU, it's the other way around).
>>
>>>
>>> It would successfully log to the tmp log file, but obviously, KVM
>>> would
>>> be disabled.
>>>
>>> If I use sudo, it won't log to the file, is this a known issue?
>>>
>>> 2. -enable-nesting on AMD hardware using a xen guest OS causes xen
>>> to
>>> GPF somewhere in svm_cpu_up. Is nesting supposed to work w/ Xen
>>> based
>>> guests?
>>
>> If your host kernel or kvm-kmod is not 2.6.32 based, update first.
>> A lot
>> of nested SVM fixes went in recently. If it still fails, put Alex
>> (Graf)
>> and Joerg (Roedel) on CC.
Also make sure you pass nested=1 to kvm-amd.ko.
Xen definitely worked for me, so you're probably just missing one of
the many magic bits :-).
Alex
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-15 23:39 ` Jan Kiszka
2009-12-16 4:07 ` Raindog
2009-12-16 8:14 ` Vadim Rozenfeld
@ 2009-12-16 22:06 ` Raindog
2009-12-17 5:36 ` Avi Kivity
2 siblings, 1 reply; 11+ messages in thread
From: Raindog @ 2009-12-16 22:06 UTC (permalink / raw)
To: Jan Kiszka; +Cc: kvm
On 12/15/2009 3:39 PM, Jan Kiszka wrote:
> Raindog wrote:
> > Hello,
> >
> > I am researching KVM as a malware analysis platform and had some
> > questions about debugging the guest OS. In my case I intend to use
> > windows guests. So my questsions are as follows:
> >
> > Questions:
> >
> > 1. What instrumentation facilities are their available?
> >
> > 2. Is it possible to extend the debugging interface so that debugging is
> > more transparent to the guest OS? IE: there is still a limit of 4 HW
> > breakpoints (which makes me wonder why a LIST is used for them...)
>
> In accelerated KVM mode, the x86 architecture restricts us to 4 break-
> or watchpoints that can be active at the same time. If you switch to
> emulation mode, there are no such limits. Actually, I just made use of
> this for debugging a subtle stack corruption in a guest, and I had more
> than 70 watchpoints active at the same time. It's just "slightly" slower
> than KVM...
>
Are there any advantages over stock qemu if using kvm w/out the kernel
module?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-16 22:06 ` Raindog
@ 2009-12-17 5:36 ` Avi Kivity
2009-12-17 7:05 ` Raindog
0 siblings, 1 reply; 11+ messages in thread
From: Avi Kivity @ 2009-12-17 5:36 UTC (permalink / raw)
To: Raindog; +Cc: Jan Kiszka, kvm
On 12/17/2009 12:06 AM, Raindog wrote:
>
> Are there any advantages over stock qemu if using kvm w/out the kernel
> module?
No. qemu-kvm is not tested without kvm, so there may be disadvantages.
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-17 5:36 ` Avi Kivity
@ 2009-12-17 7:05 ` Raindog
2009-12-17 8:37 ` Avi Kivity
0 siblings, 1 reply; 11+ messages in thread
From: Raindog @ 2009-12-17 7:05 UTC (permalink / raw)
To: Avi Kivity; +Cc: Jan Kiszka, kvm
On 12/16/2009 9:36 PM, Avi Kivity wrote:
> On 12/17/2009 12:06 AM, Raindog wrote:
>>
>> Are there any advantages over stock qemu if using kvm w/out the
>> kernel module?
>
> No. qemu-kvm is not tested without kvm, so there may be disadvantages.
>
Does that then imply that svm emulation (--enable-nesting) is not well
tested either?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: debugging windows guests
2009-12-17 7:05 ` Raindog
@ 2009-12-17 8:37 ` Avi Kivity
0 siblings, 0 replies; 11+ messages in thread
From: Avi Kivity @ 2009-12-17 8:37 UTC (permalink / raw)
To: Raindog; +Cc: Jan Kiszka, kvm
On 12/17/2009 09:05 AM, Raindog wrote:
> On 12/16/2009 9:36 PM, Avi Kivity wrote:
>> On 12/17/2009 12:06 AM, Raindog wrote:
>>>
>>> Are there any advantages over stock qemu if using kvm w/out the
>>> kernel module?
>>
>> No. qemu-kvm is not tested without kvm, so there may be disadvantages.
>>
>
> Does that then imply that svm emulation (--enable-nesting) is not well
> tested either?
It's an unrelated feature (but I wouldn't say it is heavily tested).
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 11+ messages in thread