[Qemu-devel] Guest VM debug (Int 3 panic)

[Qemu-devel] Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

Hi All,
I am trying to debug guest OS through qemu with kvm enabled.
Following is what I have done:
1: fire the qemu-kvm
<snip>
sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c -s
</snip>

2: wait until login into guest OS (ubuntu 10.04)

3: fire gdb
<snip>
gdb vmlinux
target remote :1234
b do_fork
set arch i386:x86-64
c
</snip>

4: after I typed "ls" in guest OS. The guest OS paniced with some message
related to "int 3 blah blah". Then crashed.

Someone said we should use hardware breakpoint when kvm is enabled, or
"monitor system_reset" after set the breakpoint, but it didn't work for me.
The hardware breakpoint could not been hit anyway.

I have tried with "-no-kvm", it works normally with breakpoints. But I want
to debug the guest OS with kvm enabled. I don't know whether someone has
met this similar situation.

Thanks for your time!

Best Wishes,
Yaohui

[-- Attachment #2: Type: text/html, Size: 1470 bytes --]

[Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

Hi All,
I am trying to debug guest OS through qemu with kvm enabled.
Following is what I have done:
1: fire the qemu-kvm
<snip>
sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c -s
</snip>

2: wait until login into guest OS (ubuntu 10.04)

3: fire gdb
<snip>
gdb vmlinux
target remote :1234
b do_fork
set arch i386:x86-64
c
</snip>

4: after I typed "ls" in guest OS. The guest OS paniced with some message
related to "int 3 blah blah". Then crashed.

Someone said we should use hardware breakpoint when kvm is enabled, or
"monitor system_reset" after set the breakpoint, but it didn't work for me.
The hardware breakpoint could not been hit anyway.

I have tried with "-no-kvm", it works normally with breakpoints. But I want
to debug the guest OS with kvm enabled. I don't know whether someone has
met this similar situation.

Thanks for your time!

Best Wishes,
Yaohui

[-- Attachment #2: Type: text/html, Size: 1376 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

On 2013-09-25 20:08, Hu Yaohui wrote:
> Hi All,
> I am trying to debug guest OS through qemu with kvm enabled.
> Following is what I have done:
> 1: fire the qemu-kvm
> <snip>
> sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c -s
> </snip>
> 
> 2: wait until login into guest OS (ubuntu 10.04)
> 
> 3: fire gdb
> <snip>
> gdb vmlinux
> target remote :1234
> b do_fork
> set arch i386:x86-64

"set arch" is unneeded. vmlinux already tells gdb that you are debugging
x86-64.

> c
> </snip>
> 
> 4: after I typed "ls" in guest OS. The guest OS paniced with some message
> related to "int 3 blah blah". Then crashed.
> 
> Someone said we should use hardware breakpoint when kvm is enabled, or

You can use hardware breakpoints as well but it is not required unless
the target code can be overwritten (e.g. due to a reset).

> "monitor system_reset" after set the breakpoint, but it didn't work for me.
> The hardware breakpoint could not been hit anyway.
> 
> I have tried with "-no-kvm", it works normally with breakpoints. But I want
> to debug the guest OS with kvm enabled. I don't know whether someone has
> met this similar situation.

You didn't tell us which version of QEMU (or is it old qemu-kvm?) you
are using, what host kernel and which CPU type (AMD vs. Intel). Did you
try a recent version of all of them already? I'm currently not aware of
gdb problems with QEMU/KVM, I'm rather using it on an almost daily basis
(typically git head versions).

If you want to debug your issue: there is ftrace to record what KVM
events happen, and you can switch gdb into verbose mode as well,
comparing the communication between KVM on/off: set debug remote 1.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 2370 bytes --]

Hi Jan,
Thanks for your reply.
On Thu, Sep 26, 2013 at 2:08 AM, Jan Kiszka <jan.kiszka@web.de> wrote:

> On 2013-09-25 20:08, Hu Yaohui wrote:
> > Hi All,
> > I am trying to debug guest OS through qemu with kvm enabled.
> > Following is what I have done:
> > 1: fire the qemu-kvm
> > <snip>
> > sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c -s
> > </snip>
> >
> > 2: wait until login into guest OS (ubuntu 10.04)
> >
> > 3: fire gdb
> > <snip>
> > gdb vmlinux
> > target remote :1234
> > b do_fork
> > set arch i386:x86-64
>
> "set arch" is unneeded. vmlinux already tells gdb that you are debugging
> x86-64.
>
> > c
> > </snip>
> >
> > 4: after I typed "ls" in guest OS. The guest OS paniced with some message
> > related to "int 3 blah blah". Then crashed.
> >
> > Someone said we should use hardware breakpoint when kvm is enabled, or
>
> You can use hardware breakpoints as well but it is not required unless
> the target code can be overwritten (e.g. due to a reset).
>
> > "monitor system_reset" after set the breakpoint, but it didn't work for
> me.
> > The hardware breakpoint could not been hit anyway.
> >
> > I have tried with "-no-kvm", it works normally with breakpoints. But I
> want
> > to debug the guest OS with kvm enabled. I don't know whether someone has
> > met this similar situation.
>
> You didn't tell us which version of QEMU (or is it old qemu-kvm?) you
> are using, what host kernel and which CPU type (AMD vs. Intel). Did you
> try a recent version of all of them already? I'm currently not aware of
> gdb problems with QEMU/KVM, I'm rather using it on an almost daily basis
> (typically git head versions).
>
I am using a nested VM. My CPU type is intel.
On L0, the QEMU-KVM version is 1.0, host kernel version: 2.6.32.10,
kvm-kmod version: 3.2
On L1, the QEMU-KVM version is 1.2, kernel version: 3.2.2, kvm-kmod
version: 3.2
On L2, guest kernel version: 2.6.32.10
I am trying to debug L2 guest kernel on L1 QEMU. It gives me "INT 3"
related kernel oops.
I also have tried to debug the L1 guest kernel through L0 QEMU which works
fine.

>
> If you want to debug your issue: there is ftrace to record what KVM
> events happen, and you can switch gdb into verbose mode as well,
> comparing the communication between KVM on/off: set debug remote 1.
>
> Thanks for your suggestion! I will give that a try.

> Jan
>
>
>

[-- Attachment #2: Type: text/html, Size: 3490 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3000 bytes --]

On 2013-09-26 16:14, Hu Yaohui wrote:
> Hi Jan,
> Thanks for your reply.
> On Thu, Sep 26, 2013 at 2:08 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> 
>> On 2013-09-25 20:08, Hu Yaohui wrote:
>>> Hi All,
>>> I am trying to debug guest OS through qemu with kvm enabled.
>>> Following is what I have done:
>>> 1: fire the qemu-kvm
>>> <snip>
>>> sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c -s
>>> </snip>
>>>
>>> 2: wait until login into guest OS (ubuntu 10.04)
>>>
>>> 3: fire gdb
>>> <snip>
>>> gdb vmlinux
>>> target remote :1234
>>> b do_fork
>>> set arch i386:x86-64
>>
>> "set arch" is unneeded. vmlinux already tells gdb that you are debugging
>> x86-64.
>>
>>> c
>>> </snip>
>>>
>>> 4: after I typed "ls" in guest OS. The guest OS paniced with some message
>>> related to "int 3 blah blah". Then crashed.
>>>
>>> Someone said we should use hardware breakpoint when kvm is enabled, or
>>
>> You can use hardware breakpoints as well but it is not required unless
>> the target code can be overwritten (e.g. due to a reset).
>>
>>> "monitor system_reset" after set the breakpoint, but it didn't work for
>> me.
>>> The hardware breakpoint could not been hit anyway.
>>>
>>> I have tried with "-no-kvm", it works normally with breakpoints. But I
>> want
>>> to debug the guest OS with kvm enabled. I don't know whether someone has
>>> met this similar situation.
>>
>> You didn't tell us which version of QEMU (or is it old qemu-kvm?) you
>> are using, what host kernel and which CPU type (AMD vs. Intel). Did you
>> try a recent version of all of them already? I'm currently not aware of
>> gdb problems with QEMU/KVM, I'm rather using it on an almost daily basis
>> (typically git head versions).
>>
> I am using a nested VM.

Oh, "minor" detail ;) - why nested? But this used to work for me with a
patched 3.9+ kernel some while ago.

> My CPU type is intel.
> On L0, the QEMU-KVM version is 1.0, host kernel version: 2.6.32.10,
> kvm-kmod version: 3.2

Try at least the latest kvm-kmod version, but there are even more fixes
in kvm.git. Not sure if any of them has direct impact on your scenario,
but it's generally better to use a recent kernel with this still
experimental feature (VMX nesting).

As this is likely a KVM issue, I'm also CC'ing the corresponding list

Jan

> On L1, the QEMU-KVM version is 1.2, kernel version: 3.2.2, kvm-kmod
> version: 3.2
> On L2, guest kernel version: 2.6.32.10
> I am trying to debug L2 guest kernel on L1 QEMU. It gives me "INT 3"
> related kernel oops.
> I also have tried to debug the L1 guest kernel through L0 QEMU which works
> fine.
> 
>>
>> If you want to debug your issue: there is ftrace to record what KVM
>> events happen, and you can switch gdb into verbose mode as well,
>> comparing the communication between KVM on/off: set debug remote 1.
>>
>> Thanks for your suggestion! I will give that a try.
> 
>> Jan
>>
>>
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 3537 bytes --]

Hi Jan,
I am working on some Nested VM related projects. Some other teammates have
made the modifications to the kvm module. Most of my work depends on his.
If I could not use Qemu Debug method. Could you please suggest some other
debugging methods to debug the L2 guest OS(printk, hijack kernel function,
or something else)?

Thanks for your time!

Best Wishes,
Yaohui Hu


On Thu, Sep 26, 2013 at 1:26 PM, Jan Kiszka <jan.kiszka@web.de> wrote:

> On 2013-09-26 16:14, Hu Yaohui wrote:
> > Hi Jan,
> > Thanks for your reply.
> > On Thu, Sep 26, 2013 at 2:08 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> >
> >> On 2013-09-25 20:08, Hu Yaohui wrote:
> >>> Hi All,
> >>> I am trying to debug guest OS through qemu with kvm enabled.
> >>> Following is what I have done:
> >>> 1: fire the qemu-kvm
> >>> <snip>
> >>> sudo qemu-system-x86_64 -hda vdisk.img -m 4096 -smp 2 -vnc :2 -boot c
> -s
> >>> </snip>
> >>>
> >>> 2: wait until login into guest OS (ubuntu 10.04)
> >>>
> >>> 3: fire gdb
> >>> <snip>
> >>> gdb vmlinux
> >>> target remote :1234
> >>> b do_fork
> >>> set arch i386:x86-64
> >>
> >> "set arch" is unneeded. vmlinux already tells gdb that you are debugging
> >> x86-64.
> >>
> >>> c
> >>> </snip>
> >>>
> >>> 4: after I typed "ls" in guest OS. The guest OS paniced with some
> message
> >>> related to "int 3 blah blah". Then crashed.
> >>>
> >>> Someone said we should use hardware breakpoint when kvm is enabled, or
> >>
> >> You can use hardware breakpoints as well but it is not required unless
> >> the target code can be overwritten (e.g. due to a reset).
> >>
> >>> "monitor system_reset" after set the breakpoint, but it didn't work for
> >> me.
> >>> The hardware breakpoint could not been hit anyway.
> >>>
> >>> I have tried with "-no-kvm", it works normally with breakpoints. But I
> >> want
> >>> to debug the guest OS with kvm enabled. I don't know whether someone
> has
> >>> met this similar situation.
> >>
> >> You didn't tell us which version of QEMU (or is it old qemu-kvm?) you
> >> are using, what host kernel and which CPU type (AMD vs. Intel). Did you
> >> try a recent version of all of them already? I'm currently not aware of
> >> gdb problems with QEMU/KVM, I'm rather using it on an almost daily basis
> >> (typically git head versions).
> >>
> > I am using a nested VM.
>
> Oh, "minor" detail ;) - why nested? But this used to work for me with a
> patched 3.9+ kernel some while ago.
>
> > My CPU type is intel.
> > On L0, the QEMU-KVM version is 1.0, host kernel version: 2.6.32.10,
> > kvm-kmod version: 3.2
>
> Try at least the latest kvm-kmod version, but there are even more fixes
> in kvm.git. Not sure if any of them has direct impact on your scenario,
> but it's generally better to use a recent kernel with this still
> experimental feature (VMX nesting).
>
> As this is likely a KVM issue, I'm also CC'ing the corresponding list
>
> Jan
>
> > On L1, the QEMU-KVM version is 1.2, kernel version: 3.2.2, kvm-kmod
> > version: 3.2
> > On L2, guest kernel version: 2.6.32.10
> > I am trying to debug L2 guest kernel on L1 QEMU. It gives me "INT 3"
> > related kernel oops.
> > I also have tried to debug the L1 guest kernel through L0 QEMU which
> works
> > fine.
> >
> >>
> >> If you want to debug your issue: there is ftrace to record what KVM
> >> events happen, and you can switch gdb into verbose mode as well,
> >> comparing the communication between KVM on/off: set debug remote 1.
> >>
> >> Thanks for your suggestion! I will give that a try.
> >
> >> Jan
> >>
> >>
> >>
> >
>
>
>

[-- Attachment #2: Type: text/html, Size: 5038 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

On 2013-09-26 20:53, Hu Yaohui wrote:
> Hi Jan,
> I am working on some Nested VM related projects. Some other teammates have
> made the modifications to the kvm module. 

And these modifications cannot cause the misguided INT3?

> Most of my work depends on his.
> If I could not use Qemu Debug method. Could you please suggest some other
> debugging methods to debug the L2 guest OS(printk, hijack kernel function,
> or something else)?

Remove L0 while debugging L2 and, once it works, move L1/L2 back over
L0? Your setup seems to be pretty special with (for us) unknown
requirements, so it's hard to suggest what to do best.

In any case, you seem to be pretty much "off-track" and may either have
to stabilize the whole stack on your own, possibly back-porting
essential nVMX fixes from latest versions, or rebase & share your
changes so that we can help again.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

On Thu, Sep 26, 2013 at 3:07 PM, Jan Kiszka <jan.kiszka@web.de> wrote:

> On 2013-09-26 20:53, Hu Yaohui wrote:
> > Hi Jan,
> > I am working on some Nested VM related projects. Some other teammates
> have
> > made the modifications to the kvm module.
>
> And these modifications cannot cause the misguided INT3?
>
No

>
> > Most of my work depends on his.
> > If I could not use Qemu Debug method. Could you please suggest some other
> > debugging methods to debug the L2 guest OS(printk, hijack kernel
> function,
> > or something else)?
>
> Remove L0 while debugging L2 and, once it works, move L1/L2 back over
> L0? Your setup seems to be pretty special with (for us) unknown
> requirements, so it's hard to suggest what to do best.
>
> I will try that.

> In any case, you seem to be pretty much "off-track" and may either have
> to stabilize the whole stack on your own, possibly back-porting
> essential nVMX fixes from latest versions, or rebase & share your
> changes so that we can help again.
>
> Thank you!

> Jan
>
>
>

[-- Attachment #2: Type: text/html, Size: 2010 bytes --]

Re: [Qemu-devel] Fwd: Guest VM debug (Int 3 panic)

[Hu Yaohui]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

On Thu, Sep 26, 2013 at 3:20 PM, Hu Yaohui <loki2441@gmail.com> wrote:

>
>
> On Thu, Sep 26, 2013 at 3:07 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>
>> On 2013-09-26 20:53, Hu Yaohui wrote:
>> > Hi Jan,
>> > I am working on some Nested VM related projects. Some other teammates
>> have
>> > made the modifications to the kvm module.
>>
>> And these modifications cannot cause the misguided INT3?
>>
> No
>
>>
>> > Most of my work depends on his.
>> > If I could not use Qemu Debug method. Could you please suggest some
>> other
>> > debugging methods to debug the L2 guest OS(printk, hijack kernel
>> function,
>> > or something else)?
>>
>> Remove L0 while debugging L2 and, once it works, move L1/L2 back over
>> L0? Your setup seems to be pretty special with (for us) unknown
>> requirements, so it's hard to suggest what to do best.
>>
>> I will try that.
>
When I remove L0 while debugging L2, it works, but When I move it back, it
failed with the same error message in L2 guest. Thank you Jan! I will
figure out some other methods to overcome this hurdle.

>
>
>> In any case, you seem to be pretty much "off-track" and may either have
>> to stabilize the whole stack on your own, possibly back-porting
>> essential nVMX fixes from latest versions, or rebase & share your
>> changes so that we can help again.
>>
>> Thank you!
>
>> Jan
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 2803 bytes --]