[Qemu-devel] software breakpoints disappearing/reappearing in KVM/qemu

[Qemu-devel] software breakpoints disappearing/reappearing in KVM/qemu

[Craig Brozefsky]

First, thank you very much for your work on KVM/QEMU and in particular
the GDB support.  I have been a heavy user of it while implementing a
programmable debugger that monitors/traces guest operating systems.  I
need more breakpoints than hardware BPs available, and I need the
performance of KVM acceleration.

I am running into a problem with software breakpoints disappearing and
re-appearing.  I set them via the GDB stub.  If I examine the location
in the qemu monitor I will see them there, but on occasion (and with
greater frequency the more breakpoints I have) a given breakpoint will
disappear according to the monitor, and then re-appear.

I also have processes executing the instructions at the breakpoint without
stopping.  This makes my results quite unpredictable, and it has led too a
few days of learning all about Windows system working set paging.

My first approach was to try and re-install the breakpoints after a
page was faulted in, thinking that the INT3 was disappearing when the
page was read back into memory.  Setting a hbreak, I was trapping
MMAccessFault, and reinstalling all by breakpoints when it hit its
return address (after the IO Page Read Calls).  That didn't solve
the problem, and left me wondering if I need to take into consideration the way
KVM/qemu handle guest memory.  And that brings me to my question.

Assuming I handle the guest kernel paging issue properly, are you
aware of issues with the debugging support in KVM/qemu that would
result in the INT3 blinking in and out of existence?

In the meantime, I've been reading source code, but it'll be a few
days before I figure out the details of memory mgmt under KVM.  Any
hints would be greatly appreciated.

Thank you for your time.

Re: [Qemu-devel] software breakpoints disappearing/reappearing in KVM/qemu

[Blue Swirl]

On Fri, Apr 8, 2011 at 9:50 PM, Craig Brozefsky <craig@red-bean.com> wrote:
> First, thank you very much for your work on KVM/QEMU and in particular
> the GDB support.  I have been a heavy user of it while implementing a
> programmable debugger that monitors/traces guest operating systems.  I
> need more breakpoints than hardware BPs available, and I need the
> performance of KVM acceleration.
>
> I am running into a problem with software breakpoints disappearing and
> re-appearing.  I set them via the GDB stub.  If I examine the location
> in the qemu monitor I will see them there, but on occasion (and with
> greater frequency the more breakpoints I have) a given breakpoint will
> disappear according to the monitor, and then re-appear.
>
> I also have processes executing the instructions at the breakpoint without
> stopping.  This makes my results quite unpredictable, and it has led too a
> few days of learning all about Windows system working set paging.
>
> My first approach was to try and re-install the breakpoints after a
> page was faulted in, thinking that the INT3 was disappearing when the
> page was read back into memory.  Setting a hbreak, I was trapping
> MMAccessFault, and reinstalling all by breakpoints when it hit its
> return address (after the IO Page Read Calls).  That didn't solve
> the problem, and left me wondering if I need to take into consideration the way
> KVM/qemu handle guest memory.  And that brings me to my question.
>
> Assuming I handle the guest kernel paging issue properly, are you
> aware of issues with the debugging support in KVM/qemu that would
> result in the INT3 blinking in and out of existence?
>
> In the meantime, I've been reading source code, but it'll be a few
> days before I figure out the details of memory mgmt under KVM.  Any
> hints would be greatly appreciated.

When not using KVM but TCG, guest instructions are translated to host
instructions and these are stored to translation blocks. QEMU should
detect writes to the source pages of translation and flushes the TBs.
This is needed to support self-modifying code.

For KVM, native self-modifying code support of the CPU should be
enough. Are you sure you are using KVM, what does "info kvm" tell?

For TCG, I'd think what could happen here is that a block is
translated with (or without) INT3, but somehow the write access to
remove (or add) INT3 is undetected and the old TB is used.

How are you setting the breakpoint, with GDB interface?

Re: [Qemu-devel] software breakpoints disappearing/reappearing in KVM/qemu

[Craig Brozefsky]

On Fri, Apr 8, 2011 at 3:16 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
> On Fri, Apr 8, 2011 at 9:50 PM, Craig Brozefsky <craig@red-bean.com> wrote:
>
> For KVM, native self-modifying code support of the CPU should be
> enough. Are you sure you are using KVM, what does "info kvm" tell?

Yup.

> How are you setting the breakpoint, with GDB interface?

Correct.

[Qemu-devel] Re: software breakpoints disappearing/reappearing in KVM/qemu

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3025 bytes --]

On 2011-04-08 20:50, Craig Brozefsky wrote:
> First, thank you very much for your work on KVM/QEMU and in particular
> the GDB support.  I have been a heavy user of it while implementing a
> programmable debugger that monitors/traces guest operating systems.  I
> need more breakpoints than hardware BPs available, and I need the
> performance of KVM acceleration.
> 
> I am running into a problem with software breakpoints disappearing and
> re-appearing.  I set them via the GDB stub.  If I examine the location
> in the qemu monitor I will see them there, but on occasion (and with
> greater frequency the more breakpoints I have) a given breakpoint will
> disappear according to the monitor, and then re-appear.
> 
> I also have processes executing the instructions at the breakpoint without
> stopping.  This makes my results quite unpredictable, and it has led too a
> few days of learning all about Windows system working set paging.

KVM injects software breakpoints into the guest pages - and that
intransparently. If the guest starts to read those breakpoints or even
manipulate the code pages, all kinds of weird things can happen.

This particularly includes swapping. QEMU does not track if a guest
reads and then replaces the page. There is some basic check when
removing a software breakpoint that triggers if the page was manipulated
in the meantime. But when that check triggers, it's too late: the guest
likely already saved the patched page elsewhere and can stumble and fall
the next time it restores it to some other linear address and then
executes it. BTW, that's also not what gdb is designed to handle.

> 
> My first approach was to try and re-install the breakpoints after a
> page was faulted in, thinking that the INT3 was disappearing when the
> page was read back into memory.  Setting a hbreak, I was trapping
> MMAccessFault, and reinstalling all by breakpoints when it hit its
> return address (after the IO Page Read Calls).  That didn't solve
> the problem, and left me wondering if I need to take into consideration the way
> KVM/qemu handle guest memory.  And that brings me to my question.

This kind of swapping is guest business, specifically with EPT/NPT.

> 
> Assuming I handle the guest kernel paging issue properly, are you
> aware of issues with the debugging support in KVM/qemu that would
> result in the INT3 blinking in and out of existence?
> 
> In the meantime, I've been reading source code, but it'll be a few
> days before I figure out the details of memory mgmt under KVM.  Any
> hints would be greatly appreciated.

QEMU's gdbstub in KVM mode is simply not designed to account for guests
swapping out code pages that contain breakpoints. Due to the fact that
the Linux kernel does not do these weird things to its own code, at
least I never seriously thought about potential workarounds. As your
approach indicates, those might become fairly guest-specific - if they
can be implemented reliably at all.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: software breakpoints disappearing/reappearing in KVM/qemu

[Avi Kivity]

On 04/08/2011 11:52 PM, Jan Kiszka wrote:
> QEMU's gdbstub in KVM mode is simply not designed to account for guests
> swapping out code pages that contain breakpoints. Due to the fact that
> the Linux kernel does not do these weird things to its own code,

It actually happily patches its own code at certain circumstances.  If 
you debug early boot or suspend/resume or cpu hotplug you're likely to 
encounter them.

>   at
> least I never seriously thought about potential workarounds. As your
> approach indicates, those might become fairly guest-specific - if they
> can be implemented reliably at all.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: software breakpoints disappearing/reappearing in KVM/qemu

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On 2011-04-10 16:01, Avi Kivity wrote:
> On 04/08/2011 11:52 PM, Jan Kiszka wrote:
>> QEMU's gdbstub in KVM mode is simply not designed to account for guests
>> swapping out code pages that contain breakpoints. Due to the fact that
>> the Linux kernel does not do these weird things to its own code,
> 
> It actually happily patches its own code at certain circumstances.  If
> you debug early boot or suspend/resume or cpu hotplug you're likely to
> encounter them.

Yes, it patches (alternates, jump_labels, mcount etc.), but it doesn't
swap or move code pages. At least so far.

If you need to break at such locations, there are still hardware
breakpoints, though only 4.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: software breakpoints disappearing/reappearing in KVM/qemu

[Avi Kivity]

On 04/10/2011 05:23 PM, Jan Kiszka wrote:
> On 2011-04-10 16:01, Avi Kivity wrote:
> >  On 04/08/2011 11:52 PM, Jan Kiszka wrote:
> >>  QEMU's gdbstub in KVM mode is simply not designed to account for guests
> >>  swapping out code pages that contain breakpoints. Due to the fact that
> >>  the Linux kernel does not do these weird things to its own code,
> >
> >  It actually happily patches its own code at certain circumstances.  If
> >  you debug early boot or suspend/resume or cpu hotplug you're likely to
> >  encounter them.
>
> Yes, it patches (alternates, jump_labels, mcount etc.), but it doesn't
> swap or move code pages. At least so far.
>

Try using modules (though that's different - the virtual addresses will 
change too).  Or debugging a mixed kernel/user workload.

> If you need to break at such locations, there are still hardware
> breakpoints, though only 4.

Right.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: software breakpoints disappearing/reappearing in KVM/qemu

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On 2011-04-10 16:41, Avi Kivity wrote:
> On 04/10/2011 05:23 PM, Jan Kiszka wrote:
>> On 2011-04-10 16:01, Avi Kivity wrote:
>> >  On 04/08/2011 11:52 PM, Jan Kiszka wrote:
>> >>  QEMU's gdbstub in KVM mode is simply not designed to account for
>> guests
>> >>  swapping out code pages that contain breakpoints. Due to the fact
>> that
>> >>  the Linux kernel does not do these weird things to its own code,
>> >
>> >  It actually happily patches its own code at certain circumstances.  If
>> >  you debug early boot or suspend/resume or cpu hotplug you're likely to
>> >  encounter them.
>>
>> Yes, it patches (alternates, jump_labels, mcount etc.), but it doesn't
>> swap or move code pages. At least so far.
>>
> 
> Try using modules (though that's different - the virtual addresses will
> change too).  Or debugging a mixed kernel/user workload.

Modules don't go unexpectedly. Once they are loaded, they can perfectly
be debugged just like kernel code. Debugging init code requires care to
avoid leaving breakpoints behind in to-be-removed sections.

But that's not comparable to swapping. For that reason, debugging user
code via the gdbstub was always discouraged (and it's unneeded in 99% of
the time as such targets come with their own debugger).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]