[Qemu-devel] Debugging with paging enabled

[Qemu-devel] Debugging with paging enabled

[Marcel Kilgus]

Hi,

I'm using QEMU 0.8.2 to debug a new experimental OS (x86) I'm
currently writing for my university. So this is not based on Linux and
the "-kernel" option, instead I use a floppy image and load my kernel
using GRUB.

Now debugging was a breeze so far, but while implementing virtual
memory I'm hitting a huge problem. The kernel is loaded by GRUB at
physical address 0x100000 and, IIRC like Linux, I simply remap that to
be mirrored at 0xc0000000 using the page table. Afterwards I define my
kernel code and data segments to be based there as well, so that all
offsets in the code are still right. This works fine.

But after these initialisations I have problems with breakpoints:

    (gdb) b timer_handler
    Breakpoint 1 at 0x108294: file cpu_x86.c, line 29.
    (gdb) c

now never returns, even though the code does run. If I do

    (gdb) b *0xc0108294
    Breakpoint 1 at 0xc0108294
    (gdb) c
    Continuing.

    Program received signal SIGTRAP, Trace/breakpoint trap.
    timer_handler at cpu_x86.c:71

I get the exception. Trying to continue with step or next results in a
complete GDB crash. It only works properly if I set both breakpoints:

    (gdb) b timer_handler
    Breakpoint 1 at 0x108294: file cpu_x86.c, line 29.
    (gdb) b *0xc0108294
    Breakpoint 2 at 0xc0108294
    (gdb) c
    Continuing.

    Breakpoint 1, timer_handler at cpu_x86.c:71

Having to always set them both in the high and low area is a bit
cumbersome to say the least. Any suggestion on what I'm missing or can
do to get breakpoints working properly again?

Many thanks in advance, Marcel

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

/me wrote:
> Having to always set them both in the high and low area is a bit
> cumbersome to say the least. Any suggestion on what I'm missing or can
> do to get breakpoints working properly again?

Okay, the QEMU source code was a lot more readable than I expected it
to be, so I had a look myself. For the record, my solution to my
problem is thus:

/tmptmp/qemu-0.8.2/target-i386 $ diff translate.bak translate.c
6425c6425
<                 if (env->breakpoints[j] == pc_ptr) {
---
>                 if (env->breakpoints[j] == pc_ptr - dc->cs_base) {

Now everything works as expected. Don't know if this breaks any other
scenarios, though.

All the best, Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Fabrice Bellard]

Marcel Kilgus wrote:
> /me wrote:
> 
>>Having to always set them both in the high and low area is a bit
>>cumbersome to say the least. Any suggestion on what I'm missing or can
>>do to get breakpoints working properly again?
> 
> 
> Okay, the QEMU source code was a lot more readable than I expected it
> to be, so I had a look myself. For the record, my solution to my
> problem is thus:
> 
> /tmptmp/qemu-0.8.2/target-i386 $ diff translate.bak translate.c
> 6425c6425
> <                 if (env->breakpoints[j] == pc_ptr) {
> ---
> 
>>                if (env->breakpoints[j] == pc_ptr - dc->cs_base) {
> 
> 
> Now everything works as expected. Don't know if this breaks any other
> scenarios, though.

Hi,

The breakpoints are set for a given virtual address. So IMHO testing 
only the EIP value instead of EIP + CS.base is not logical...

Regards,

Fabrice.

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

Fabrice Bellard wrote:
> The breakpoints are set for a given virtual address. So IMHO testing
> only the EIP value instead of EIP + CS.base is not logical...

That I just check for EIP is a convenience for me. The code is linked
to 0x00100000 but moves itself to the virtual address 0xC0100000 (CS
starts at 0xC0000000 so code offsets still fit!).
All debugging symbols however still point to 0x00100000. Only looking
at EIP regardless of CS makes the symbols fit for me again.

Leaving that aside, if I do set the breakpoint correctly at virtual
address (e.g.) 0xC0123456 qemu will (correctly I guess) cause an
exception for code offset 0x123456 (as CS base is 0xC0000000). GDB
however then doesn't recognize its own breakpoint as it only remembers
having set one at 0xC0123456, and apparently doesn't translate the
given exception address of CS:0x123456 back to a virtual one.

So all in all GDB just doesn't seem to cope very well with segmented
memory. At least that's my theory. If I didn't already have a huge
headache today thinking about the mix of physical, virtual, linear and
segment:offset addresses would surely give me one.

Cheers, Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Daniel Jacobowitz]

On Thu, Nov 09, 2006 at 12:33:05AM +0100, Marcel Kilgus wrote:
> Leaving that aside, if I do set the breakpoint correctly at virtual
> address (e.g.) 0xC0123456 qemu will (correctly I guess) cause an
> exception for code offset 0x123456 (as CS base is 0xC0000000). GDB
> however then doesn't recognize its own breakpoint as it only remembers
> having set one at 0xC0123456, and apparently doesn't translate the
> given exception address of CS:0x123456 back to a virtual one.

Which segment is it running from at this point?  Qemu may be reporting
the wrong address.

> So all in all GDB just doesn't seem to cope very well with segmented
> memory.

Correct.  It doesn't know anything at all about i386 segmentation.

-- 
Daniel Jacobowitz
CodeSourcery

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

Daniel Jacobowitz wrote:
>> Leaving that aside, if I do set the breakpoint correctly at virtual
>> address (e.g.) 0xC0123456 qemu will (correctly I guess) cause an
>> exception for code offset 0x123456 (as CS base is 0xC0000000). GDB
>> however then doesn't recognize its own breakpoint as it only remembers
>> having set one at 0xC0123456, and apparently doesn't translate the
>> given exception address of CS:0x123456 back to a virtual one.
> Which segment is it running from at this point?  Qemu may be reporting
> the wrong address.

After paging is switched on, all code always runs in the CS that is
based at 0xC0000000. As I see it qemu reports the address just fine.
As I wrote in my original mail it only works if I set two breakpoints,
one at 0x123456 and one at 0xC0123456. Makes sense then, the
0xC0123456 triggers the breakpoint exception within qemu and the
0x123456 is used by GDB to recognize the offset it gets from qemu.
It's a mess, really, and probably not qemu's fault.

>> So all in all GDB just doesn't seem to cope very well with segmented
>> memory.
> Correct.  It doesn't know anything at all about i386 segmentation.

Well, that explains it then, I guess. In that case I don't really see
a clean solution for it.

Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Daniel Jacobowitz]

On Thu, Nov 09, 2006 at 12:57:29AM +0100, Marcel Kilgus wrote:
> Daniel Jacobowitz wrote:
> >> Leaving that aside, if I do set the breakpoint correctly at virtual
> >> address (e.g.) 0xC0123456 qemu will (correctly I guess) cause an
> >> exception for code offset 0x123456 (as CS base is 0xC0000000). GDB
> >> however then doesn't recognize its own breakpoint as it only remembers
> >> having set one at 0xC0123456, and apparently doesn't translate the
> >> given exception address of CS:0x123456 back to a virtual one.
> > Which segment is it running from at this point?  Qemu may be reporting
> > the wrong address.
> 
> After paging is switched on, all code always runs in the CS that is
> based at 0xC0000000. As I see it qemu reports the address just fine.
> As I wrote in my original mail it only works if I set two breakpoints,
> one at 0x123456 and one at 0xC0123456. Makes sense then, the
> 0xC0123456 triggers the breakpoint exception within qemu and the
> 0x123456 is used by GDB to recognize the offset it gets from qemu.
> It's a mess, really, and probably not qemu's fault.

If qemu is ever sending "0x123456" back to GDB in the scenario you've
described, you can probably get the right behavior by simulating flat
memory: make qemu add the segment base to the pc when it sends it to
gdb.

At this point you're past what I know about i386 though.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [Qemu-devel] Re: Debugging with paging enabled

[andrzej zaborowski]

On 08/11/06, Marcel Kilgus <qemu@mail.kilgus.net> wrote:
> > Correct.  It doesn't know anything at all about i386 segmentation.
>
> Well, that explains it then, I guess. In that case I don't really see
> a clean solution for it.

If I understand the problem, the clean solution is having the
debugging symbols at the right addresses: in code that runs with
paging enabled symbols should be at their virtual addresses and the
other symbols at physical addresses. Since the period from start to
enabling paging is usually short, I think most kernels generate only
virtual addresses for the symbols. I don't remember the ld option to
move text-base to the right address but see Linux. GDB works perfectly
for it. It doesn't need to know about segmentation if your symbols are
correct.
-- 
balrog 2oo6

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

andrzej zaborowski wrote:
>> Well, that explains it then, I guess. In that case I don't really see
>> a clean solution for it.
> If I understand the problem, the clean solution is having the
> debugging symbols at the right addresses: in code that runs with
> paging enabled symbols should be at their virtual addresses and the
> other symbols at physical addresses. Since the period from start to
> enabling paging is usually short, I think most kernels generate only
> virtual addresses for the symbols. I don't remember the ld option to
> move text-base to the right address but see Linux. GDB works perfectly
> for it. It doesn't need to know about segmentation if your symbols are
> correct.

No, as I see it, that doesn't solve the problem (and I did try to do
that by simply using objcopy and telling it to rebase all symbols).

OK, I dug a bit deeper and now I probably know far more about the
internals of qemu and the GDB serial debugging protocol than I ever
wanted. ;-)

Let's say the symbols are at the right location, then this happens
(all based on viewing the qemu code, I haven't actually debugged it):

GDB = ->, QEMU-GDB-STUB = <-

-> Z1,0xC0123456        Set hardware breakpoint at address 0xC0123456
<- OK                   Breakpoint set
-> c                    Continue to run virtual machine
<- S05                  qemu comes to the bp and returns with SIGTRAP.

At this point GDB knows that something has triggered the TRAP
exception, but from the answer alone it cannot know what. It will
certainly continue by requesting all registers of the virtual machine
and then look at the EIP value. But EIP, as CS is based at 0xC0000000,
will be 0x00123456 (note the lacking C at the most significant
nibble). It has no clue what to do with that one and simply passes the
exception on to the user (as seen in my mail that started the
thread).

Assuming that breakpoint locations are indeed meant to be virtual
addresses, GDB would have to evaluate the CS descriptor, add the CS
base to the EIP address and THEN check whether it knows the address.
But as it seems to be segment-agnostic it doesn't do that and things
break as a result.

Anybody seeing anything wrong with my reasoning?

All the best, Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Paul Brook]

> Assuming that breakpoint locations are indeed meant to be virtual
> addresses, GDB would have to evaluate the CS descriptor, add the CS
> base to the EIP address and THEN check whether it knows the address.
> But as it seems to be segment-agnostic it doesn't do that and things
> break as a result.

As Dan said, gdb knows nothing about x86 segmentation. As soon as you have 
nonzero segment bases you're pretty much on your own.

If you care about fixing this I suggest you get GDB folks to agree (and 
document) how segmented memory models should work, then implement that model 
in qemu. Hacking qemu to work for your particular case with unmodified gdb 
probably "breaks" something else, so doesn't seem to improve the overall 
usefulness.

I believe all common x86 OS use zero segment bases, and in long mode the CS/DS 
segment bases are ignored.

Paul

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

Paul Brook wrote:
> As Dan said, gdb knows nothing about x86 segmentation. As soon as you have
> nonzero segment bases you're pretty much on your own.

I was just wondering, because I didn't invent that "move the code to
3GB base" but it is (was?) the model used by Linux, at least in very
early versions (I'm not accustomed with the current code base). So I
didn't expect to be the first person stumbling over this shortcoming.

> If you care about fixing this I suggest you get GDB folks to agree (and
> document) how segmented memory models should work, then implement that model
> in qemu. Hacking qemu to work for your particular case with unmodified gdb
> probably "breaks" something else, so doesn't seem to improve the overall
> usefulness.

Nah, I've already spent way more time on this than I can really
afford, though it was quite fun for a change. And my hack works
perfectly for my needs, so I'm fine. At least my findings are now
documented in the list archives and anybody with a similar problem can
now read it up.

All the best, Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Daniel Jacobowitz]

On Fri, Nov 10, 2006 at 03:56:01PM +0000, Paul Brook wrote:
> > Assuming that breakpoint locations are indeed meant to be virtual
> > addresses, GDB would have to evaluate the CS descriptor, add the CS
> > base to the EIP address and THEN check whether it knows the address.
> > But as it seems to be segment-agnostic it doesn't do that and things
> > break as a result.
> 
> As Dan said, gdb knows nothing about x86 segmentation. As soon as you have 
> nonzero segment bases you're pretty much on your own.
> 
> If you care about fixing this I suggest you get GDB folks to agree (and 
> document) how segmented memory models should work, then implement that model 
> in qemu. Hacking qemu to work for your particular case with unmodified gdb 
> probably "breaks" something else, so doesn't seem to improve the overall 
> usefulness.

I'd recommend the even simpler hack of having qemu report a PC that
included the segment base :-)

-- 
Daniel Jacobowitz
CodeSourcery

[Qemu-devel] Re: Debugging with paging enabled

[Marcel Kilgus]

Daniel Jacobowitz wrote:
> I'd recommend the even simpler hack of having qemu report a PC that
> included the segment base :-)

Probably sounds easier than it is, as seen the serial protocol doesn't
include the PC when a breakpoint fires. qemu would have to
intentionally report a wrong EIP the next time GDB requests the
complete register set and the consequences of that I don't even want
to imagine (what if it writes that set of registers back?).

Or do you mean that qemu should always and in all situations report an
adjusted EIP to GDB and re-adjust all EIPs it gets? Interesting idea,
but I guess that too would lead to dozens of other problems.

All the best, Marcel

Re: [Qemu-devel] Re: Debugging with paging enabled

[Daniel Jacobowitz]

On Fri, Nov 10, 2006 at 08:01:54PM +0100, Marcel Kilgus wrote:
> Or do you mean that qemu should always and in all situations report an
> adjusted EIP to GDB and re-adjust all EIPs it gets? Interesting idea,
> but I guess that too would lead to dozens of other problems.

Yes precisely.  I doubt it would lead to any problems at all; and it
would be a localized change in two places in the GDB stub.

-- 
Daniel Jacobowitz
CodeSourcery