[Qemu-devel] Re: gdbstub: packet reply is too long

[Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1982 bytes --]

Andreas Schultz wrote:
> Hi,
> 
> I'm trying to debug a 32bit (i386) kernel on a 64bit host under kvm, but
> gdb always refuses it with:
> Remote 'g' packet reply is too long:
> 0000000000000000000000000000000000000000000000002306000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f0ff0000000000000230020000f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
>  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 
> I tried kvm-79 and kvm-git from today. gdb is v6.8 (ubuntu intrepid)
> 
> Any hints?

QEMU and also KVM's x86_64 version are currently broken /wrt debugging
targets < 64 bits. I've posted a fix a while ago, but there were
concerns that gdb should better be enhanced (which is basically true,
but unrealistic to achieve in the near future).

In the meantime, you may want to have a look at my kvm [1] or qemu patch
queue [2]. Note that unless you provide a binary to gdb that is 32 bit
already, you may have to switch the architecture manually ("set arch
i386" or "i8086") before attaching.

Jan

[1] git://git.kiszka.org/kvm-userspace.git gdb-queue
[2] git://git.kiszka.org/qemu.git gdb-queue


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Paul Brook]

> > I'm trying to debug a 32bit (i386) kernel on a 64bit host under kvm, but
> > gdb always refuses it with:

> QEMU and also KVM's x86_64 version are currently broken /wrt debugging
> targets < 64 bits. I've posted a fix a while ago, but there were
> concerns that gdb should better be enhanced (which is basically true,
> but unrealistic to achieve in the near future).

I still maintain that making the g packet format depend on the current CPU 
more is absolutely the wrong way to fix this.

Paul

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

Paul Brook wrote:
>>> I'm trying to debug a 32bit (i386) kernel on a 64bit host under kvm, but
>>> gdb always refuses it with:
> 
>> QEMU and also KVM's x86_64 version are currently broken /wrt debugging
>> targets < 64 bits. I've posted a fix a while ago, but there were
>> concerns that gdb should better be enhanced (which is basically true,
>> but unrealistic to achieve in the near future).
> 
> I still maintain that making the g packet format depend on the current CPU 
> more is absolutely the wrong way to fix this.

From a higher perspective, it is surely not the cleanest approach. But
it still appears to be the only one which helps us working around this
gdb shortcoming.

There are internal issues in gdb (hard coupling of current and target
arch) that will not allow this to be fixed in the near future, check the
code (or my posting I sent out last month). Moreover, this only restores
qemu to a state that it used to have for quite a while. I really wonder
why we should voluntarily break this support for the users.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Paul Brook]

On Saturday 20 December 2008, Jan Kiszka wrote:
> Paul Brook wrote:
> >>> I'm trying to debug a 32bit (i386) kernel on a 64bit host under kvm,
> >>> but gdb always refuses it with:
> >>
> >> QEMU and also KVM's x86_64 version are currently broken /wrt debugging
> >> targets < 64 bits. I've posted a fix a while ago, but there were
> >> concerns that gdb should better be enhanced (which is basically true,
> >> but unrealistic to achieve in the near future).
> >
> > I still maintain that making the g packet format depend on the current
> > CPU more is absolutely the wrong way to fix this.
>
> From a higher perspective, it is surely not the cleanest approach. But
> it still appears to be the only one which helps us working around this
> gdb shortcoming.

Actually it isn't. You could add an explicit switch.

Paul

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

Paul Brook wrote:
> On Saturday 20 December 2008, Jan Kiszka wrote:
>> Paul Brook wrote:
>>>>> I'm trying to debug a 32bit (i386) kernel on a 64bit host under kvm,
>>>>> but gdb always refuses it with:
>>>> QEMU and also KVM's x86_64 version are currently broken /wrt debugging
>>>> targets < 64 bits. I've posted a fix a while ago, but there were
>>>> concerns that gdb should better be enhanced (which is basically true,
>>>> but unrealistic to achieve in the near future).
>>> I still maintain that making the g packet format depend on the current
>>> CPU more is absolutely the wrong way to fix this.
>> From a higher perspective, it is surely not the cleanest approach. But
>> it still appears to be the only one which helps us working around this
>> gdb shortcoming.
> 
> Actually it isn't. You could add an explicit switch.

And what would this buy us? I would have to go from your gdb terminal to
qemu, probably the monitor, just to switch manually what now happens
automatically. I don't see the case where you wouldn't want to switch
when you try to debug 16 or 32 bit code, so what would be the gain? Or
do you want some switch to disable this automatic register format switching?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Paul Brook]

> >> From a higher perspective, it is surely not the cleanest approach. But
> >> it still appears to be the only one which helps us working around this
> >> gdb shortcoming.
> >
> > Actually it isn't. You could add an explicit switch.
>
> And what would this buy us? I would have to go from your gdb terminal to
> qemu, probably the monitor, just to switch manually what now happens
> automatically. I don't see the case where you wouldn't want to switch
> when you try to debug 16 or 32 bit code, so what would be the gain? Or
> do you want some switch to disable this automatic register format
> switching?

Because, as I've said several times before, the "automatic switching" is 
wrong. It may happen to work in your very limited circumstances, but there 
are many fairly common circumstances (e.g. stepping between a 64-bit kernel 
and a 32-bit userspace) where it's just plain broken. I object strongly to 
a "fix" that prevents a proper gdb from working.

> There are internal issues in gdb (hard coupling of current and target
> arch) that will not allow this to be fixed in the near future

Really? I'm pretty sure other architectures already manage it.

Paul

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

Paul Brook wrote:
>>>> From a higher perspective, it is surely not the cleanest approach. But
>>>> it still appears to be the only one which helps us working around this
>>>> gdb shortcoming.
>>> Actually it isn't. You could add an explicit switch.
>> And what would this buy us? I would have to go from your gdb terminal to
>> qemu, probably the monitor, just to switch manually what now happens
>> automatically. I don't see the case where you wouldn't want to switch
>> when you try to debug 16 or 32 bit code, so what would be the gain? Or
>> do you want some switch to disable this automatic register format
>> switching?
> 
> Because, as I've said several times before, the "automatic switching" is 
> wrong. It may happen to work in your very limited circumstances, but there 
> are many fairly common circumstances (e.g. stepping between a 64-bit kernel 
> and a 32-bit userspace) where it's just plain broken. I object strongly to 
> a "fix" that prevents a proper gdb from working.

Well, I'm using gdb over qemu and kvm in precisely that hybrid
scenarios, but also in normal ones. Heavily. And I'm only able to do
this because of the workaround. But I'm willing to learn about scenarios
where it causes /regressions/.

> 
>> There are internal issues in gdb (hard coupling of current and target
>> arch) that will not allow this to be fixed in the near future
> 
> Really? I'm pretty sure other architectures already manage it.

What other archs are comparably weird like x86?

Please have a look at the gdb code that I pointed out and explain to me
how it can be fixed so that the next gdb version will be fine again with
current qemu. I will happily do the patching then. Though, it would
still leave us with broken setups until that release is spread sufficiently.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Paul Brook]

> Well, I'm using gdb over qemu and kvm in precisely that hybrid
> scenarios, but also in normal ones. Heavily. And I'm only able to do
> this because of the workaround. But I'm willing to learn about scenarios
> where it causes /regressions/.

I find that hard to believe. Doesn't it break horribly as soon as the CPU 
switches modes?

It's not just regressions that are important. It's the fact that once qemu has 
your automatic switching hack it's impossible to make it work properly.

> >> There are internal issues in gdb (hard coupling of current and target
> >> arch) that will not allow this to be fixed in the near future
> >
> > Really? I'm pretty sure other architectures already manage it.
>
> What other archs are comparably weird like x86?

ARM has multiple instruction sets/cpu modes (and can mix the two within the 
same process). PPC and MIPS also have something similar, though I'm not sure 
how well they're supported by gdb.

I suspect you may be approaching this the wrong way. Instead of trying to 
switch architectures, you probably need to teach the 64-bit debugger how to 
debug 32-bit code.

Paul

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2555 bytes --]

Paul Brook wrote:
>> Well, I'm using gdb over qemu and kvm in precisely that hybrid
>> scenarios, but also in normal ones. Heavily. And I'm only able to do
>> this because of the workaround. But I'm willing to learn about scenarios
>> where it causes /regressions/.
> 
> I find that hard to believe. Doesn't it break horribly as soon as the CPU 
> switches modes?

It still breaks in very few corner cases, but that's (most probably) due
to the fact the gdb is not yet well prepared to switch sub-architectures
properly during runtime. However, it only breaks in cases that wouldn't
work without the switching anyway. And you can heal them by restarting gdb.

> 
> It's not just regressions that are important. It's the fact that once qemu has 
> your automatic switching hack it's impossible to make it work properly.

Interestingly, qemu used to work precisely like that before.

> 
>>>> There are internal issues in gdb (hard coupling of current and target
>>>> arch) that will not allow this to be fixed in the near future
>>> Really? I'm pretty sure other architectures already manage it.
>> What other archs are comparably weird like x86?
> 
> ARM has multiple instruction sets/cpu modes (and can mix the two within the 
> same process). PPC and MIPS also have something similar, though I'm not sure 
> how well they're supported by gdb.

Do those archs also have multiple register layouts that are coupled to
those different instruction sets? Do they switch the instruction sets
via 'set arch'? I think x86 is (historically) special here.

> 
> I suspect you may be approaching this the wrong way. Instead of trying to 
> switch architectures, you probably need to teach the 64-bit debugger how to 
> debug 32-bit code.

As I said, the problem in gdb is the hard coupling of the target
architecture (which could perfectly stay at x86_64 for a session) and
the current debugger architecture (which should be switched according to
the current CPU mode). Unfortunately, this is not yet feasible with gdb,
see gdbarch_update_p().

Fixing this (once understood what are all the problems preventing a fix
for several years now) is one thing, keeping the workaround for current
gdb in qemu is, IMHO, another. Right now we don't have a gdb fix in
sight, so I'm simply voting for reintroducing the workaround. That's
all. We can kill it or make it optional once the issue is solved. But we
should _not_ do this _before_ it is solved, causing only pain to people
who just want to use the gdbstub.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Paul Brook]

> Do those archs also have multiple register layouts that are coupled to
> those different instruction sets? Do they switch the instruction sets
> via 'set arch'?

No, but neither does x86. You just need to teach gdb how to debug 32-bit code 
on a 64-bit target.

> I think x86 is (historically) special here.

I think it's just that noone's made the effort to fix this problem.

> Fixing this (once understood what are all the problems preventing a fix
> for several years now) is one thing, keeping the workaround for current
> gdb in qemu is, IMHO, another. Right now we don't have a gdb fix in
> sight, so I'm simply voting for reintroducing the workaround. That's
> all. We can kill it or make it optional once the issue is solved. But we
> should _not_ do this _before_ it is solved, causing only pain to people
> who just want to use the gdbstub.

I don't buy this argument. You already have to tell gdb which mode to work in, 
so I don't see any reason why you shouldn't also tell qemu.

Recent GDB interrogate the target when they connect, and the target describes 
the registers it has. This isn't currently implemented on x86 in qemu, but 
I'm pretty sure it'd be incompatible with dynamic switching.

Paul

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

Paul Brook wrote:
>> Do those archs also have multiple register layouts that are coupled to
>> those different instruction sets? Do they switch the instruction sets
>> via 'set arch'?
> 
> No, but neither does x86. You just need to teach gdb how to debug 32-bit code 
> on a 64-bit target.

And that means setting current_gdbarch while keeping target_gdbarch -
that's where reality (existing gdb code) bites us. Again, I'm not
arguing against fixing this, I'm arguing in keeping qemu's workaround
until this is done. I will look into the gdb part, but one after the other.

> 
>> I think x86 is (historically) special here.
> 
> I think it's just that noone's made the effort to fix this problem.
> 
>> Fixing this (once understood what are all the problems preventing a fix
>> for several years now) is one thing, keeping the workaround for current
>> gdb in qemu is, IMHO, another. Right now we don't have a gdb fix in
>> sight, so I'm simply voting for reintroducing the workaround. That's
>> all. We can kill it or make it optional once the issue is solved. But we
>> should _not_ do this _before_ it is solved, causing only pain to people
>> who just want to use the gdbstub.
> 
> I don't buy this argument. You already have to tell gdb which mode to work in, 
> so I don't see any reason why you shouldn't also tell qemu.

Sorry, this doesn't parse for me. What mechanism for qemu do you have in
mind precisely?

However, do we agree now that we have to adopt qemu to make it work with
current gdb?

> 
> Recent GDB interrogate the target when they connect, and the target describes 
> the registers it has. This isn't currently implemented on x86 in qemu, but 
> I'm pretty sure it'd be incompatible with dynamic switching.

I haven't checked gdb /wrt this protocol and x86 in details yet. But as
far as I understand it, it does negotiation during startup. I wouldn't
be surprised that all what the backend told gdb becomes pointless once
you type "set arch i386". But this seems to be an orthogonal issue. Of
course, we would have to communicate x86-64 capabilities this way, not
the mode-dependent set.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Daniel Jacobowitz]

On Sun, Dec 21, 2008 at 12:44:04AM +0100, Jan Kiszka wrote:
> And that means setting current_gdbarch while keeping target_gdbarch -
> that's where reality (existing gdb code) bites us. Again, I'm not
> arguing against fixing this, I'm arguing in keeping qemu's workaround
> until this is done. I will look into the gdb part, but one after the other.

No, it does not mean setting current_gdbarch different from
target_gdbarch.  With the current gdbarch set to a 64-bit one that
accurately describes the target, GDB should be able to debug code
running in 32-bit mode.  If it can't, there are simply bugs in GDB to
fix.

If you'd like to reach some solution to this problem, which I've seen
come up on the QEMU list a half-dozen times now, please describe how
you're using GDB on the gdb@sourceware.org mailing list and let's see
if we can't fix the GDB bugs.  I'm pretty sure that any solution is
going to involve always transferring the x86-64 register set, though.

-- 
Daniel Jacobowitz
CodeSourcery

[Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

Daniel Jacobowitz wrote:
> On Sun, Dec 21, 2008 at 12:44:04AM +0100, Jan Kiszka wrote:
>> And that means setting current_gdbarch while keeping target_gdbarch -
>> that's where reality (existing gdb code) bites us. Again, I'm not
>> arguing against fixing this, I'm arguing in keeping qemu's workaround
>> until this is done. I will look into the gdb part, but one after the other.
> 
> No, it does not mean setting current_gdbarch different from
> target_gdbarch.  With the current gdbarch set to a 64-bit one that
> accurately describes the target, GDB should be able to debug code
> running in 32-bit mode.  If it can't, there are simply bugs in GDB to
> fix.

Well, in the current gdb design, current_gdbarch is consulted when
disassembling the code while target_gdbarch defines the register set
that is exchanged with the remote stub.

> 
> If you'd like to reach some solution to this problem, which I've seen
> come up on the QEMU list a half-dozen times now, please describe how
> you're using GDB on the gdb@sourceware.org mailing list and let's see
> if we can't fix the GDB bugs.  I'm pretty sure that any solution is
> going to involve always transferring the x86-64 register set, though.

I'm pretty sure that the final solution will involve extended x86
register sets in order to inform the frontend about the full target CPU
state so that it can set the right current_gdbarch automatically. That's
one reason (the other is current/target_gdbarch decoupling) why I see no
quick "bug fix" on the gdb side to actually solve the issue and suggest
the reintroduction of the qemu workaround until gdb is enhanced
appropriately.

But you I right, it's time to start a discussion on the gdb list,
hopefully laying the ground for a better x86 low-level support. And
Maybe I actually miss some smart intermediate step towards this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Daniel Jacobowitz]

On Mon, Dec 29, 2008 at 03:58:47PM +0100, Jan Kiszka wrote:
> Well, in the current gdb design, current_gdbarch is consulted when
> disassembling the code while target_gdbarch defines the register set
> that is exchanged with the remote stub.

This is a transitional state.  Really, there isn't supposed to be a
'current' gdbarch; we're already moving away from it.

Thinking about it some more you may be right about the overall
solution though, sorry.  The target_gdbarch idea is likely to stick
around for a while.  But some work will have to be done if current and
target architectures have different register sets :-(

> I'm pretty sure that the final solution will involve extended x86
> register sets in order to inform the frontend about the full target CPU
> state so that it can set the right current_gdbarch automatically.

Isn't everything we need for this in eflags already?

-- 
Daniel Jacobowitz
CodeSourcery

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

Daniel Jacobowitz wrote:
> On Mon, Dec 29, 2008 at 03:58:47PM +0100, Jan Kiszka wrote:
>> Well, in the current gdb design, current_gdbarch is consulted when
>> disassembling the code while target_gdbarch defines the register set
>> that is exchanged with the remote stub.
> 
> This is a transitional state.  Really, there isn't supposed to be a
> 'current' gdbarch; we're already moving away from it.
> 
> Thinking about it some more you may be right about the overall
> solution though, sorry.  The target_gdbarch idea is likely to stick
> around for a while.  But some work will have to be done if current and
> target architectures have different register sets :-(

I'll start a thread on the gdb list today, CC'ing you. Would be nice if
you could then add more details on what you think would be required to
achieve this.

> 
>> I'm pretty sure that the final solution will involve extended x86
>> register sets in order to inform the frontend about the full target CPU
>> state so that it can set the right current_gdbarch automatically.
> 
> Isn't everything we need for this in eflags already?

You need CR0.PE to detect if you are in real or protected mode. And then
you need GDTR/LDTR to find the descriptor CS is pointing at, parsing it
to detect if you are running 16, 32 or 64 bit code (by default). Those
extensions would also be useful in order to decode memory addresses in
case descriptor.base != 0 (or if it's CS >> 4, ie. you are in real
mode). We have some usable patches for this @work, at least for 16 vs.
32 bit. But it's clear that more work is needed to get things upstream
and we should first agree on how things should be done there, e.g. how
to extend the register set and how to communicate that extension between
backend and frontend.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jamie Lokier]

Jan Kiszka wrote:
> You need CR0.PE to detect if you are in real or protected mode. And then
> you need GDTR/LDTR to find the descriptor CS is pointing at, parsing it
> to detect if you are running 16, 32 or 64 bit code (by default). Those
> extensions would also be useful in order to decode memory addresses in
> case descriptor.base != 0 (or if it's CS >> 4, ie. you are in real
> mode).

If you're going to decode segment descriptors (great idea, btw, and
helpful for threaded code), it might be better to supply the CPU's
internal segment state, if that's possible, instead of looking at the
LDT/GDT in memory, since the CPU's state can differ from the memory
version when the latter is written to.

-- Jamie

Re: [Qemu-devel] Re: gdbstub: packet reply is too long

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

Jamie Lokier wrote:
> Jan Kiszka wrote:
>> You need CR0.PE to detect if you are in real or protected mode. And then
>> you need GDTR/LDTR to find the descriptor CS is pointing at, parsing it
>> to detect if you are running 16, 32 or 64 bit code (by default). Those
>> extensions would also be useful in order to decode memory addresses in
>> case descriptor.base != 0 (or if it's CS >> 4, ie. you are in real
>> mode).
> 
> If you're going to decode segment descriptors (great idea, btw, and
> helpful for threaded code), it might be better to supply the CPU's
> internal segment state, if that's possible, instead of looking at the
> LDT/GDT in memory, since the CPU's state can differ from the memory
> version when the latter is written to.

Good point. I included this in an initial suggestion of an extended
register set, see [1]. Providing this information will likely remain
VM-business, but that doesn't mean we shouldn't use it when available.

Jan

[1] http://sourceware.org/ml/gdb/2009-01/msg00008.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]