[Qemu-devel] multi-bytes nop and 64bits

[Qemu-devel] multi-bytes nop and 64bits

[Mark Karpeles]

Hello,

I'm having some troubles with multibytes nops in qemu 0.10.1.

After following the initial 2006 thread :

http://www.mail-archive.com/qemu-devel@nongnu.org/msg06470.html

I found out that support for 64bits system wasn't 100% sure.

Code given to gas:

//nopl 0x0(%rax)
.byte 0x0f, 0x1f, 0x40, 0x00
jmp EXT_C(init_paging)

Dumped by objdump:

ffffffffc0001000 <_text>:
ffffffffc0001000:       0f 1f 40 00             nopl   0x0(%rax)
ffffffffc0001004:       e9 29 51 00 00          jmpq   ffffffffc0006132
<init_paging>
ffffffffc0001009:       00 00                   add    %al,(%rax)

As seen by qemu (QEMU PC emulator version 0.10.1):

0x0000000000201000:  (bad)
0x0000000000201002:  add    %bpl,%cl
0x0000000000201005:  sub    %edx,0x0(%rcx)
0x0000000000201008:  add    %al,(%rax)
0x000000000020100a:  add    %al,(%rax)
(The code after this line is not meant to be executed, it's read-only
data, and so the system will crash)

Because the system I'm compiling code on supports multibytes nops, gcc
will use those in some parts of the generated code, rendering it
unusable under qemu. I'll add some flags to avoid this for now, however
as qemu has (partial) support for multibytes nops, I believe this is
something that should be fixed.

The bytecodes are documented in intel manual 2B, "NOP" chapter. NOP
operations can be up to 9 bytes long.

Valid NOP bytecodes are (according to Intel manual):

90H
66 90H
0F 1F 00H
0F 1F 40 00H (this is mine)
0F 1F 44 00 00H
66 0F 1F 44 00 00H
0F 1F 80 00 00 00 00H
0F 1F 84 00 00 00 00 00H
66 0F 1F 84 00 00 00 00 00H

I didn't test with an older qemu to see if it works, I'll eventually try
if this problem can't be easily solved by someone who knows qemu better
than I do (this is not going to be hard).

Best regards,
Mark Karpeles

Re: [Qemu-devel] multi-bytes nop and 64bits

[malc]

On Mon, 20 Apr 2009, Mark Karpeles wrote:

> Hello,
> 
> I'm having some troubles with multibytes nops in qemu 0.10.1.
> 
> After following the initial 2006 thread :
> 
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg06470.html
> 
> I found out that support for 64bits system wasn't 100% sure.
> 
> Code given to gas:
> 
> //nopl 0x0(%rax)
> .byte 0x0f, 0x1f, 0x40, 0x00
> jmp EXT_C(init_paging)
> 
> Dumped by objdump:
> 
> ffffffffc0001000 <_text>:
> ffffffffc0001000:       0f 1f 40 00             nopl   0x0(%rax)
> ffffffffc0001004:       e9 29 51 00 00          jmpq   ffffffffc0006132
> <init_paging>
> ffffffffc0001009:       00 00                   add    %al,(%rax)
> 
> As seen by qemu (QEMU PC emulator version 0.10.1):
> 
> 0x0000000000201000:  (bad)
> 0x0000000000201002:  add    %bpl,%cl
> 0x0000000000201005:  sub    %edx,0x0(%rcx)
> 0x0000000000201008:  add    %al,(%rax)
> 0x000000000020100a:  add    %al,(%rax)
> (The code after this line is not meant to be executed, it's read-only
> data, and so the system will crash)


The disassembler is disconnected from the main translation engine, so
what you see (produced by the older version of i386 disassmbler from
GNU binutils) might not be what QEMU actually translates/executes. To
be sure use gdbstub and recent GDB to see, hopefully, more coherent view.

> 
> Because the system I'm compiling code on supports multibytes nops, gcc
> will use those in some parts of the generated code, rendering it
> unusable under qemu. I'll add some flags to avoid this for now, however
> as qemu has (partial) support for multibytes nops, I believe this is
> something that should be fixed.
> 
> The bytecodes are documented in intel manual 2B, "NOP" chapter. NOP
> operations can be up to 9 bytes long.
> 
> Valid NOP bytecodes are (according to Intel manual):
> 
> 90H
> 66 90H
> 0F 1F 00H
> 0F 1F 40 00H (this is mine)
> 0F 1F 44 00 00H
> 66 0F 1F 44 00 00H
> 0F 1F 80 00 00 00 00H
> 0F 1F 84 00 00 00 00 00H
> 66 0F 1F 84 00 00 00 00 00H
> 
> I didn't test with an older qemu to see if it works, I'll eventually try
> if this problem can't be easily solved by someone who knows qemu better
> than I do (this is not going to be hard).
> 

FWIW if i'm reading the sources correctly the above sequence (marked as
"mine") should work just fine with current QEMU.

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] multi-bytes nop and 64bits

[M. Karpelès]

Hi,

Le mardi 21 avril 2009 à 02:10 +0400, malc a écrit :
> On Mon, 20 Apr 2009, Mark Karpeles wrote:
> > As seen by qemu (QEMU PC emulator version 0.10.1):
> > 
> > 0x0000000000201000:  (bad)
> > 0x0000000000201002:  add    %bpl,%cl
> > 0x0000000000201005:  sub    %edx,0x0(%rcx)
> > 0x0000000000201008:  add    %al,(%rax)
> > 0x000000000020100a:  add    %al,(%rax)
> > (The code after this line is not meant to be executed, it's read-only
> > data, and so the system will crash)
> The disassembler is disconnected from the main translation engine, so
> what you see (produced by the older version of i386 disassmbler from
> GNU binutils) might not be what QEMU actually translates/executes. To
> be sure use gdbstub and recent GDB to see, hopefully, more coherent view.

Oh my... I checked more and tried running this without the qemu debug
mode "in_asm", and it's working indeed.
So, in_asm will decompile memory until it reaches a "stop" operation
(ret, %cr3 changed, etc), however if it doesn't reach such an operation
it will continue reading memory.

I wasn't expecting this bug because the fact it was continuing to read
memory caused a page-fault which was catched by the kernel I'm writing.
I never though the debugger could trigger stuff like OS pagefaults (and
of course it was happening with %rip exactly pointing to a long nop,
that's why I wasn't understanding the origin of the page fault).

Maybe this could be added to some documentation: "If guest OS crashes
because of a page fault on a long nop operation, try disabling in_asm
debugging mode".

Or maybe this debugging mode could be fixed to handle long nops? This
debugging mode is really helpful to understand what happened during a
crash, and for people writing their own kernel (like me) it's a life
savior.

I'm not sure how the decompiler works in qemu, but I'll have a look and
see if anything can be done to at least prevent it throwing pagefaults
on bogus addresses when a long nop happens.

Anyway thanks for the tip, you just saved me more days of head
scratching.

Mark

Re: [Qemu-devel] multi-bytes nop and 64bits

[Mark Karpeles]

Hi,

Le mardi 21 avril 2009 à 02:10 +0400, malc a écrit :
> On Mon, 20 Apr 2009, Mark Karpeles wrote:
> > As seen by qemu (QEMU PC emulator version 0.10.1):
> > 
> > 0x0000000000201000:  (bad)
> > 0x0000000000201002:  add    %bpl,%cl
> > 0x0000000000201005:  sub    %edx,0x0(%rcx)
> > 0x0000000000201008:  add    %al,(%rax)
> > 0x000000000020100a:  add    %al,(%rax)
> > (The code after this line is not meant to be executed, it's read-only
> > data, and so the system will crash)
> The disassembler is disconnected from the main translation engine, so
> what you see (produced by the older version of i386 disassmbler from
> GNU binutils) might not be what QEMU actually translates/executes. To
> be sure use gdbstub and recent GDB to see, hopefully, more coherent view.

Oh my... I checked more and tried running this without the qemu debug
mode "in_asm", and it's working indeed.
So, in_asm will decompile memory until it reaches a "stop" operation
(ret, %cr3 changed, etc), however if it doesn't reach such an operation
it will continue reading memory.

I wasn't expecting this bug because the fact it was continuing to read
memory caused a page-fault which was catched by the kernel I'm writing.
I never though the debugger could trigger stuff like OS pagefaults (and
of course it was happening with %rip exactly pointing to a long nop,
that's why I wasn't understanding the origin of the page fault).

Maybe this could be added to some documentation: "If guest OS crashes
because of a page fault on a long nop operation, try disabling in_asm
debugging mode".

Or maybe this debugging mode could be fixed to handle long nops? This
debugging mode is really helpful to understand what happened during a
crash, and for people writing their own kernel (like me) it's a life
savior.

I had a look at qemu's decompiler (an old version of i386-dec.c from
gdb), and "fixed" it to support nopl correctly (and at least avoid
weird crashes).

Patch:
http://ookoo.org/svn/snip/qemu-0.10.1-nopl-fix.patch

Anyway thanks for the tip, you just saved me more days of head
scratching.

Mark

Re: [Qemu-devel] multi-bytes nop and 64bits

[malc]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2961 bytes --]

On Tue, 21 Apr 2009, Mark Karpeles wrote:

> Hi,
> 
> Le mardi 21 avril 2009 ЪЪ 02:10 +0400, malc a ЪЪcrit :
> > On Mon, 20 Apr 2009, Mark Karpeles wrote:
> > > As seen by qemu (QEMU PC emulator version 0.10.1):
> > > 
> > > 0x0000000000201000:  (bad)
> > > 0x0000000000201002:  add    %bpl,%cl
> > > 0x0000000000201005:  sub    %edx,0x0(%rcx)
> > > 0x0000000000201008:  add    %al,(%rax)
> > > 0x000000000020100a:  add    %al,(%rax)
> > > (The code after this line is not meant to be executed, it's read-only
> > > data, and so the system will crash)
> > The disassembler is disconnected from the main translation engine, so
> > what you see (produced by the older version of i386 disassmbler from
> > GNU binutils) might not be what QEMU actually translates/executes. To
> > be sure use gdbstub and recent GDB to see, hopefully, more coherent view.
> 
> Oh my... I checked more and tried running this without the qemu debug
> mode "in_asm", and it's working indeed.
> So, in_asm will decompile memory until it reaches a "stop" operation
> (ret, %cr3 changed, etc), however if it doesn't reach such an operation
> it will continue reading memory.
> 
> I wasn't expecting this bug because the fact it was continuing to read
> memory caused a page-fault which was catched by the kernel I'm writing.
> I never though the debugger could trigger stuff like OS pagefaults (and
> of course it was happening with %rip exactly pointing to a long nop,
> that's why I wasn't understanding the origin of the page fault).
> 
> Maybe this could be added to some documentation: "If guest OS crashes
> because of a page fault on a long nop operation, try disabling in_asm
> debugging mode".

I think fixing disassembler is a better idea, disas.c:target_disas is
clearly broken, since:

void target_disas(FILE *out, target_ulong code, target_ulong size, int 
flags)
{
...
    for (pc = code; size > 0; pc += count, size -= count) {

So here size is unsigned and as such the stop condition can only become
true when translator and disassembler have the same idea how they decode
instructions (which was clearly not the case for hint_nop).
 
> Or maybe this debugging mode could be fixed to handle long nops? This
> debugging mode is really helpful to understand what happened during a
> crash, and for people writing their own kernel (like me) it's a life
> savior.
> 
> I had a look at qemu's decompiler (an old version of i386-dec.c from
> gdb), and "fixed" it to support nopl correctly (and at least avoid
> weird crashes).
> 
> Patch:
> http://ookoo.org/svn/snip/qemu-0.10.1-nopl-fix.patch

I'm not intimately familiar with binutils' disasm but it looks correct
save for the fact that hint_nop take M and not Ev as operand, which
might be irrelevant for disassembler though...

> Anyway thanks for the tip, you just saved me more days of head
> scratching.

You are welcome..

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] [PATCH] multi-bytes nop and 64bits

[Mark Karpeles]

Le mardi 21 avril 2009 à 21:20 +0400, malc a écrit :
> > Patch:
> > http://ookoo.org/svn/snip/qemu-0.10.1-nopl-fix.patch
> 
> I'm not intimately familiar with binutils' disasm but it looks correct
> save for the fact that hint_nop take M and not Ev as operand, which
> might be irrelevant for disassembler though...

To tell you the truth, I had a look at the current (ie. gdb 6.8) gdb
disassembler, and saw it was "Ev". It also produces correct result when
I look at the disassembled code (while the operand is irrelevant, it
gives a hint regarding the whole opcode's size).
I believe those who knows best how this decompiler works are those who
have continued to improve it, so I didn't look too deep in this, just
backported support for multibyte nop to code used in qemu (tried to port
the whole decompiler, but it changed too much, and I'm too lazy to
attempt to fix that).

I'm just adding [PATCH] to the subject, just to let everyone here know
this mail contains a patch. I successfully tested it on 3 machines
without problems and it already saved my day once by showing me where my
kernel was stopping (I'm in the process of porting from 32bits to
64bits, knowing what happens inside is a great help).

Anyway the decompiler obviously needs to be either re-made, or
re-imported from gdb, but this little patch will help until someone with
better knowledge of qemu and gdb does it.


Mark

Re: [Qemu-devel] [PATCH] multi-bytes nop and 64bits

[Avi Kivity]

Mark Karpeles wrote:
> I'm just adding [PATCH] to the subject, just to let everyone here know
> this mail contains a patch.
>   

It happens not to contain a patch, attachment fail?


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] [PATCH] multi-bytes nop and 64bits

[malc]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1999 bytes --]

On Tue, 21 Apr 2009, Mark Karpeles wrote:

> Le mardi 21 avril 2009 ЪЪ 21:20 +0400, malc a ЪЪcrit :
> > > Patch:
> > > http://ookoo.org/svn/snip/qemu-0.10.1-nopl-fix.patch
> > 
> > I'm not intimately familiar with binutils' disasm but it looks correct
> > save for the fact that hint_nop take M and not Ev as operand, which
> > might be irrelevant for disassembler though...
> 
> To tell you the truth, I had a look at the current (ie. gdb 6.8) gdb
> disassembler, and saw it was "Ev". It also produces correct result when
> I look at the disassembled code (while the operand is irrelevant, it
> gives a hint regarding the whole opcode's size).
> I believe those who knows best how this decompiler works are those who
> have continued to improve it, so I didn't look too deep in this, just
> backported support for multibyte nop to code used in qemu (tried to port
> the whole decompiler, but it changed too much, and I'm too lazy to
> attempt to fix that).

Sandpile also states that hint_nop's take Ev 
(http://sandpile.org/ia32/opc_grp.htm)

That said this particular instance (0f 1f) should be covered by the code
that is currently in HEAD, strangely enough all other hint_nops and
prefetches are not. FWIW your patch seems wrong in prefetch[nta|t0]
regard.

> I'm just adding [PATCH] to the subject, just to let everyone here know
> this mail contains a patch. I successfully tested it on 3 machines
> without problems and it already saved my day once by showing me where my
> kernel was stopping (I'm in the process of porting from 32bits to
> 64bits, knowing what happens inside is a great help).
> 
> Anyway the decompiler obviously needs to be either re-made, or
> re-imported from gdb, but this little patch will help until someone with
> better knowledge of qemu and gdb does it.
> 

Licensing issues prevents us from doing so, the disassembler was hand
picked by Blue Swirl to the latest version released under GPL2.

-- 
mailto:av1474@comtv.ru