[Qemu-devel] qmeu-arm SIGSEGV for self-modifying code

[Qemu-devel] qmeu-arm SIGSEGV for self-modifying code

[John Reiser]

[Moving here from  https://bugzilla.redhat.com/show_bug.cgi?id=1493304 ]

qemu-arm from qemu-user-2.10.0-1.fc27.x86_64 (thus emulating 32-bit ARM on x86_64)
generates SIGSEGV when code modifies a never-previously executed instruction
that is on a writable page and is 848 bytes ahead of pc.
A real armv7l processor allows this and executes as desired.
Why the difference?  How can it be changed?  Where is the documentation?

=====  the real processor (RaspberryPi 3B in 32-bit mode running Fedora 27)
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4
=====

The memory region in question is allocated via
   mmap2(0xf7000000,228092,PROT_EXEC|PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xf7000000
[and not changed via mprotect()] and written once to contain:
=====
0xf703704c:
         ldr r2,mflg_here  // pc+856
         orr r2,r2,r3  @ modify the instruction
=>      str r2,mflg_here  // pc+848    the faulting instruction

      [[snip about 848 bytes containing instructions only]]

0xf70373ac:
   mflg_here:  // The next instruction is re-written once.
         orr r3,r3,#0  @ flags |= MAP_{PRIVATE|ANON}  [QNX vs Linux]
=====

-- 

Re: [Qemu-devel] qmeu-arm SIGSEGV for self-modifying code

[Peter Maydell]

On 20 September 2017 at 00:13, John Reiser <jreiser@bitwagon.com> wrote:
> [Moving here from  https://bugzilla.redhat.com/show_bug.cgi?id=1493304 ]
>
> qemu-arm from qemu-user-2.10.0-1.fc27.x86_64 (thus emulating 32-bit ARM on
> x86_64)
> generates SIGSEGV when code modifies a never-previously executed instruction
> that is on a writable page and is 848 bytes ahead of pc.
> A real armv7l processor allows this and executes as desired.
> Why the difference?  How can it be changed?  Where is the documentation?
> The memory region in question is allocated via

> mmap2(0xf7000000,228092,PROT_EXEC|PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0)
> = 0xf7000000
> [and not changed via mprotect()] and written once to contain:
> =====
> 0xf703704c:
>         ldr r2,mflg_here  // pc+856
>         orr r2,r2,r3  @ modify the instruction
> =>      str r2,mflg_here  // pc+848    the faulting instruction
>
>      [[snip about 848 bytes containing instructions only]]
>
> 0xf70373ac:
>   mflg_here:  // The next instruction is re-written once.
>         orr r3,r3,#0  @ flags |= MAP_{PRIVATE|ANON}  [QNX vs Linux]

Is your guest program correctly performing the necessary cache
maintenance operations between writing to the instruction and
executing the modified version? The ARM architecture provides
very wide latitude to an implementation to prefetch and precache
instructions, so it's quite easy to write code that happens
to run by accident in one implementation but not in another.
QEMU's implementation means that for straightline code with
no branches it will tend to prefetch more aggressively than a
hardware implementation, but it's still architecturally valid
to do so.

For 32-bit ARM Linux guest code, this usually means you need to
call the cacheflush syscall -- the lack of any cacheflush
lines in your strace output suggests maybe you've forgotten
to do this.

The gcc __builtin___clear_cache() is usually the simplest
way to achieve the required cache maintenance (portably,
for any CPU architecture).

thanks
-- PMM

Re: [Qemu-devel] qmeu-arm SIGSEGV for self-modifying code

[Peter Maydell]

On 20 September 2017 at 11:21, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 20 September 2017 at 00:13, John Reiser <jreiser@bitwagon.com> wrote:
>> [Moving here from  https://bugzilla.redhat.com/show_bug.cgi?id=1493304 ]
>>
>> qemu-arm from qemu-user-2.10.0-1.fc27.x86_64 (thus emulating 32-bit ARM on
>> x86_64)
>> generates SIGSEGV when code modifies a never-previously executed instruction
>> that is on a writable page and is 848 bytes ahead of pc.
>> A real armv7l processor allows this and executes as desired.
>> Why the difference?  How can it be changed?  Where is the documentation?
>> The memory region in question is allocated via
>
>> mmap2(0xf7000000,228092,PROT_EXEC|PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0)
>> = 0xf7000000
>> [and not changed via mprotect()] and written once to contain:
>> =====
>> 0xf703704c:
>>         ldr r2,mflg_here  // pc+856
>>         orr r2,r2,r3  @ modify the instruction
>> =>      str r2,mflg_here  // pc+848    the faulting instruction
>>
>>      [[snip about 848 bytes containing instructions only]]
>>
>> 0xf70373ac:
>>   mflg_here:  // The next instruction is re-written once.
>>         orr r3,r3,#0  @ flags |= MAP_{PRIVATE|ANON}  [QNX vs Linux]
>
> Is your guest program correctly performing the necessary cache
> maintenance operations

...wait, I think I misread your bug report. You get the SEGV
on the store to the code, before it even gets to trying to
execute it?

thanks
-- PMM

Re: [Qemu-devel] qemu-arm SIGSEGV for self-modifying code

[John Reiser]

Thanks for your reply, Peter.  [I fixed my typo in the Subject: field of the header.]

>>> [Moving here from  https://bugzilla.redhat.com/show_bug.cgi?id=1493304 ]
>>>
>>> qemu-arm from qemu-user-2.10.0-1.fc27.x86_64 (thus emulating 32-bit ARM on
>>> x86_64)
>>> generates SIGSEGV when code modifies a never-previously executed instruction
>>> that is on a writable page and is 848 bytes ahead of pc.
>>> A real armv7l processor allows this and executes as desired.
>>> Why the difference?  How can it be changed?  Where is the documentation?
>>> The memory region in question is allocated via
>>
>>> mmap2(0xf7000000,228092,PROT_EXEC|PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0)
>>> = 0xf7000000
>>> [and not changed via mprotect()] and written once to contain:
>>> =====
>>> 0xf703704c:
>>>          ldr r2,mflg_here  // pc+856
>>>          orr r2,r2,r3  @ modify the instruction
>>> =>      str r2,mflg_here  // pc+848    the faulting instruction
>>>
>>>       [[snip about 848 bytes containing instructions only]]
>>>
>>> 0xf70373ac:
>>>    mflg_here:  // The next instruction is re-written once.
>>>          orr r3,r3,#0  @ flags |= MAP_{PRIVATE|ANON}  [QNX vs Linux]
>>
>> Is your guest program correctly performing the necessary cache
>> maintenance operations
> 
> ...wait, I think I misread your bug report. You get the SEGV
> on the store to the code, before it even gets to trying to
> execute it?

Yes, the SEGV occurs on the store, "long" before the re-written instruction ever is executed
(848 bytes in space, at least a thousand instructions in time.)
The region between the 'str' and the re-written instruction contains more than a dozen
branches (conditional, unconditional, subroutine calls) and several 'svc' system calls.

The execution sequence is:
   Use mmap2() to allocate 228092 bytes of new pages with read+write+execute permission.
   Generate code into those pages, writing each byte once in order, including at 'mflg_here'.
   Jump to 0xf703704c which is 0x3704c into the new region.  This is the first-ever
     execution in the new region.
   Immediately read-modify-write the instruction at 'mflg_here', which is 0x373ac bytes
     into the region and 848 bytes ahead of the pc.
   SIGSEGV at the 'str' of the read-modify-write.

The qemu-arm generated x86_64 code in the vicinity of the SIGSEGV is
=====
    0x55555599364c <static_code_gen_buffer+13308>:	mov    $0xf70373ac,%ebp
    0x555555993651 <static_code_gen_buffer+13313>:	mov    %gs:0x0(%ebp),%ebp
    0x555555993656 <static_code_gen_buffer+13318>:	mov    0xc(%r14),%ebx
    0x55555599365a <static_code_gen_buffer+13322>:	or     %ebx,%ebp
    0x55555599365c <static_code_gen_buffer+13324>:	mov    %ebp,0x8(%r14)
    0x555555993660 <static_code_gen_buffer+13328>:	mov    $0xf70373ac,%ebx
=> 0x555555993665 <static_code_gen_buffer+13333>:	mov    %ebp,%gs:(%ebx)
    0x555555993669 <static_code_gen_buffer+13337>:	mov    0x34(%r14),%ebp
    0x55555599366d <static_code_gen_buffer+13341>:	mov    %gs:0x0(%ebp),%ebx
    0x555555993672 <static_code_gen_buffer+13346>:	mov    %ebx,0x10(%r14)
(gdb) info reg
rax            0x1	1
rbx            0xf70373ac	4144198572
rcx            0x0	0
rdx            0x55555597b880	93824996587648
rsi            0x555555993640	93824996685376
rdi            0x5555582b6290	93825039819408
rbp            0xe3833022	0xe3833022
rsp            0x7fffffffd2c0	0x7fffffffd2c0
r8             0x0	0
r9             0x0	0
r10            0x55555597b7f0	93824996587504
r11            0x206	518
r12            0x555555993640	93824996685376
r13            0xf703704c	4144197708
r14            0x5555582b6290	93825039819408
r15            0x5555582ae5b8	93825039787448
rip            0x555555993665	0x555555993665 <static_code_gen_buffer+13333>
eflags         0x10286	[ PF SF IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
=====
Unfortunately gdb does not show the correct value of segment register %gs.
/proc/{qemu-arm}/maps shows no mapping at any address less than 4GiB, yet the x86_64 code
    0x55555599364c <static_code_gen_buffer+13308>:	mov    $0xf70373ac,%ebp
    0x555555993651 <static_code_gen_buffer+13313>:	mov    %gs:0x0(%ebp),%ebp
did perform a successful fetch from %gs:0xf70373ac.  The value 0xe3833022 in %ebp
is the correct new (modified by the OR) contents to be stored into address 0xf70373ac.


Environment variation:
    SIGSEGV with qemu-user-2.7.1-7.fc25.x86_64   under kernel 4.12.11-200.fc25.x86_64
    SIGSEGV with qemu-user-2.9.1-1.fc26.x86_64   under kernel 4.12.13-300.fc26.x86_64
    SIGSEGV with qemu-user-2.10.0-1.fc27.x86_64  under kernel 4.13.2-300.fc27.x86_64
    SIGSEGV with qemu-user 1:2.1+dfsg-12+deb8u6  under kernel 3.16.43-2+deb8u3 (x86_64)
       All SIGSEGV are cross-architecture emulation: running qemu-arm on x86_64.

    Success with qemu-user-2.10.0-1.fc27.armv7hl under kernel 4.13.2-300.fc27.armv7hl
       This success is "native emulation": running qemu-arm on armv7hl.

-- 
John

Re: [Qemu-devel] qemu-arm SIGSEGV for self-modifying code

[Peter Maydell]

On 20 September 2017 at 18:05, John Reiser <jreiser@bitwagon.com> wrote:
> Yes, the SEGV occurs on the store, "long" before the re-written
> instruction ever is executed

OK, I've identified the immediate cause for this SEGV.

(1) when the guest initially mmap()s at 0xf7000000 and
above we pass that through to the host as an mmap rwx
(2) later, the guest wants to execute from some part
of this region; QEMU marks those pages as non-writable
so that we can catch guest writes and invalidate our
translated code cache (we then mark the page writable
and resume the guest code). This is a host page at a time,
so it covers the memory we're trying to modify
(3) when the translated guest code writes to the memory,
we get a host SIGSEGV, which is expected. Unfortunately
we then fail to recognize it as a case of a guest
write to a page that QEMU marked non-writeable.
(4) The reason we don't recognize the address is that
our test for "is this valid" (h2g_valid()) checks that the
guest address is within the chunk of the host address
space that we've carved out for the guest, and the
amount of space we carve out for that is 0xf7000000.
So guest execution above that won't work properly
(really we should probably fail the mmap() rather than
letting it succeed but misbehave).

I don't really know why we use 0xf7000000 as our
reserved_va value here, though. Alex, you added that
years ago, can you remember why you used that value?

You can work around this by passing a different
reserved-space value to QEMU with -R -- in theory
0xffff0000 would be the right answer (there's a
kernel-page above that), but QEMU says it can't
reserved that much space. -R 0xfffe0000 seems to get it
past the immediate segv problem.

thanks
-- PMM

Re: [Qemu-devel] qemu-arm SIGSEGV for self-modifying code

[Alexander Graf]

On 20.09.17 20:04, Peter Maydell wrote:
> On 20 September 2017 at 18:05, John Reiser <jreiser@bitwagon.com> wrote:
>> Yes, the SEGV occurs on the store, "long" before the re-written
>> instruction ever is executed
> 
> OK, I've identified the immediate cause for this SEGV.
> 
> (1) when the guest initially mmap()s at 0xf7000000 and
> above we pass that through to the host as an mmap rwx
> (2) later, the guest wants to execute from some part
> of this region; QEMU marks those pages as non-writable
> so that we can catch guest writes and invalidate our
> translated code cache (we then mark the page writable
> and resume the guest code). This is a host page at a time,
> so it covers the memory we're trying to modify
> (3) when the translated guest code writes to the memory,
> we get a host SIGSEGV, which is expected. Unfortunately
> we then fail to recognize it as a case of a guest
> write to a page that QEMU marked non-writeable.
> (4) The reason we don't recognize the address is that
> our test for "is this valid" (h2g_valid()) checks that the
> guest address is within the chunk of the host address
> space that we've carved out for the guest, and the
> amount of space we carve out for that is 0xf7000000.
> So guest execution above that won't work properly
> (really we should probably fail the mmap() rather than
> letting it succeed but misbehave).
> 
> I don't really know why we use 0xf7000000 as our
> reserved_va value here, though. Alex, you added that
> years ago, can you remember why you used that value?

IIRC I wanted to map the full 32 bits of address space possibly in use 
by a 32bit application, but leave some room for something, but I don't 
remember what that something was :)


Alex

Re: [Qemu-devel] qemu-arm SIGSEGV for self-modifying code

[John Reiser]

>> I don't really know why we use 0xf7000000 as our
>> reserved_va value here, though. Alex, you added that
>> years ago, can you remember why you used that value?
> 
> IIRC I wanted to map the full 32 bits of address space possibly in use by a 32bit application, but leave some room for something, but I don't remember what that something was :)

Now that I know the nature of the conflict, then I will spend a handful of instructions
to avoid [0xf7000000, +), and also the stack if it gets placed immediately below that.

Thank you, Peter and Alex.

-- 
John

Re: [Qemu-devel] qemu-arm SIGSEGV for self-modifying code

[Peter Maydell]

On 20 September 2017 at 23:13, John Reiser <jreiser@bitwagon.com> wrote:
> Now that I know the nature of the conflict, then I will spend a
> handful of instructions to avoid [0xf7000000, +), and also the
> stack if it gets placed immediately below that.

Cool. I figured out why -R 0xffff0000 didn't work and have sent
a patch for that; rth has some patches he's going to respin and
resend which will make -R 0xffff0000 the default and should help
in the sense of avoiding the guest SIGSEGV.

With your test binary and those fixes I now get it exiting early
with "PROT_EXEC|PROT_WRITE failed", which seems to be because it
does
 mmap2(0xffff0000,228092,PROT_EXEC|PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0)
= -1 errno=22

which is trying to mmap off the end of the valid address space
(as well as over the kernel commpage). I'm not sure how it's
calculating the address to mmap, because it's different this
time around.

thanks
-- PMM