[Qemu-devel] qemu-arm user mode does not work with ARM EABI

[Qemu-devel] qemu-arm user mode does not work with ARM EABI

[Martin Guy]

Summary:
  Ever since QEMU 0.8.1, qemu-arm-system mode works fine with ARM EABI
kernel and rootfs, but qemu-arm single-process mode is broken with
EABI, failing with "Unknown system call 0xf0005".
  This is ARM private system call "set_tls", called from both the
linuxthreads and the NPTL pthread implementations in glibc-2.4, and
unimplemented in QEMU.
  It is important that this mode work because scratchbox needs it to
be able to cross-compile Debian packages for the forthcoming
EABI-based Debian architecture.

Detail:
I have three kinds of ARM root filesystem:

- standard Debian ARM sarge using old-ABI: (glibc-2.3.6, linuxthreads)
- Voipio's "eabi-rootfs" which I gather uses EABI internally and glibc
shims to convert the system calls to old-ABI just before they hit the
kernel (glibc-2.3.4)
- Koen's EABI-toolchain-rootfs (from the Angstrom distro of
OpenEmbedded) which uses pure EABI (glibc-2.4,nptl)

# Here they are
martin:~/tftpboot$ ls
EABI-toolchain-rootfs/  debian-sarge-arm/  eabi-rootfs/

In qemu-system-arm (softmmu target) all three run fine (given
appropriate kernels) but qemu-arm fails on the second two.

# Even qemu-arm 0.8.1 works on old ABI
martin:~/tftpboot$ /usr/bin/qemu-arm -L `pwd`/deb* deb*/bin/echo hi
hi
martin:~/tftpboot$

# Both of the EABI scenarios fail the same way.

# With Debian QEMU (0.8.1):
martin:~/tftpboot$ /usr/bin/qemu-arm -L `pwd`/EABI* EABI*/bin/echo hi
Error: f0005
qemu: uncaught target signal 6 (Aborted) - exiting
martin:~/tftpboot$

# With qemu-arm 0.8.2 compiled from source (using gcc-3.4 of course):
martin:~/tftpboot$ /usr/local/bin/qemu-arm -L `pwd`/EABI* EABI*/bin/echo hi
qemu: Unsupported syscall: 983045
cannot set up thread-local storage: unknown error
martin:~/tftpboot$ echo 'obase=16; 983045' | bc
F0005
martin:~/tftpboot$

This is not a case of the new EABI picking up old ABI system calls,
which were encoded as SWI #(0x900000 + n).

A couple of debugging printfs show that with the pure EABI rootfs, the
system call number is picked up by QEMU from r7 as it should, and when
executing a simple binary it correctly picks up about 20 system calls
starting with 'brk; uname; access; open; fstat64; mmap2' and ending
with 'close; mmap2; [F0005]; truncate; io_submit; [fail]'.
Using the intermediate glibc-shims version, the same bogus system call
number is received via the old-ABI system call mechanism.

The Linux kernel source, while decoding ARM system calls says
in arch/arm/kernel/traps.c
 *  0x9f0000 - 0x9fffff are some more esoteric system calls
and if we subtract the old syscall offset of 0x900000 from this we get
our symptom of  0xf000* The extra __ARM_NR_* system call numbers run
from __NR_SYSCALL_BASE+__ARM_NR_BASE (0x900000 + 0xf0000) and are
defined in include/asm-arm/unistd.h as:
1   __ARM_NR_breakpoint
2   __ARM_NR_cacheflush
3   __ARM_NR_usr26
4   __ARM_NR_usr32
5   __ARM_NR_set_tls   <--- this one

This is glibc-2.4 using the internal ARM system call set_tls to
initialise(?) the thread local storage (something which the shims for
glibc-2.3.4 also evidently do). It is called from the macro:

glibc-2.4/ports/sysdeps/arm/{linuxthreads,nptl}/tls.h:
/* Code to initially initialize the thread pointer.  This might need
   special attention since 'errno' is not yet available and if the
   operation can cause a failure 'errno' must not be touched.  */
# define TLS_INIT_TP(TCBP, SECONDCALL) \
  ({ INTERNAL_SYSCALL_DECL (err); \
     long result_var; \
     result_var = INTERNAL_SYSCALL_ARM (set_tls, err, 1, (TCBP)); \
     INTERNAL_SYSCALL_ERROR_P (result_var, err) \
       ? "unknown error" : NULL; })

And this is what the Linux kernel does with that:

linux-2.6.17/arch/arm/kernel/traps.c:
        case NR(set_tls):
                thread->tp_value = regs->ARM_r0;
#if defined(CONFIG_HAS_TLS_REG)
                asm ("mcr p15, 0, %0, c13, c0, 3" : : "r" (regs->ARM_r0) );
#elif !defined(CONFIG_TLS_REG_EMUL)
                /*
                 * User space must never try to access this directly.
                 * Expect your app to break eventually if you do so.
                 * The user helper at 0xffff0fe0 must be used instead.
                 * (see entry-armv.S for details)
                 */
                *((unsigned int *)0xffff0ff0) = regs->ARM_r0;
#endif
                return 0;

Quite what that really means, and what QEMU should do about it, are
unclear to me. I've tried ignoring the f0005 system call, and the ARM
program segfaults straight away, so I guess *something* is necessary.
The 32-bit parameter ARM_r0 is always in the 0x7e* and 0x7f* range and
is different at each execution. Does this call set the area used for
thread-local storage?

Any ideas?

     M

[Qemu-devel] Re: qemu-arm user mode does not work with ARM EABI

[Martin Guy]

2006/9/26, Martin Guy <martinwguy@gmail.com>:
>   Ever since QEMU 0.8.1, qemu-arm-system mode works fine with ARM EABI
> kernel and rootfs, but qemu-arm single-process mode is broken with
> EABI, failing with "Unknown system call 0xf0005".

Fixed for simple cases by pbrook's preliminary patches mentioned at
http://lists.gnu.org/archive/html/qemu-devel/2006-09/msg00194.html

These patches do not apply cleanly using "patch" (it can't find most
of the files to patch), so I've repackaged them and put the
trouble-free version (cd qemu-0.8.2; patch -p1 < ../patch*mg1) under
http://freaknet.org/martin/QEMU

Bless

   M

Re: [Qemu-devel] qemu-arm user mode does not work with ARM EABI

[Rafael Espíndola]

> - Voipio's "eabi-rootfs" which I gather uses EABI internally and glibc
> shims to convert the system calls to old-ABI just before they hit the
> kernel (glibc-2.3.4)

I have a similar setup with a maemo 2.0 rootstrap (glibc 2.3.5). I am
currently able to compile a functional python package. I am not sure
if there is any particular patch applied to glibc, but you can find
the patches at http://repository.maemo.org/pool/maemo2.0/free/source/

Best Regards,
Rafael

Re: [Qemu-devel] qemu-arm user mode does not work with ARM EABI

[Paul Brook]

On Tuesday 26 September 2006 19:27, Martin Guy wrote:
> Summary:
>   Ever since QEMU 0.8.1, qemu-arm-system mode works fine with ARM EABI
> kernel and rootfs, but qemu-arm single-process mode is broken with
> EABI, failing with "Unknown system call 0xf0005".
>   This is ARM private system call "set_tls", called from both the
> linuxthreads and the NPTL pthread implementations in glibc-2.4, and
> unimplemented in QEMU.
>   It is important that this mode work because scratchbox needs it to
> be able to cross-compile Debian packages for the forthcoming
> EABI-based Debian architecture.

I've already answered this question three times in the last month...

> Fixed for simple cases by pbrook's preliminary patches mentioned at
> http://lists.gnu.org/archive/html/qemu-devel/2006-09/msg00194.html

... the answer is in that email.

> These patches do not apply cleanly using "patch" (it can't find most
> of the files to patch)

The original patch applies fine with patch -p0.

Paul

Re: [Qemu-devel] qemu-arm user mode

[K. Richard Pixley]

I have now managed to run a null program and a helloworld, (both eabi, 
linked statically, and without any thread calls), using the qemu-arm 
user mode, both inside and outside of scratchbox.  To do this with 
qemu-0.8.2 I needed the following:

1) Paul's patch for NLS.  
http://lists.gnu.org/archive/html/qemu-devel/2006-09/msg00194.html  Note 
that you will need to remove your object directory, reconfigure, and 
rebuild after patching.  The patch changes configuration data which will 
not be updated from a simple rebuild so you will need to remove your 
object and reconfigure before building again.  I didn't notice and this 
held me up for several days.

2) to build a qemu that can be debugged, you'll need to build it 
statically.  To do this, you can configure qemu using "--static".

3) qemu cannot be built with gcc-4, (due to a problem I don't 
understand), nor with gcc-3 using -O0, (due to an apparent collision 
between register allocation and inlining).  You'll need to use a gcc-3, 
with at least -O1 if you want to debug.  I changed the -O values by 
editing the Makefiles directly, but this change isn't really necessary 
if you don't plan to debug qemu.

4) Configure using "--enable-uname-revision=2.6.16".  Paul partially 
explained this recently.  Glibc, (the one your target program linked 
against when it was compiled), runs a check to see what kernel version 
it's running over.  When qemu-user emulates this call, by default, it 
asks the underlying host kernel what version the underlying host kernel 
is and then tells gdb that the emulated kernel version is the same.

While there's an argument that this is reasonable, in the case of arm 
eabi, it's a problem for many people.  Since eabi support was only added 
to the kernel in 2.6.16, earlier kernels can't handle eabi kernel 
calls.  Glibc knows this and will fail out if it believes itself to be 
running on an older kernel.  The fix, then, is to either run qemu-arm 
with the command line option, "-r 2.6.16" or to configure using 
"--enable-uname-revision=2.6.16" which will ask qemu to simply report 
these version numbers rather than querying the underlying host kernel.  
(Note: if your underlying (x86) host kernel happens to be 2.6.16 or 
later, then you probably don't need this workaround.)

None of these are new issues or fixes.  It just took me a few days to 
collect, run into them, and sort through them all.  So I'm posting my 
results here.

--rich

Re: [Qemu-devel] qemu-arm user mode

[Paul Brook]

> 1) Paul's patch for NLS.

You mean TLS and/or NPTL. NLS is something completely different :-)

Paul

Re: [Qemu-devel] qemu-arm user mode

[K. Richard Pixley]

Paul Brook wrote:
>> 1) Paul's patch for NLS.
>>     
> You mean TLS and/or NPTL. NLS is something completely different :-)
Bah.  Of course you're right.  And I'm sure you can explain your patch 
better than I can.

--rich