[Qemu-devel] forking i386 binaries on arm linux user mode

[Qemu-devel] forking i386 binaries on arm linux user mode

[Damion Yates]

I've grabbed the latest stable qemu and compiled under scratchbox.  I
hit an issue compiling it, with no __builtin__clear_cache() so linked in
a kludge.c containing a call to __clear_cache() with the params passed
as they would be to __builtin__clear_cache().

Firstly does this sound like it should work as a workaround?

It certainly got me to the next level, which is that I can now run loads
of linux binaries on my armlinux system (a Nokia n900). I've tried tower
toppler (http://toppler.sourceforge.net/) which uses SDL (via X11) and
this was surprisingly fast, in fact it almost felt as fast as the native
toppler that somebody crosscompiled already. Most linux utils work when
I copy then and any dependant libs from my x86 laptop to the phone. I'm
lucky (I guess) that /lib/ld-linux.so.3 is the arm version and I'm using
a slightly older .2 for x86 so I can have both files there. I also
enabled arbitrary execution of binaries via binfmt_misc. The 600 Mhz Arm
V8 Cortex (I think it is), feels like it's running at about Pentium 90
speeds, which I'm hoping is enough for what I really want to get going.

I want to run an old, possibly win16 Windows game under wine. I saw that
user mode qemu-i386 was able to run wine in a post in 2004:
http://lists.terrasoftsolutions.com/pipermail/yellowdog-general/2004-June/014468.html - This was on a PPC however.

When I run wine it SEGVs out and the strace of it shows it dies trying
to do clone(). I also can't run things like xterm which can't do fork().
Is this because by default it's trying to go via the arm "/bin/sh" to
invoke whatever it wants to exec() in to?

Should clone()/fork() work?  Has anyone been able to run wine ./blah.exe
under user-linux mode of qemu on arm or indeed any other non x86 based
CPU ?

Damion

Re: [Qemu-devel] forking i386 binaries on arm linux user mode

[Laurent Desnogues]

On Wed, Feb 10, 2010 at 10:38 PM, Damion Yates <damion@google.com> wrote:
> I've grabbed the latest stable qemu and compiled under scratchbox.  I
> hit an issue compiling it, with no __builtin__clear_cache() so linked in
> a kludge.c containing a call to __clear_cache() with the params passed
> as they would be to __builtin__clear_cache().
>
> Firstly does this sound like it should work as a workaround?

That's how I did it (except that I replaced the calls to
__builtin__clear_cache with calls to __clear_cache in
tcg/arm/tcg-target.h and exec-all.h.

> It certainly got me to the next level, which is that I can now run loads
> of linux binaries on my armlinux system (a Nokia n900). I've tried tower
> toppler (http://toppler.sourceforge.net/) which uses SDL (via X11) and
> this was surprisingly fast, in fact it almost felt as fast as the native
> toppler that somebody crosscompiled already. Most linux utils work when
> I copy then and any dependant libs from my x86 laptop to the phone. I'm
> lucky (I guess) that /lib/ld-linux.so.3 is the arm version and I'm using
> a slightly older .2 for x86 so I can have both files there. I also
> enabled arbitrary execution of binaries via binfmt_misc. The 600 Mhz Arm
> V8 Cortex (I think it is), feels like it's running at about Pentium 90
> speeds, which I'm hoping is enough for what I really want to get going.

The N900 has an ARMv7 Cortex-A8 (OMAP3).

P90 is about the speed I'd expect.  Note it will seem
much slower if your program makes heavy use of floating
point.

> I want to run an old, possibly win16 Windows game under wine. I saw that
> user mode qemu-i386 was able to run wine in a post in 2004:
> http://lists.terrasoftsolutions.com/pipermail/yellowdog-general/2004-June/014468.html - This was on a PPC however.
>
> When I run wine it SEGVs out and the strace of it shows it dies trying
> to do clone(). I also can't run things like xterm which can't do fork().
> Is this because by default it's trying to go via the arm "/bin/sh" to
> invoke whatever it wants to exec() in to?
>
> Should clone()/fork() work?  Has anyone been able to run wine ./blah.exe
> under user-linux mode of qemu on arm or indeed any other non x86 based
> CPU ?

I have seen a complex Linux i386 program run on an
ARM platform (with multithreading and Qt;  note it
crashed at some point but it was due to the
application doing access to uninitialized memory).
IIRC it was using chroot from inside QEMU (in main)
to make sure that all calls to exec would happen on
an x86 file system.  You might give that a try.


Laurent

Re: [Qemu-devel] forking i386 binaries on arm linux user mode

[Laurent Desnogues]

On Wed, Feb 10, 2010 at 10:38 PM, Damion Yates <damion@google.com> wrote:
[...]
> Should clone()/fork() work?  Has anyone been able to run wine ./blah.exe
> under user-linux mode of qemu on arm or indeed any other non x86 based
> CPU ?

I forgot to mention NPTL is not supported for x86 which
will be an issue.


Laurent

Re: [Qemu-devel] forking i386 binaries on arm linux user mode

[Damion Yates]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5099 bytes --]

On Wed, 10 Feb 2010, Laurent Desnogues wrote:

> On Wed, Feb 10, 2010 at 10:38 PM, Damion Yates <damion@google.com>
> wrote:
> 
> > I can now run loads of linux binaries on my armlinux system (a Nokia
> > n900). I've tried tower toppler (http://toppler.sourceforge.net/)
> > which uses SDL (via X11) and this was surprisingly fast, in fact it
> > almost felt as fast as the native toppler that somebody
> > crosscompiled already.

> > I want to run an old, possibly win16 Windows game under wine. I saw
> > that user mode qemu-i386 was able to run wine in a post in 2004:
> > http://lists.terrasoftsolutions.com/pipermail/yellowdog-general/2004-June/014468.html
> > - This was on a PPC however.

I've also since seen linked via http://wiki.winehq.org/ARM of others
doing this in 2006-09, although no details were posted other than
"installed their gnemul-x86 package. I then went and fetched Debian's
Wine and all its dependencies from their ARM distribution."

It doesn't make sense to use Debian's wine from their ARM dist, maybe he
meant just the wine from x86 circa 2006.

Also is there some magic in gnemul-x86 beyond being a set of x86 libs?
Does it do any shortcutting to system calls in native rather than
sticking with the libs under emulation more?

As I understand it qemu (in the arm compiled x86 user emulation mode I
have it) emulates an x86 to let my arm cpu start to execute x86 elf
binaries.  As soon as that tries to do any system calls, those are able
to run natively on arm, for things like opening files before returning a
pointer to the filehandle etc?  So it's sort of quite fast at times?

With special x86 libs it could also do some clever shortcuts getting in
to native mode sooner at some points?

> > When I run wine it SEGVs out and the strace of it shows it dies
> > trying to do clone(). I also can't run things like xterm which can't
> > do fork().  Is this because by default it's trying to go via the arm
> > "/bin/sh" to invoke whatever it wants to exec() in to?
> >
> > Should clone()/fork() work?  Has anyone been able to run wine
> > ./blah.exe under user-linux mode of qemu on arm or indeed any other
> > non x86 based CPU ?
> 
[added from other post]
> NPTL is not supported for x86 which will be an issue.

This could be quite problematic, as I believe all recent wine versions
use this.  My running of older versions of wine is likely to lead to a
lack of support when I approach the wine team.

> I have seen a complex Linux i386 program run on an ARM platform (with
> multithreading and Qt;  note it crashed at some point but it was due
> to the application doing access to uninitialized memory).  IIRC it was
> using chroot from inside QEMU (in main) to make sure that all calls to
> exec would happen on an x86 file system.  You might give that a try.

Could you explain what you did?  I've tried the following:

ch/ is a small x86 base image (color.gz from slackware).

chroot ch/ ./ld-linux-arm.so --library-path `pwd`/armlibs ./qemu-i386 bin/sh

This works, I'm sat in an x86 bash binary with / and upwards being
mostluy x86 (except ./ld-linux-arm.so, ./qemu-i386 and the armlibs/
directory).  I can run the x86 stuff, with ld-linux.so and qemu but I
can't just do /bin/ls.

I could use statically linked /bin/bash-static on #! path wrappers which
spawn qemu (via an arm ld-linux.so) to fake /bin/sh as an executable,
which would function and would be x86, but I'm confused as to any gain.
Basically I don't think a chroot is the issue.  So...

I tried using the versions of libs and wine from the 2004 post of the
successful PPC wine execution and now get:

~/winefiles $ ../qemu-i386 -L /home/user/winefiles/l lib/ld-2.3.2.so usr/bin/wineserver
~/winefiles $ ../qemu-i386 -L /home/user/winefiles/l lib/ld-2.3.2.so usr/bin/wine-pthread notepad
Could not stat /mnt/floppy (No such file or directory), ignoring drive A: err:dosmem:setup_dos_mem Cannot use first megabyte for DOS address space, please report
err:heap:HEAP_CreateSystemHeap system heap base address 0x65430000 not available

Segmentation fault

~/winefiles $ ../qemu-i386 -L /home/user/winefiles/l lib/ld-2.3.2.so usr/bin/wineserver
~/winefiles $ ../qemu-i386 -L /home/user/winefiles/l lib/ld-2.3.2.so usr/bin/wine-pthread pscp.exe
Could not stat /mnt/floppy (No such file or directory), ignoring drive A:
err:virtual:map_image Standard load address for a Win32 program
(0x00400000) not available - security-patched kernel ?
wine: could not load L"C:\\windows\\pscp.exe" as Win32 binary
~/winefiles $

I need help from a qemu/wine/armlinux/kernel expert to continue beyond
this.

Oh I should note that I echoed 0 to /proc/sys/vm/max_map_count which is
suggested on the wine forums as the fix for the security-patched kernel
? error.  But this didn't work.

Any ideas?  Should I direct this to wine ?  Is this something anyone is
interested in helping me get working?

Damion

-- 
Damion Yates - Google UK Ltd
Belgrave House, 76 Buckingham Palace Rd, London SW1W 9TQ - reg:3977902

Re: [Qemu-devel] forking i386 binaries on arm linux user mode

[Richard Henderson]

On 05/11/2010 10:08 AM, Damion Yates wrote:
> Also is there some magic in gnemul-x86 beyond being a set of x86 libs?

No.

> Does it do any shortcutting to system calls in native rather than
> sticking with the libs under emulation more?

No.

> Could you explain what you did?  I've tried the following:

Use the binfmt_misc filesystem to set up qemu as the interpreter
for x86 binaries.

> chroot ch/ ./ld-linux-arm.so --library-path `pwd`/armlibs ./qemu-i386 bin/sh

You may find it easier to staticly link the arm qemu in that case.


r~

Re: [Qemu-devel] forking i386 binaries on arm linux user mode

[Damion Yates]

On Tue, 11 May 2010, Richard Henderson wrote:

> On 05/11/2010 10:08 AM, Damion Yates wrote:
> > Also is there some magic in gnemul-x86 beyond being a set of x86
> > libs?
> 
> No.

Okay, good to know.
 
> > Does it do any shortcutting to system calls in native rather than
> > sticking with the libs under emulation more?
> 
> No.

Okay, fine.  I expect this has been thought of (or dismissed as not
feasible) on the qemu mailing list over the years anyway.
 
> > Could you explain what you did?  I've tried the following:
> 
> Use the binfmt_misc filesystem to set up qemu as the interpreter for
> x86 binaries.

Did that (mentioned in the original email).  I think what I must have
missed was making sure the qemu binary was found in matching paths in
the chroot (in fact it's probably not needed outside the chroot really).

I'm not quite sure how I missed that, I know what I'm doing but I think
I just lacked faith in whether this could work at all on a fundamental
level.  Of course that was since confirmed on this list, so I just
needed to try again harder.
 
> > chroot ch/ ./ld-linux-arm.so --library-path `pwd`/armlibs \
> > ./qemu-i386 bin/sh
> 
> You may find it easier to staticly link the arm qemu in that case.

Yeah, I figured this was going to be the only way, it was that or the
interpreter after the #! within any wrapper to run stuff.  As I didn't
even have a static sh (arm) around, I rechecked the build of qemu and
found it trivial to statically compile*.

I've now got it working as planned, I can chroot in, I've bind mounted
/tmp/.X11-unix/ in to the chroot so X works (I think my phone has
X with -nolisten tcp and changing that is phone-brickingly scary).

So yeah... everything sort of works.  I can run arbitrary (fairly)
complex binaries like screen, xterm (both need to fork), graphical stuff
like XV and of course wine sol.exe (which I think does some threading)!

http://www.youtube.com/damionyates2

Thanks for all your help.

I still have some issues, surprisingly a few reboots, which is odd as
I'm su'ed to a user after needing root for the chroot, I suspect
something I'm mapping with /dev/ or /proc/ mounts.  I've found although
a lot of stuff doesn't work if it needs those mounts working, my tests
are more reliable for the binaries that don't need /proc or /dev being
there, if I don't mount /proc or bindmount /dev/pts at all.  It's also
not quite doing what I want in terms of wine capabilities.  I suspect
the revisions of x86 libs and wine version, possibly NPTL.  But I think
the wine mailing list is the next best stop for me for that unless any
of you are particularly interested in this?

The other threads on this list are all developery with loads of patches
and stuff so I'll stop annoying you all now and close this thread as a
success :)

Damion

*This is increasingly hard, I think many modern GNU projects have given
up providing a way to statically link stuff, and so many things pull in
via dlopen now you can't even tell from an ldd whether it's likely to
work.

-- 
Damion Yates - Google UK Ltd
Belgrave House, 76 Buckingham Palace Rd, London SW1W 9TQ - reg:3977902