[Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Nathan Froyd]

This patch series adds NPTL support in Linux user-mode emulation to
32-bit PowerPC targets.

The main complication comes from implementing atomic instructions
properly.  We chose to implement a simplistic model:

- reserved loads record the value loaded;

- conditional stores check that the memory at the effective address
  contains the value loaded by the previous reserved load, in addition
  to all other checks.  if so, the store succeeds; otherwise, it fails.

It is possible to implement something more sophisticated using mprotect:

- reserved loads write-protect the page from which the value is loaded;

- regular stores to the page (through SIGSEGV handling) remove the write
  protection (which is roughly how the architecture really works);

- conditional stores fail if the page was not write-protected, in
  addition to all other checks.  If the store succeeds, then the page is
  unprotected.

but the simple scheme works well enough and should be somewhat faster.
The simple scheme is what's already done for system mode, too; it's even
slightly dumber in system mode because we don't check for equality of
values.

The patch series has been tested against the glibc testsuite, where it
passes a good chunk (90%+) of the testsuite.  The other 10% are
basically things that are not going to work in QEMU anytime soon
(e.g. sharing futexes between multiple processes, using clone(2)
directly, etc.).  I should note that proper testing requires a patch to
use a correct exit status for uncaught signals; such a patch has been
posted to this list before by Riku Voipio.  (I have a different local
version that I used instead.)  Testing with recent glibc also requires
adding support for private futexes and a few other futex operations;
again, a patch for this has been posted by Riku and I used a slightly
different local version.

-Nathan

Nathan Froyd (7):
  linux-user: initialize mmap_mutex properly
  target-ppc: fix cpu_clone_regs
  target-ppc: add cpu_set_tls
  target-ppc: retain l{w,d}arx loaded value
  target-ppc: add exceptions for conditional stores
  linux-user: handle POWERPC_EXCP_STCX
  enable NPTL for ppc-linux-user targets in configure

 configure              |    2 +
 linux-user/main.c      |   68 ++++++++++++++++++++++++++++++++++++++
 linux-user/mmap.c      |    2 +-
 target-ppc/cpu.h       |   24 +++++++++++--
 target-ppc/helper.c    |    2 +-
 target-ppc/machine.c   |    4 +-
 target-ppc/op_helper.c |    4 +-
 target-ppc/translate.c |   84 ++++++++++++++++++++++++++++++++++--------------
 8 files changed, 156 insertions(+), 34 deletions(-)

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[malc]

On Thu, 4 Jun 2009, Nathan Froyd wrote:

> This patch series adds NPTL support in Linux user-mode emulation to
> 32-bit PowerPC targets.
> 
> The main complication comes from implementing atomic instructions
> properly.  We chose to implement a simplistic model:
> 
> - reserved loads record the value loaded;
> 
> - conditional stores check that the memory at the effective address
>   contains the value loaded by the previous reserved load, in addition
>   to all other checks.  if so, the store succeeds; otherwise, it fails.

I think this will break code that relies on the fact that ll/sc is not
affected by the ABA problem.

> 
> It is possible to implement something more sophisticated using mprotect:
> 
> - reserved loads write-protect the page from which the value is loaded;
> 
> - regular stores to the page (through SIGSEGV handling) remove the write
>   protection (which is roughly how the architecture really works);
> 
> - conditional stores fail if the page was not write-protected, in
>   addition to all other checks.  If the store succeeds, then the page is
>   unprotected.
> 
> but the simple scheme works well enough and should be somewhat faster.
> The simple scheme is what's already done for system mode, too; it's even
> slightly dumber in system mode because we don't check for equality of
> values.

[..snip..]

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Paul Brook]

On Saturday 06 June 2009, malc wrote:
> On Thu, 4 Jun 2009, Nathan Froyd wrote:
> > This patch series adds NPTL support in Linux user-mode emulation to
> > 32-bit PowerPC targets.
> >
> > The main complication comes from implementing atomic instructions
> > properly.  We chose to implement a simplistic model:
> >
> > - reserved loads record the value loaded;

An important point here is that the address/value pair is per thread/cpu.
A nice side-effect is that these loads reduce to a simple atomic load and some 
thread local bookkeeping. Conditional stores require somewhat more exotic 
atomic operations, but still don't need to go poking at system global state or 
other CPUs.
This sounds strange at first reading, but ll/sc semantics are deliberately 
designed to minimize contention between unrelated CPUs/resources in large 
systems.

> > - conditional stores check that the memory at the effective address
> >   contains the value loaded by the previous reserved load, in addition
> >   to all other checks.  if so, the store succeeds; otherwise, it fails.
>
> I think this will break code that relies on the fact that ll/sc is not
> affected by the ABA problem.

I'm not absolutely certain about PPC, but on other architectures (ARM, MIPS, 
Alpha) this implementation is sufficient.

The only questionable case is when a second thread overwrites and then 
restores the original value between a locked load and a conditional store. 
However limited coherency and memory ordering between CPUs make it impossible 
to know whether this modify+restore occurred before or after the initial load.

The worst that can happen here is that another thread gains and releases the 
lock[1] while the current thread is in the process of acquiring the lock.  
Even when this happens it is impossible for two threads to acquire the lock 
simultaneously. The only difference is the window between ll and sc. During 
this period we don't know whether we have the lock or not, so it's extremely 
unlikely that we will do anything that relies on no other thread having the 
lock. In practice ll/sc are always used as matching pairs with no intervening 
memory accesses, so this is never a problem.

I could probably come up with synthetic testcases where qemu behavior is 
observably different to real hardware. However I'm pretty certain this never 
occurs in real code, and it is questionable whether such behavior is 
architecturally defined.

If you still believe this is a problem you need come up with an actual 
testcase that demonstrates how this can introduce a race condition.

Paul

[1] Lock acquisition is the most obvious example, but the same applies to any 
atomic operation implemented on top of ll/sc. 

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

On Thu, Jun 04, 2009 at 11:51:55AM -0700, Nathan Froyd <froydnj@codesourcery.com> wrote:
> The patch series has been tested against the glibc testsuite, where it
> passes a good chunk (90%+) of the testsuite.  The other 10% are
> basically things that are not going to work in QEMU anytime soon
> (e.g. sharing futexes between multiple processes, using clone(2)
> directly, etc.).  I should note that proper testing requires a patch to
> use a correct exit status for uncaught signals; such a patch has been
> posted to this list before by Riku Voipio.  (I have a different local
> version that I used instead.)  Testing with recent glibc also requires
> adding support for private futexes and a few other futex operations;
> again, a patch for this has been posted by Riku and I used a slightly
> different local version.

Hi,

I tried this series (on top of current git 3a41759) + Riku's patch. I
built a static qemu-ppc binary, then tried:

host# chroot . /qemu-ppc -L . /bin/bash

Where the current directory was a PPC chroot. It launched bash just
fine, but when I tried to launch a command from the ppc bash, I got:

chroot# /bin/uname
bash: /bin/uname: No such file or directory

While the binary is there:

host# file bin/uname
bin/uname: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
with unknown capability 0x41000000 = 0x11676e75, with unknown capability
0x10000 = 0x90401, stripped

Let me know if you need more info to reproduce the issue.

Thanks.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Daniel Jacobowitz]

On Wed, Jun 10, 2009 at 01:37:45AM +0200, Miklos Vajna wrote:
> On Thu, Jun 04, 2009 at 11:51:55AM -0700, Nathan Froyd <froydnj@codesourcery.com> wrote:
> > The patch series has been tested against the glibc testsuite, where it
> > passes a good chunk (90%+) of the testsuite.  The other 10% are
> > basically things that are not going to work in QEMU anytime soon
> > (e.g. sharing futexes between multiple processes, using clone(2)
> > directly, etc.).  I should note that proper testing requires a patch to
> > use a correct exit status for uncaught signals; such a patch has been
> > posted to this list before by Riku Voipio.  (I have a different local
> > version that I used instead.)  Testing with recent glibc also requires
> > adding support for private futexes and a few other futex operations;
> > again, a patch for this has been posted by Riku and I used a slightly
> > different local version.
> 
> Hi,
> 
> I tried this series (on top of current git 3a41759) + Riku's patch. I
> built a static qemu-ppc binary, then tried:
> 
> host# chroot . /qemu-ppc -L . /bin/bash
> 
> Where the current directory was a PPC chroot. It launched bash just
> fine, but when I tried to launch a command from the ppc bash, I got:
> 
> chroot# /bin/uname
> bash: /bin/uname: No such file or directory

This isn't expected to work, is it?  QEMU does not intercept exec
system calls for target binaries.

[Maybe it should...]

-- 
Daniel Jacobowitz
CodeSourcery

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Tue, Jun 09, 2009 at 08:24:47PM -0400, Daniel Jacobowitz <drow@false.org> wrote:
> > host# chroot . /qemu-ppc -L . /bin/bash
> > 
> > Where the current directory was a PPC chroot. It launched bash just
> > fine, but when I tried to launch a command from the ppc bash, I got:
> > 
> > chroot# /bin/uname
> > bash: /bin/uname: No such file or directory
> 
> This isn't expected to work, is it?  QEMU does not intercept exec
> system calls for target binaries.
> 
> [Maybe it should...]

Hmm. According to this mail it used to work in the non-NPTL case:

http://osdir.com/ml/emulators.qemu/2004-02/msg00162.html

Or have I missed something?

Thanks.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Laurent Vivier]

Le mercredi 10 juin 2009 à 01:37 +0200, Miklos Vajna a écrit :
> On Thu, Jun 04, 2009 at 11:51:55AM -0700, Nathan Froyd <froydnj@codesourcery.com> wrote:
> > The patch series has been tested against the glibc testsuite, where it
> > passes a good chunk (90%+) of the testsuite.  The other 10% are
> > basically things that are not going to work in QEMU anytime soon
> > (e.g. sharing futexes between multiple processes, using clone(2)
> > directly, etc.).  I should note that proper testing requires a patch to
> > use a correct exit status for uncaught signals; such a patch has been
> > posted to this list before by Riku Voipio.  (I have a different local
> > version that I used instead.)  Testing with recent glibc also requires
> > adding support for private futexes and a few other futex operations;
> > again, a patch for this has been posted by Riku and I used a slightly
> > different local version.
> 
> Hi,
> 
> I tried this series (on top of current git 3a41759) + Riku's patch. I
> built a static qemu-ppc binary, then tried:
> 
> host# chroot . /qemu-ppc -L . /bin/bash
> 
> Where the current directory was a PPC chroot. It launched bash just
> fine, but when I tried to launch a command from the ppc bash, I got:
> 
> chroot# /bin/uname
> bash: /bin/uname: No such file or directory

Did you configure binfmt_misc kernel module to load PPC binaries with
qemu-ppc ?

Regards,
Laurent

> While the binary is there:
> 
> host# file bin/uname
> bin/uname: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> with unknown capability 0x41000000 = 0x11676e75, with unknown capability
> 0x10000 = 0x90401, stripped
> 
> Let me know if you need more info to reproduce the issue.
> 
> Thanks.
-- 
--------------------- laurent@vivier.eu ----------------------
"Tout ce qui est impossible reste à accomplir"    Jules Verne
"Things are only impossible until they're not" Jean-Luc Picard

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Daniel Jacobowitz]

On Wed, Jun 10, 2009 at 02:30:48AM +0200, Miklos Vajna wrote:
> On Tue, Jun 09, 2009 at 08:24:47PM -0400, Daniel Jacobowitz <drow@false.org> wrote:
> > > host# chroot . /qemu-ppc -L . /bin/bash
> > > 
> > > Where the current directory was a PPC chroot. It launched bash just
> > > fine, but when I tried to launch a command from the ppc bash, I got:
> > > 
> > > chroot# /bin/uname
> > > bash: /bin/uname: No such file or directory
> > 
> > This isn't expected to work, is it?  QEMU does not intercept exec
> > system calls for target binaries.
> > 
> > [Maybe it should...]
> 
> Hmm. According to this mail it used to work in the non-NPTL case:
> 
> http://osdir.com/ml/emulators.qemu/2004-02/msg00162.html
> 
> Or have I missed something?

Oh, are you using binfmt_misc?  That's different...

-- 
Daniel Jacobowitz
CodeSourcery

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On Wed, Jun 10, 2009 at 02:57:08AM +0200, Laurent Vivier <Laurent@vivier.eu> wrote:
> > chroot# /bin/uname
> > bash: /bin/uname: No such file or directory
> 
> Did you configure binfmt_misc kernel module to load PPC binaries with
> qemu-ppc ?

Yes:

$ cat /proc/sys/fs/binfmt_misc/qemu-ppc
enabled
interpreter /usr/bin/qemu-ppc
flags:
offset 0
magic 7f454c4601020100000000000000000000020014
mask fffffffffffffffffffffffffffffffffbffffff

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

On Tue, Jun 09, 2009 at 10:49:55PM -0400, Daniel Jacobowitz <drow@false.org> wrote:
> > Hmm. According to this mail it used to work in the non-NPTL case:
> > 
> > http://osdir.com/ml/emulators.qemu/2004-02/msg00162.html
> > 
> > Or have I missed something?
> 
> Oh, are you using binfmt_misc?  That's different...

Sorry, I forgot to mention it. I did a

echo ":qemu-ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff\xff:/usr/bin/qemu-ppc:" > /proc/sys/fs/binfmt_misc/register

And the a test binary using fork() works fine as well:

$ file hello
hello: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.3, not
stripped

$ ./hello
pid is 0
pid is 14461

Thanks.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Laurent Vivier]

>On Wed, Jun 10, 2009 at 02:57:08AM +0200, Laurent Vivier <Laurent@vivier.eu> 
>wrote:
>> > chroot# /bin/uname
>> > bash: /bin/uname: No such file or directory
>> 
>> Did you configure binfmt_misc kernel module to load PPC binaries with
>> qemu-ppc ?
>
>Yes:
>
>$ cat /proc/sys/fs/binfmt_misc/qemu-ppc
>enabled
>interpreter /usr/bin/qemu-ppc

so qemu must be located at /usr/bin/ not / under your chrooted filesystem.

>flags:
>offset 0
>magic 7f454c4601020100000000000000000000020014
>mask fffffffffffffffffffffffffffffffffbffffff

Laurent
-- 
--------------------- Laurent@vivier.eu  ---------------------
"Tout ce qui est impossible reste à accomplir"    Jules Verne
"Things are only impossible until they're not" Jean-Luc Picard

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On Wed, Jun 10, 2009 at 11:36:17AM +0200, Laurent Vivier <laurent@vivier.eu> wrote:
> >$ cat /proc/sys/fs/binfmt_misc/qemu-ppc
> >enabled
> >interpreter /usr/bin/qemu-ppc
> 
> so qemu must be located at /usr/bin/ not / under your chrooted filesystem.

Ah, wow.

So it was a bug on my side, now it works!

OTOH, there is an other odd error. Here is a sample test program:

#include <stdio.h>
#include <sys/stat.h>

int main()
{
        struct stat buf;
        if (stat("usr/bin/ls", &buf)) {
                printf("not found\n");
        } else {
                printf("found\n");
        }
        return 0;
}

host$ ppc-frugalware-linux-gcc hello2.c -o hello2
host$ file hello2
hello2: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.3, not
stripped
host$ ./hello2
found
host$ sudo chroot .
chroot:/# ./hello2
not found
chroot:/# grep usr hello2.c
        if (stat("usr/bin/ls", &buf)) {
chroot:/# ls usr/bin/ls
usr/bin/ls

Is it a bug that stat() does not found what ls does, or have I missed
something? :)

Thanks.

PS: Just in case somebody wants to play with it, here is the PPC chroot
tarball I'm using:

http://ftp.frugalware.org/pub/frugalware/frugalware-1.0-iso/fwchroot-1.0-ppc.tar.bz2

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Laurent Vivier]

>On Tue, Jun 09, 2009 at 10:49:55PM -0400, Daniel Jacobowitz <drow@false.org> 
>wrote:
>> > Hmm. According to this mail it used to work in the non-NPTL case:
>> > 
>> > http://osdir.com/ml/emulators.qemu/2004-02/msg00162.html
>> > 
>> > Or have I missed something?
>> 
>> Oh, are you using binfmt_misc?  That's different...
>
>Sorry, I forgot to mention it. I did a
>
>echo 
>":qemu-ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x
>00\x14:\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff
>\xff\xff:/usr/bin/qemu-ppc:" > /proc/sys/fs/binfmt_misc/register
>
>And the a test binary using fork() works fine as well:
[...]

The problem is not with fork() but with exec().

Laurent
-- 
--------------------- Laurent@vivier.eu  ---------------------
"Tout ce qui est impossible reste à accomplir"    Jules Verne
"Things are only impossible until they're not" Jean-Luc Picard

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

On Wed, Jun 10, 2009 at 12:44:16PM +0200, Laurent Vivier <laurent@vivier.eu> wrote:
> >And the a test binary using fork() works fine as well:
> [...]
> 
> The problem is not with fork() but with exec().

Yes, but as I showed in the other mail stat() has some weird behaviour
as well. I just noticed this all when I wanted to see if qemu-ppc can
execute a native gcc, where it failed to find cc1, which was - of course
- there.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Laurent Vivier]

>On Wed, Jun 10, 2009 at 11:36:17AM +0200, Laurent Vivier <laurent@vivier.eu> 
>wrote:
>> >$ cat /proc/sys/fs/binfmt_misc/qemu-ppc
>> >enabled
>> >interpreter /usr/bin/qemu-ppc
>> 
>> so qemu must be located at /usr/bin/ not / under your chrooted filesystem.
>
>Ah, wow.
>
>So it was a bug on my side, now it works!
>
>OTOH, there is an other odd error. Here is a sample test program:
>
>#include <stdio.h>
>#include <sys/stat.h>
>
>int main()
>{
>        struct stat buf;
>        if (stat("usr/bin/ls", &buf)) {
>                printf("not found\n");
>        } else {
>                printf("found\n");
>        }
>        return 0;
>}
>
>host$ ppc-frugalware-linux-gcc hello2.c -o hello2
>host$ file hello2
>hello2: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1
>(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.4.3, not
>stripped
>host$ ./hello2
>found
>host$ sudo chroot .
>chroot:/# ./hello2
>not found
>chroot:/# grep usr hello2.c
>        if (stat("usr/bin/ls", &buf)) {
>chroot:/# ls usr/bin/ls
>usr/bin/ls
>
>Is it a bug that stat() does not found what ls does, or have I missed
>something? :)
>

IMHO, it looks like a bug, you should display errno, try also "ls -l".

Laurent
-- 
--------------------- Laurent@vivier.eu  ---------------------
"Tout ce qui est impossible reste à accomplir"    Jules Verne
"Things are only impossible until they're not" Jean-Luc Picard

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

On Wed, Jun 10, 2009 at 12:54:38PM +0200, Laurent Vivier <laurent@vivier.eu> wrote:
> >Is it a bug that stat() does not found what ls does, or have I missed
> >something? :)
> >
> 
> IMHO, it looks like a bug, you should display errno, try also "ls -l".

chroot:/# ls -l usr/bin/ls
lrwxrwxrwx 1 vmiklos users 7 Jun 10 01:17 usr/bin/ls -> /bin/ls*

chroot:/# ./hello2
not found: Value too large for defined data type

Thanks.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Qemu-devel] [PATCH 0/7] target-ppc/linux-user: NPTL support

[Nathan Froyd]

On Sat, Jun 06, 2009 at 03:04:38AM +0400, malc wrote:
> On Thu, 4 Jun 2009, Nathan Froyd wrote:
> > The main complication comes from implementing atomic instructions
> > properly.  We chose to implement a simplistic model:
> > 
> > - reserved loads record the value loaded;
> > 
> > - conditional stores check that the memory at the effective address
> >   contains the value loaded by the previous reserved load, in addition
> >   to all other checks.  if so, the store succeeds; otherwise, it fails.
> 
> I think this will break code that relies on the fact that ll/sc is not
> affected by the ABA problem.
>
> > It is possible to implement something more sophisticated using mprotect:

The simpler scheme is slightly faster than mprotect, but not
astonishingly so.  In practice, it works out OK, even if it doesn't
adhere to a particular interpretation of the chip docs.

I'm ambivalent about which scheme goes in: I just would like to see
something go in so atomic instructions/user-mode emulation threading can
be properly supported on PPC (and other architectures to follow, I'm
sure...).

-Nathan