[Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[John Paul Adrian Glaubitz]

Hi Rasmus!

Your recent commit "linux: spawni.c: simplify error reporting to parent" apparently
broke both qemu-user and Microsoft's Windows Services for Linux which both fail
with:

dpkg: warning: ignoring pre-dependency problem!
Preparing to unpack .../archives/bash_4.4-5_m68k.deb ...
preinst: ../sysdeps/unix/sysv/linux/spawni.c:366: __spawnix: Assertion `ec >= 0' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
dpkg: error processing archive /var/cache/apt/archives/bash_4.4-5_m68k.deb (--unpack):
 new bash package pre-installation script subprocess was killed by signal (Aborted)
Selecting previously unselected package bsdutils.
dpkg: regarding .../bsdutils_1%3a2.30.2-0.1_m68k.deb containing bsdutils, pre-dependency problem:
 bsdutils pre-depends on libsystemd0
  libsystemd0 is not installed.

I'm not sure yet what the actual problem is but I thought it should be necessary
to point you at the problem.

Cheers,
Adrian

> [1] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=4b4d4056bb154603f36c6f8845757c1012758158;hp=8d3bd947483f50b57aee7c35c07dc1927d6e8a27
> [2] https://github.com/Microsoft/WSL/issues/1878

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[John Paul Adrian Glaubitz]

On 11/26/2017 09:28 PM, John Paul Adrian Glaubitz wrote:
> I'm not sure yet what the actual problem is but I thought it should be necessary
> to point you at the problem.

Ok, there is already a QEMU bug report for this [1].

Adrian

> [1] https://bugs.launchpad.net/qemu/+bug/1673976

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[Adhemerval Zanella]

On 26/11/2017 18:35, John Paul Adrian Glaubitz wrote:
> On 11/26/2017 09:28 PM, John Paul Adrian Glaubitz wrote:
>> I'm not sure yet what the actual problem is but I thought it should be necessary
>> to point you at the problem.
> 
> Ok, there is already a QEMU bug report for this [1].
> 
> Adrian
> 
>> [1] https://bugs.launchpad.net/qemu/+bug/1673976
> 

We found out this potential bogus assert on 2.27 development [1] which
resulted in two fixes [2][3].

It should not be an issue for generic posix_spawn usage where there is
no expectation system/user/program kills random pids (since posix_spawn
auxiliary process has not yet returned). Some say the possible kind of
behaviour is rather undefined, but it shouldn't also trigger an assert.

I am not really sure what is happening in qemu usermode because comment
#4 in the bug reports states clone is returning an error and it should
not trigger the assert in first place.  What seems to be happening in
this scenario is clone is actually returning a success, but the auxiliary
process is being killed before actually calling execve.

In any case I think both fixes I pointed out (which are also on 2.26 and
2.25) should avoid the assert issue. Regarding the CLONE_VFORK support
I think we will need to address it on qemu.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=22273
[2] https://sourceware.org/git/?p=glibc.git;a=commit;h=fe05e1cb6d64dba6172249c79526f1e9af8f2bfd
[3] https://sourceware.org/git/?p=glibc.git;a=commit;h=aa95a2414e4f664ca740ad5f4a72d9145abbd426

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[Peter Maydell]

On 27 November 2017 at 12:57, Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
> We found out this potential bogus assert on 2.27 development [1] which
> resulted in two fixes [2][3].
>
> It should not be an issue for generic posix_spawn usage where there is
> no expectation system/user/program kills random pids (since posix_spawn
> auxiliary process has not yet returned). Some say the possible kind of
> behaviour is rather undefined, but it shouldn't also trigger an assert.
>
> I am not really sure what is happening in qemu usermode because comment
> #4 in the bug reports states clone is returning an error and it should
> not trigger the assert in first place.  What seems to be happening in
> this scenario is clone is actually returning a success, but the auxiliary
> process is being killed before actually calling execve.

The bug report is a bit confused, but I think what is happening
in the QEMU case is that QEMU implements clone(CLONE_VFORK) as having
the same semantics as fork() (ie the parent will not autowait for
the child, and the child does not share a memory map with the parent).
(ie QEMU treats it as having the semantics of a vfork() call, which
is allowed to be implemented as fork()).
Previous versions of glibc's posix_spawn() could cope with this
divergence from the kernel's native clone() behaviour, but the
rewrite can't. It's not unreasonable for glibc() to rely on the
kernel behaviour, but on the other hand it's not too surprising
if this breaks non-kernel implementations of the syscall ABI
like QEMU and the MS Linux subsystem, because it's a tricky
corner case that previously nobody was trying to use.

Unfortunately I can't really think of a mechanism for implementing
this in QEMU usermode, because the only tools we have available
for creating new threads and processes are the ones the host libc
gives us: so we can spawn new threads with pthread_create() and
fork the process with fork(), but we don't have a safe way to
create a new process which shares the memory map and where the
new process can call the various libc functions which QEMU will
do as it executes the guest code.

thanks
-- PMM

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[Adhemerval Zanella]

On 27/11/2017 13:24, Peter Maydell wrote:
> On 27 November 2017 at 12:57, Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
>> We found out this potential bogus assert on 2.27 development [1] which
>> resulted in two fixes [2][3].
>>
>> It should not be an issue for generic posix_spawn usage where there is
>> no expectation system/user/program kills random pids (since posix_spawn
>> auxiliary process has not yet returned). Some say the possible kind of
>> behaviour is rather undefined, but it shouldn't also trigger an assert.
>>
>> I am not really sure what is happening in qemu usermode because comment
>> #4 in the bug reports states clone is returning an error and it should
>> not trigger the assert in first place.  What seems to be happening in
>> this scenario is clone is actually returning a success, but the auxiliary
>> process is being killed before actually calling execve.
> 
> The bug report is a bit confused, but I think what is happening
> in the QEMU case is that QEMU implements clone(CLONE_VFORK) as having
> the same semantics as fork() (ie the parent will not autowait for
> the child, and the child does not share a memory map with the parent).
> (ie QEMU treats it as having the semantics of a vfork() call, which
> is allowed to be implemented as fork()).

Right, that explains what is happening. 

> Previous versions of glibc's posix_spawn() could cope with this
> divergence from the kernel's native clone() behaviour, but the
> rewrite can't. It's not unreasonable for glibc() to rely on the
> kernel behaviour, but on the other hand it's not too surprising
> if this breaks non-kernel implementations of the syscall ABI
> like QEMU and the MS Linux subsystem, because it's a tricky
> corner case that previously nobody was trying to use.

The problem is vfork is such a broken API [1] that even POSIX has 
deprecated it on the latest 2008 standard. It was used on GLIBC
posix_spawn on some specific usage (old POSIX_SPAWN_USEVFORK 
flag) only because it was 'faster' than using fork, however
it also created its own set bugs [2][3][4][5].

Current implementation is as fast as using vfork on Linux using
which should be platform neutral clone flags and assumptions
(in fact we found out that Linux does not work as expected with
clone (CLONE_VFORK | CLONE_VM) -> exit -> waitpid (WNOHANG)
which resulted in aa95a2414).

GLIBC also maintains another implementation at 
sysdeps/posix/spawni.c which should be more platform neutral
since it uses only POSIX expected semantics (the synchronization
is done using a pipe2 instead of CLONE_VM, so a vfork acting
as fork shouldn't be a problem). It is not used in any architecture
on GLIBC currently.

However I am not very compelled to change internal posix_spawn
on GLIBC on Linux mainly because it uses a slight less resources
than the generic POSIX one (check e83be730910c) and it works
on Linux kernel as expected.

[1] https://ewontfix.com/7/
[2] https://sourceware.org/bugzilla/show_bug.cgi?id=14750
[3] https://sourceware.org/bugzilla/show_bug.cgi?id=14749
[4] https://sourceware.org/bugzilla/show_bug.cgi?id=14499
[5] https://sourceware.org/bugzilla/show_bug.cgi?id=10354

> 
> Unfortunately I can't really think of a mechanism for implementing
> this in QEMU usermode, because the only tools we have available
> for creating new threads and processes are the ones the host libc
> gives us: so we can spawn new threads with pthread_create() and
> fork the process with fork(), but we don't have a safe way to
> create a new process which shares the memory map and where the
> new process can call the various libc functions which QEMU will
> do as it executes the guest code.

Current GLIBC won't trigger any assert anymore (and it was backported
to 2.25 and 2.26 branch as well), however I am not sure if posix_spawn 
semantic will works for all the expected scenarios in qemu user-mode.

Most likely any failure (sched_set{param,scheduler}, setsid, setpgid,
seteuid, any file action or execve itself) won't be advertise to main
process, since err is set 0 as default and the auxiliary process will
write to a expected shared memory to signalling an issue. 

Also, I don't think trying to emulate "CLONE_VM | CLONE_VFORK" with
pthread_create without actually synchronize the threads will work
as expected. If clone actually uses CLONE_VFORK I would expect the
underlying qemu usermode to block the caller thread (using a condition
variable or a barrier) and to release its execution only for execve
or exit in the callee. I am not very versed on qemu code, so I 
am not sure how complex it would be.

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[John Paul Adrian Glaubitz]

On 11/27/2017 05:07 PM, Adhemerval Zanella wrote:> However I am not very compelled to change internal posix_spawn
> on GLIBC on Linux mainly because it uses a slight less resources
> than the generic POSIX one (check e83be730910c) and it works
> on Linux kernel as expected.

But it breaks QEMU and Microsoft Windows Services for Linux who -
combined together - are certainly not a small number of users.

Isn't there any workaround we can use for the time being?

Adrian

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [Qemu-devel] glibc "linux: spawni.c: simplify error reporting to parent" breaks qemu-user/Windows Service For Linux

[Adhemerval Zanella]

On 27/11/2017 14:10, John Paul Adrian Glaubitz wrote:
> On 11/27/2017 05:07 PM, Adhemerval Zanella wrote:> However I am not very compelled to change internal posix_spawn
>> on GLIBC on Linux mainly because it uses a slight less resources
>> than the generic POSIX one (check e83be730910c) and it works
>> on Linux kernel as expected.
> 
> But it breaks QEMU and Microsoft Windows Services for Linux who -
> combined together - are certainly not a small number of users.

You can bring this to libc-alpha, but I am not sure how other 
maintainers will see to limit the possible Linux API glibc can use 
because of potentially unsupported semantics of non default emulation 
layers.

I personally do not see it a good precedence because we can't really
foretell what kind of kernel ABI or semantic the non default 
emulation runtime will support or not, so we can't really plan to 
set what will break or not depending of the underlying emulation.

> 
> Isn't there any workaround we can use for the time being?
> 
> Adrian
> 

Maybe recompile glibc with the posix implementation instead of the
Linux one. If I recall correctly it should be a workable replacement.