Re: gcc-4.8+ and R10000+

[Joshua Kinard <kumba@gentoo.org>]

On 09/08/2014 05:44, Joshua Kinard wrote:
> On 09/08/2014 04:11, Miod Vallat wrote:
>>> Disassembling a statically-built copy of the "sln" binary generated by
>>> glibc's compile phase, there are slight differences in how gcc-4.7 and
>>> gcc-4.8 are compiling the __lll_lock_wait_private function.  The key
>>> differences in the output asm are
>>> this:
>>
>> [...]
>>
>>> gcc-4.8:
>>>    x+4   <START>
>>>          ...
>>>    x+24  bne     v1,v0,<x+56>
>>>          ...
>>>    x+32  0x7c03e83b /* rdhwr */
>>>    x+36  li      a2,2
>>>    x+40  lw      a1,-29832(v1)
>>>    x+44  move    a3,zero
>>>    x+48  li      v0,4238
>>>    x+52  syscall
>>> *  x+56  ll      v0,0(s0)
>>> *  x+60  li      at,2
>>> *  x+64  sc      at,0
>>
>> Note how the sc address is no longer 0(s0). Since the address does
>> not match the address used in the ll instruction, sc will always
>> fail on the R10k.
> 
> That would be a typo on my part.  I typed that out by hand and just missed it.  It should read:
> 
> gcc-4.8:
>    x+4   <START>
>          ...
>    x+24  bne     v1,v0,<x+56>
>          ...
>    x+32  0x7c03e83b /* rdhwr */
>    x+36  li      a2,2
>    x+40  lw      a1,-29832(v1)
>    x+44  move    a3,zero
>    x+48  li      v0,4238
>    x+52  syscall
> *  x+56  ll      v0,0(s0)
> *  x+60  li      at,2
> *  x+64  sc      at,0(s0)
>    x+68  beqzl   at,<x+56>
>    x+72  nop
>    x+76  sync
>    x+80  bnez    v0,<x+32>

I did some more tracing.  It seems the issue with glibc itself stems from the
addition of __atomic_* builtins added generally in gcc-4.7 and
MIPS-specifically in gcc-4.8:

From ports/sysdeps/mips/bits/atomic.h (for 2.19) or sysdeps/mips/bits/atomic.h
(for 2.20):

/* The __atomic_* builtins are available in GCC 4.7 and later, but MIPS
   support for their efficient implementation was added only in GCC 4.8.
   We still want to use them even with GCC 4.7 for MIPS16 code where we
   have no assembly alternative available and want to avoid the __sync_*
   builtins if at all possible.  */

#if __GNUC_PREREQ (4, 8) || (defined __mips16 && __GNUC_PREREQ (4, 7))
[snip]

This is why the assembly is different between the two gcc versions.  This same
code is in the kernel's atomic.h copy under arch/mips/include/asm/ as well.

I tested by removing the top part of the #if macro and basically forcing the
inline versions only, then rebuilt glibc-2.20 with gcc-4.9.2 (20140921
prerelease), and lo and behold, sln executes and returns its usage information.
 When using the gcc internal builtins, a futex gets used, which is why I wasn't
seeing futexes in 4.7-built copies of sln, only in 4.8 or greater-built copies.
 This means that the gcc internal __atomic_* builtins may be somewhat to blame
for this problem on R1x000 systems.

I traced the kernel side of the problem out and figured out that when the futex
is taken by sln, the process gets frozen by the scheduler via a call to
freezable_schedule() in function futex_wait_queue_me in kernel/futex.c.  I
added two printk statements, one before freezable_schedule() and one after, and
the first statement executes (verified by dumping /proc/kmsg directly because
dmesg itself generates futexes), but not the printk after.  The printk after
freezable_schedule() only executes when I ctrl+C the frozen process and it
exits out of the futex code.

I visually checked through include/linux/freezer.h and noticed that
freezable_schedule eventually calls freezing(), which executes an atomic_read()
on system_freezing_cnt.  In the mips code, that just comes out as a pointer
dereference of a volatile variable.  I'm not certain, though, if in gcc's case,
the use of volatile means it tries to use its builtin __atomic_ functions
again, and tries to take another futex /while it's trying to take a futex/.
Chicken and egg?

So, could still very well be a gcc issue, or maybe it's something really subtle
in the kernel code.  I am not sure which.  I at least know of a specific gcc
commit that enables/disables the problem, and that's pointing the finger at gcc
here.

Ideas?

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic