[Qemu-devel] qemu-arm-static, multiprocess, atomic

[Qemu-devel] qemu-arm-static, multiprocess, atomic

[Jeff Epler]

Hi.  Before I take the time to prepare a detailed bug report, I wanted to
ask whether I'm doing something that is expected to work, or expected not
to work.

I am a developer of the LinuxCNC, Free and Open Source software for control
of milling machines, lathes, and so forth.

Our software uses an IPC model where multiple processes share a
memory-mapped region which contains data as well as mutexes to manipulate
the data (using gcc intrinsics like
__sync_fetch_and_or).

On real multiprocessor ARM systems, our testsuite passes, but when running
in a debian schroot using qemu-arm-static it regularly deadlocks,
apparently with two processes both trying to take the same mutex in a
shared memory region.

>From what I have been able to tell, the emulation of atomics in qemu is
based on a single-process, multi-thread model (based on start_exclusive()
in linux-user/main.c,
http://git.qemu.org/?p=qemu.git;a=blob;f=linux-user/main.c;h=95ed11d85c1#l160)
so right now I think that our code is expected not to work on qemu-user.
If that's the case, for our purposes I will simply document that while our
software can be built in a qemu-user environment, it cannot be used and its
testsuite experiences spurious failures.

Thanks for your time,
Jeff

Re: [Qemu-devel] qemu-arm-static, multiprocess, atomic

[Sergey Fedorov]

On 25/05/16 15:34, Jeff Epler wrote:
> Hi.  Before I take the time to prepare a detailed bug report, I wanted to
> ask whether I'm doing something that is expected to work, or expected not
> to work.
>
> I am a developer of the LinuxCNC, Free and Open Source software for control
> of milling machines, lathes, and so forth.
>
> Our software uses an IPC model where multiple processes share a
> memory-mapped region which contains data as well as mutexes to manipulate
> the data (using gcc intrinsics like
> __sync_fetch_and_or).
>
> On real multiprocessor ARM systems, our testsuite passes, but when running
> in a debian schroot using qemu-arm-static it regularly deadlocks,
> apparently with two processes both trying to take the same mutex in a
> shared memory region.
>
> From what I have been able to tell, the emulation of atomics in qemu is
> based on a single-process, multi-thread model (based on start_exclusive()
> in linux-user/main.c,
> http://git.qemu.org/?p=qemu.git;a=blob;f=linux-user/main.c;h=95ed11d85c1#l160)
> so right now I think that our code is expected not to work on qemu-user.
> If that's the case, for our purposes I will simply document that while our
> software can be built in a qemu-user environment, it cannot be used and its
> testsuite experiences spurious failures.
>
> Thanks for your time,
> Jeff

Hi Jeff,

Looks like you are right that ARM exclusive access implementation in
QEMU implies a single-process, multi-thread model and it's not supposed
to work for your use-case. I'm afraid even our recent efforts in
multi-threaded TCG won't change the situation. The problem is that it
would require to translate somehow ARM's exclusive access monitor to x86
model.

Kind regards,
Sergey

Re: [Qemu-devel] qemu-arm-static, multiprocess, atomic

[Paolo Bonzini]

On 26/05/2016 12:42, Sergey Fedorov wrote:
> I'm afraid even our recent efforts in
> multi-threaded TCG won't change the situation. The problem is that it
> would require to translate somehow ARM's exclusive access monitor to x86
> model.
> 

The cmpxchg-based variant would work.  It doesn't implement LL/SC
exactly, but it works fine if the program is written against GCC atomic
builtins.

Thanks,

Paolo

Re: [Qemu-devel] qemu-arm-static, multiprocess, atomic

[Sergey Fedorov]

On 26/05/16 14:21, Paolo Bonzini wrote:
>
> On 26/05/2016 12:42, Sergey Fedorov wrote:
>> I'm afraid even our recent efforts in
>> multi-threaded TCG won't change the situation. The problem is that it
>> would require to translate somehow ARM's exclusive access monitor to x86
>> model.
>>
> The cmpxchg-based variant would work.  It doesn't implement LL/SC
> exactly, but it works fine if the program is written against GCC atomic
> builtins.

I'm afraid, GCC can still generate ldrex/strex pair for
__atomic_compare_exchange():

    $ cat a.c
    #define atomic_cmpxchg(ptr, old,
    new)                                   \
       
    ({                                                                  \
         typeof(*ptr) _old = (old), _new =
    (new);                           \
         __atomic_compare_exchange(ptr, &_old, &_new,
    0,                    \
             __ATOMIC_SEQ_CST,
    __ATOMIC_SEQ_CST);                           \
        
    _old;                                                              \
        
    })                                                                  


    int cmpexchg(int *ptr, int old, int new)
    {
        return atomic_cmpxchg(ptr, old, new);
    }

    $ arm-linux-gnueabi-gcc -march=armv7-a -O2 -c a.c
    $ arm-linux-gnueabi-objdump -d a.o

    a.o:     file format elf32-littlearm


    Disassembly of section .text:

    00000000 <cmpexchg>:
       0:    e24dd008     sub    sp, sp, #8
       4:    e58d1004     str    r1, [sp, #4]
       8:    f57ff05f     dmb    sy
       c:    e1903f9f     ldrex    r3, [r0]
      10:    e1530001     cmp    r3, r1
      14:    1a000002     bne    24 <cmpexchg+0x24>
      18:    e180cf92     strex    ip, r2, [r0]
      1c:    e35c0000     cmp    ip, #0
      20:    1afffff9     bne    c <cmpexchg+0xc>
      24:    f57ff05f     dmb    sy
      28:    e1a00003     mov    r0, r3
      2c:    e28dd008     add    sp, sp, #8
      30:    e12fff1e     bx    lr

Kind regards,
Sergey

Re: [Qemu-devel] qemu-arm-static, multiprocess, atomic

[Paolo Bonzini]

On 26/05/2016 14:53, Sergey Fedorov wrote:
>>> I'm afraid even our recent efforts in
>>> multi-threaded TCG won't change the situation. The problem is that it
>>> would require to translate somehow ARM's exclusive access monitor to x86
>>> model.
>>>
>> The cmpxchg-based variant would work.  It doesn't implement LL/SC
>> exactly, but it works fine if the program is written against GCC atomic
>> builtins.
> 
> I'm afraid, GCC can still generate ldrex/strex pair for
> __atomic_compare_exchange():
> 

I mean the version of mttcg that uses cmpxchg to implement ldrex/strex
(save the value on ldrex, do a cmpxchg on strex).

Thanks,

Paolo