[Qemu-devel] SEV and WFE instructions on ARM

[Qemu-devel] SEV and WFE instructions on ARM

[Sebastian Huber]

Hello,

I want to use Qemu to test some SMP code.  For this I set up Qemu to fire up 
two Cortex-A9 MPCore CPUs.  I have the following ticket lock implementation:

static inline void _ARM_Data_memory_barrier( void )
{
   __asm__ volatile ( "dmb" : : : "memory" );
}

static inline void _ARM_Data_synchronization_barrier( void )
{
   __asm__ volatile ( "dsb" : : : "memory" );
}

static inline void _ARM_Send_event( void )
{
   __asm__ volatile ( "sev" : : : "memory" );
}

static inline void _ARM_Wait_for_event( void )
{
   __asm__ volatile ( "wfe" : : : "memory" );
}

typedef struct {
   uint32_t next_ticket;
   uint32_t now_serving;
} CPU_SMP_lock_Control;

#define CPU_SMP_LOCK_INITIALIZER { 0, 0 }

static inline void _CPU_SMP_lock_Acquire( CPU_SMP_lock_Control *lock )
{
   uint32_t my_ticket;
   uint32_t next_ticket;
   uint32_t status;

   __asm__ volatile (
     "1: ldrex %[my_ticket], [%[next_ticket_addr]]\n"
     "add %[next_ticket], %[my_ticket], #1\n"
     "strex %[status], %[next_ticket], [%[next_ticket_addr]]\n"
     "teq %[status], #0\n"
     "bne 1b"
     : [my_ticket] "=&r" (my_ticket),
       [next_ticket] "=&r" (next_ticket),
       [status] "=&r" (status)
     : [next_ticket_addr] "r" (&lock->next_ticket)
     : "cc", "memory"
   );

   while ( my_ticket != lock->now_serving ) {
     _ARM_Wait_for_event();
   }

   _ARM_Data_memory_barrier();
}

static inline void _CPU_SMP_lock_Release( CPU_SMP_lock_Control *lock )
{
   _ARM_Data_memory_barrier();
   ++lock->now_serving;
   _ARM_Data_synchronization_barrier();
   _ARM_Send_event();
}

I run the following code on both CPUs:

while (1) {
   _CPU_SMP_lock_Acquire(&lock);
   ++global_counter;
   _CPU_SMP_lock_Release(&lock);
}

It seems that the SEV/WFE instructions are implemented as a nop on Qemu (see in 
file "target-arm/translate.c" function gen_nop_hint()).  So the simulator 
executes the busy wait loop most of the time.  Is it possible to trigger a 
schedule event in Qemu which stops the simulation on one CPU and selects 
another CPU instead?

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: [Qemu-devel] SEV and WFE instructions on ARM

[Peter Maydell]

On 6 June 2013 15:34, Sebastian Huber
<sebastian.huber@embedded-brains.de> wrote:
> I want to use Qemu to test some SMP code.  For this I set up Qemu to fire up
> two Cortex-A9 MPCore CPUs.  I have the following ticket lock implementation:

QEMU is not a very good choice for this sort of testing,
because we do not implement any of:
 * execution of multiple CPUs in parallel
 * barriers as anything other than NOPs [this is OK because
   we don't execute CPUs in parallel]
 * non-NOP WFE
 * caches

and so there are large classes of guest code bugs that will never
manifest under QEMU.

I would strongly advise testing your SMP primitives on real hardware.

> It seems that the SEV/WFE instructions are implemented as a nop on Qemu (see
> in file "target-arm/translate.c" function gen_nop_hint()).

This is an architecturally valid implementation (though perhaps
not the most useful one possible).

>  So the simulator
> executes the busy wait loop most of the time.  Is it possible to trigger a
> schedule event in Qemu which stops the simulation on one CPU and selects
> another CPU instead?

It would be possible to implement WFE "properly" in a similar
manner to how we currently handle WFI. (You'd need to actually
implement the underlying 'event register' and make sure it got
set by SEV and all the other cases which the architecture lists.)
If you did that then a WFE would cause execution of one guest
CPU to pause and the next one to start.

thanks
-- PMM

Re: [Qemu-devel] SEV and WFE instructions on ARM

[Sebastian Huber]

Hello Peter,

thanks for your suggestions.  I will look at the WFI support and may send a 
patch for the SEV/WFE, but this will take some weeks.

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax     : +49 89 189 47 41-09
E-Mail  : sebastian.huber@embedded-brains.de
PGP     : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

Re: [Qemu-devel] SEV and WFE instructions on ARM

[Walsh, Benjamin]

Hello Sebastian,

I was looking at using WFE on QEMU, found that it is implemented as a NOP, googled and I found this thread.

I am not using this for SMP, simply to implement sleep, so the busy loop is conceptually OK. Except, of course, that QEMU floors the host CPU at 100% in the idle loop.

You mentioned that you might send some patches for WFE/SEV, but I haven't seen anything in the current QEMU repo. Did you get anywhere with this ?

Thanks,
Ben

---
Benjamin Walsh, MTS
VxMicro
Desk : (613)270-2261
Fax  : (613)592-2283
www.windriver.com

Re: [Qemu-devel] SEV and WFE instructions on ARM

[Peter Maydell]

On 26 November 2013 12:51, Walsh, Benjamin <benjamin.walsh@windriver.com> wrote:
> I was looking at using WFE on QEMU, found that it is
> implemented as a NOP, googled and I found this thread.
>
> I am not using this for SMP, simply to implement sleep, so the busy
> loop is conceptually OK. Except, of course, that QEMU floors the
> host CPU at 100% in the idle loop.

FWIW, Linux uses WFI for its idle loop:
http://lxr.linux.no/#linux+v3.12.1/arch/arm/mm/proc-v7.S#L64

If you're in a position to recode your idle loop to WFI
instead of WFE that would be a workaround, because
QEMU does halt the guest cpu on WFI, so we won't
burn host CPU in a WFI-based idle loop.

-- PMM