Re: Can we force a KVM VCPU in Guest Mode to Exit to User Mode From User Mode ?

["Mian M. Hamayun" <mian-muhammad.hamayun@imag.fr>]

[-- Attachment #1: Type: text/plain, Size: 6332 bytes --]



On 07/26/2012 12:43 PM, Avi Kivity wrote:
> On 07/26/2012 01:34 PM, Mian M. Hamayun wrote:
>> On 07/26/2012 12:06 PM, Avi Kivity wrote:
>>> On 07/26/2012 12:38 PM, Mian M. Hamayun wrote:
>>>
>>> <snip>
>>>
>>>> This mechanism 'seems' to work fine when both vcpu threads are in User
>>>> Mode. But when booting an SMP Guest, the boot processor (BSP) initially
>>>> executes the bootstrap code while the non-boot processors (APs) are
>>>> waiting for initial INIT-SIPI-SIPI messages.
>>>>
>>>> What I fail to understand is if an AP is currently waiting for an INIT
>>>> signal, and we call the "run_on_cpu" function above for this cpu, it
>>>> blocks the whole system, as the AP is in Guest mode and cannot call the
>>>> "flush_queued_work" and the BSP is waiting for this to happen.
>>>>
>>>> How can we resolve this deadlock ? Is there a way to force the AP to
>>>> quit the Guest Mode, by using some specific mechanism from the User
>>>> mode ?
>>> When a vcpu is waiting for an INIT, it still responds to signals and
>>> will return to userspace if a signal is received.  Did you observe
>>> something different?
>>>
>> Hi Avi,
>>
>> So it means that when we execute the following:
>>
>> err = pthread_kill(env->thread, SIG_IPI);
>>
>> then this VCPU thread should wake-up and force the VCPU to quit the
>> guest mode ?
> Yes.
>
>> But I am not getting this behavior as the VCPU thread remains blocked in
>> Guest Mode.
>> What could be wrong here ?
>>
> Perhaps signals are blocked?
No, thats not the case.
>
> Can you share your reproducer?
Actually its based on kvm-tool and I have integrated some code from 
qemu-kvm to add debug support to kvm-tool.
I don't have a smaller example that could reproduce the same problem.

>
>> Many Thanks,
>> Hamayun
>>
>> P.S. Please see the following trace; As it might help understanding the
>> problem. (thread 2 is the BSP and thread 3 is for AP)
>>
>> run_on_cpu: Kicked CPU#1 ... Waiting on qemu_work_cond
>>
>> Program received signal SIGUSR1, User defined signal 1.
>> ^C
>> Program received signal SIGINT, Interrupt.
>> 0xb7fdd424 in __kernel_vsyscall ()
>> (gdb) info threads
>>    Id   Target Id         Frame
>>    3    Thread 0xaea17b40 (LWP 2843) "arch.x" 0xb7fdd424 in
>> __kernel_vsyscall ()
>>    2    Thread 0xaf218b40 (LWP 2842) "arch.x" 0xb7fdd424 in
>> __kernel_vsyscall ()
>> * 1    Thread 0xb7359940 (LWP 2822) "arch.x" 0xb7fdd424 in
>> __kernel_vsyscall ()
>> (gdb) thread 2
>> [Switching to thread 2 (Thread 0xaf218b40 (LWP 2842))]
>> #0  0xb7fdd424 in __kernel_vsyscall ()
>> (gdb) bt
>> #0  0xb7fdd424 in __kernel_vsyscall ()
>> #1  0xb7f7096b in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib/i386-linux-gnu/libpthread.so.0
>> #2  0xb7fc5715 in qemu_cond_wait (cond=0xb7fda640, lock=0xb7fda670) at
>> gdb_srv_arch.c:578
>> #3  0xb7fc5981 in run_on_cpu (env=0x8465818, func=0xb7fc5417
>> <kvm_invoke_set_guest_debug>, data=0xaf20fe50) at gdb_srv_arch.c:782
>> #4  0xb7fc5b90 in kvm_update_guest_debug (env=0x8465818,
>> reinject_trap=0) at gdb_srv_arch.c:863
>> #5  0xb7fc5ef1 in kvm_remove_all_breakpoints (current_env=0x8465418) at
>> gdb_srv_arch.c:983
>> #6  0xb7fc2e0a in gdb_breakpoint_remove_all (env=0x8465418) at
>> gdb_srv.c:369
>> #7  0xb7fc3139 in gdb_handle_packet (s=0x80ac3e0, line_buf=0x80ac3fc
>> "?") at gdb_srv.c:497
>> #8  0xb7fc439c in gdb_read_byte (s=0x80ac3e0, ch=102) at gdb_srv.c:830
>> #9  0xb7fc4522 in gdb_loop (env=0x8465418) at gdb_srv.c:878
>> #10 0xb7fc4621 in gdb_srv_handle_debug (env=0x8465418) at gdb_srv.c:924
>> #11 0xb7fc5265 in kvm_arch_handle_debug (env=0x8465418) at
>> gdb_srv_arch.c:424
>> #12 0xb7fac3a8 in kvm_cpu__start (cpu=0x8465418) at kvm-cpu.c:588
>> #13 0xb7fc6521 in kvm_cpu_thread (arg=0x8465418) at libkvm-main.c:276
>> #14 0xb7f6cd4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
>> #15 0xb7d7bace in clone () from /lib/i386-linux-gnu/libc.so.6
>> (gdb) thread 3
>> [Switching to thread 3 (Thread 0xaea17b40 (LWP 2843))]
>> #0  0xb7fdd424 in __kernel_vsyscall ()
>> (gdb) bt
>> #0  0xb7fdd424 in __kernel_vsyscall ()
>> #1  0xb7d73869 in ioctl () from /lib/i386-linux-gnu/libc.so.6
>> #2  0xb7fabe9c in kvm_cpu__run (vcpu=0x8465818) at kvm-cpu.c:429
>> #3  0xb7fac376 in kvm_cpu__start (cpu=0x8465818) at kvm-cpu.c:578
>> #4  0xb7fc6521 in kvm_cpu_thread (arg=0x8465818) at libkvm-main.c:276
>> #5  0xb7f6cd4c in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
>> #6  0xb7d7bace in clone () from /lib/i386-linux-gnu/libc.so.6
>>
> Are you sure thread 3 did not receive the signal?

The thread 3 does actually receives the signal, but the order is not right.

As the BSP (Thread 2) starts, it locks the "qemu_global_mutex" and 
releases it only when it calls the "run_on_cpu" function and starts 
waiting on "qemu_work_cond".
The AP (Thread 3) wakes-up due to the SIG_IPI signal from thread 2, 
acquires the lock on "qemu_global_mutex" and enters the guest mode. 
(This is the deadlock case)

If we do not lock the "qemu_global_mutex" in each cpu thread at the 
beginning, and only lock it when we quit the guest mode, the problem 
seems to go away, as now we get the SIG_IPI when the Thread 3 is 
actually in the guest mode and it quits to user mode.
But I am not sure if this is the right way to do it, as in qemu-kvm we 
_always_ start each cpu thread by locking the "qemu_global_mutex".

i.e.

static void *qemu_kvm_cpu_thread_fn(void *arg)
{
     CPUArchState *env = arg;
     int r;

     qemu_mutex_lock(&qemu_global_mutex);
     qemu_thread_get_self(env->thread);
     env->thread_id = qemu_get_thread_id();
     cpu_single_env = env;

     r = kvm_init_vcpu(env);
     if (r < 0) {
         fprintf(stderr, "kvm_init_vcpu failed: %s\n", strerror(-r));
         exit(1);
     }

     qemu_kvm_init_cpu_signals(env);

     /* signal CPU creation */
     env->created = 1;
     qemu_cond_signal(&qemu_cpu_cond);

     while (1) {
         if (cpu_can_run(env)) {
             r = kvm_cpu_exec(env);
             if (r == EXCP_DEBUG) {
                 cpu_handle_guest_debug(env);
             }
         }
         qemu_kvm_wait_io_event(env);
     }

     return NULL;
}

>
> Try stracing the run.
>
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2685 bytes --]