Re: [ANNOUNCE] kvm-51 release

[david ahern <daahern-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>]

With kvm-52 my 32-bit host running RHEL5.1 can start an RHEL 5 SMP guest only once. Second and subsequent attempts hang. Removing kvm and kvm_intel modules have no affect; I need to reboot the host to get an SMP guest to start. My similarly configured 64-bit host does not seem to have this problem.


Second attempts to start the RHEL5 SMP guest hang at:
Starting udev: _


Looking at top on the host shows qemu in a loop:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                
 3909 root      18   0 1625m  67m 9476 R  400  2.1   2:52.32 qemu-system-x86                                         


In this case the qemu threads are:

  PID   LWP TTY          TIME CMD
 3909  3909 pts/0    00:01:12 qemu-system-x86
 3909  3911 pts/0    00:01:05 qemu-system-x86
 3909  3912 pts/0    00:01:05 qemu-system-x86
 3909  3913 pts/0    00:01:07 qemu-system-x86
 3909  3917 pts/0    00:00:00 qemu-system-x86



and their kernel side backtraces are:

 process trace for qemu-system-x86(3909)
        f5967d88 00000082 f8c125e4 bbdec465 000001c6 f5230da4 00000001 f7acf000 
        f7d7d000 bbded629 000001c6 000011c4 00000000 f7acf110 c30126e0 00000001 
        f4d8a000 f5967d90 f5967d80 f5230da0 f5967000 f8c11120 f5230da0 f4d8a000 
 Call Trace:
  [<f8c125e4>] vmx_vcpu_put+0xef/0xf6 [kvm_intel]
  [<f8c11120>] handle_external_interrupt+0x0/0xc [kvm_intel]
  [<c042169f>] __cond_resched+0x16/0x34
  [<c0604218>] cond_resched+0x2a/0x31
  [<f8b96d7f>] kvm_arch_vcpu_ioctl_run+0x28d/0x333 [kvm]
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
  [<c042169f>] __cond_resched+0x16/0x34
  [<c0604218>] cond_resched+0x2a/0x31
  [<c0480305>] core_sys_select+0x1ef/0x2ca
  [<c041ea84>] __wake_up_common+0x2f/0x53
  [<c0604141>] schedule+0x90d/0x9ba
  [<c0405953>] reschedule_interrupt+0x1f/0x24
  [<c042e759>] __dequeue_signal+0x151/0x15c
  [<c042fa99>] dequeue_signal+0x2d/0x9c
  [<c043062c>] sys_rt_sigtimedwait+0xc5/0x2c2
  [<c042cc0e>] getnstimeofday+0x30/0xb6
  [<c04386d6>] ktime_get_ts+0x16/0x44
  [<c04388b6>] ktime_get+0x12/0x34
  [<c04352a6>] common_timer_get+0xee/0x129
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<c047f1e8>] do_ioctl+0x1c/0x5d
  [<c047f473>] vfs_ioctl+0x24a/0x25c
  [<c047f4cd>] sys_ioctl+0x48/0x5f
  [<c0404eff>] syscall_call+0x7/0xb
  =======================
 process trace for qemu-system-x86(3911)
        c301a6e0 00000100 000001c7 f749baa0 00000001 c301a6e0 f749baa0 00000001 
        f51fed44 f51fed44 f51fed6c 00000001 00000001 00000046 f579ce20 f57ee000 
        00000001 c04059bf f579ce20 8005003b 00006c00 f8c113e5 f579ce20 f579ce20 
 Call Trace:
  [<c04059bf>] apic_timer_interrupt+0x1f/0x24
  [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
  [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
  [<c041fa31>] enqueue_task+0x29/0x39
  [<c041fa5d>] __activate_task+0x1c/0x29
  [<c04202a7>] try_to_wake_up+0x371/0x37b
  [<c0604141>] schedule+0x90d/0x9ba
  [<c041ea84>] __wake_up_common+0x2f/0x53
  [<c041f871>] __wake_up+0x2a/0x3d
  [<c0438eb7>] wake_futex+0x3a/0x44
  [<c0439187>] futex_wake+0xa9/0xb3
  [<c0439d66>] do_futex+0x20d/0xb15
  [<f8b94696>] kvm_ack_smp_call+0x17/0x27 [kvm]
  [<c042e759>] __dequeue_signal+0x151/0x15c
  [<c042fa99>] dequeue_signal+0x2d/0x9c
  [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
  [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
  [<c04202b1>] default_wake_function+0x0/0xc
  [<c040599b>] call_function_interrupt+0x1f/0x24
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<c047f1e8>] do_ioctl+0x1c/0x5d
  [<c047f473>] vfs_ioctl+0x24a/0x25c
  [<c047f4cd>] sys_ioctl+0x48/0x5f
  [<c0404eff>] syscall_call+0x7/0xb
  =======================
 process trace for qemu-system-x86(3912)
        f560fd88 00000082 f8c125e4 193272c5 000001c8 f52b6074 00000004 f7f09000 
        f7f09000 19328fc9 000001c8 00001d04 00000002 f52b6070 55eefb90 00000000 
        f52b6070 f5693000 f52b6070 8005003b 00006c00 f8c113e5 f52b6070 f52b6070 
 Call Trace:
  [<f8c125e4>] vmx_vcpu_put+0xef/0xf6 [kvm_intel]
  [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
  [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
  [<c0604141>] schedule+0x90d/0x9ba
  [<c041ea84>] __wake_up_common+0x2f/0x53
  [<c0461e10>] find_extend_vma+0x12/0x49
  [<c0438d53>] get_futex_key+0x40/0xd0
  [<c0439187>] futex_wake+0xa9/0xb3
  [<c0439d66>] do_futex+0x20d/0xb15
  [<f888f9b0>] ext3_ordered_writepage+0x0/0x162 [ext3]
  [<c042e759>] __dequeue_signal+0x151/0x15c
  [<c042fa99>] dequeue_signal+0x2d/0x9c
  [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
  [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
  [<c04202b1>] default_wake_function+0x0/0xc
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<c047f1e8>] do_ioctl+0x1c/0x5d
  [<c047f473>] vfs_ioctl+0x24a/0x25c
  [<c047f4cd>] sys_ioctl+0x48/0x5f
  [<c0404eff>] syscall_call+0x7/0xb
  =======================
 process trace for qemu-system-x86(3913)
        c302a6e0 00000100 000001c8 f7488550 00000003 c302a6e0 f7488550 00000003 
        f4d92d44 f4d92d44 f4d92d6c 00000001 00000001 00000046 f52b6de0 f5aaa000 
        00000001 c04059bf f52b6de0 8005003b 00006c00 f8c113e5 f52b6de0 f52b6de0 
 Call Trace:
  [<c04059bf>] apic_timer_interrupt+0x1f/0x24
  [<f8c113e5>] vmcs_writel+0x1b/0x2c [kvm_intel]
  [<f8b96cf4>] kvm_arch_vcpu_ioctl_run+0x202/0x333 [kvm]
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<f8b943d4>] kvm_vcpu_ioctl+0xbb/0x366 [kvm]
  [<c0604141>] schedule+0x90d/0x9ba
  [<c041ea84>] __wake_up_common+0x2f/0x53
  [<c0461e10>] find_extend_vma+0x12/0x49
  [<c0438d53>] get_futex_key+0x40/0xd0
  [<c0439187>] futex_wake+0xa9/0xb3
  [<c0439d66>] do_futex+0x20d/0xb15
  [<c040599b>] call_function_interrupt+0x1f/0x24
  [<c042e759>] __dequeue_signal+0x151/0x15c
  [<f8b93ea9>] kvm_vm_ioctl+0x0/0x277 [kvm]
  [<f8b9410d>] kvm_vm_ioctl+0x264/0x277 [kvm]
  [<c04202b1>] default_wake_function+0x0/0xc
  [<f8b94319>] kvm_vcpu_ioctl+0x0/0x366 [kvm]
  [<c047f1e8>] do_ioctl+0x1c/0x5d
  [<c047f473>] vfs_ioctl+0x24a/0x25c
  [<c047f4cd>] sys_ioctl+0x48/0x5f
  [<c0404eff>] syscall_call+0x7/0xb
  =======================

david


Avi Kivity wrote:
> david ahern wrote:
>> The patch worked for me -- rhel4 smp guests boot fine on stock RHEL5
>> hosts, both 32-bit and 64-bit.
>>
>>   
> 
> Excellent.  I had a premonition so it is already committed.
> 
> Do note that smp_call_function_mask() emulation is pretty bad in terms
> of performance on large multicores.  On a dual code it's basically
> equivalent to mainline, I guess it's okay for four-way, but above
> four-way you will need either mainline or a better
> smp_call_function_mask() (which is nontrivial but doable).
> 
>> david
>>
>>
>> Avi Kivity wrote:
>>  
>>> david ahern wrote:
>>>    
>>>> In RHEL 5.1 <linux/notifier.h> defines:
>>>>
>>>> #define CPU_TASKS_FROZEN    0x0010
>>>>
>>>> #define CPU_ONLINE_FROZEN   (CPU_ONLINE | CPU_TASKS_FROZEN)
>>>> #define CPU_DEAD_FROZEN     (CPU_DEAD | CPU_TASKS_FROZEN)
>>>>
>>>> which means in kvm-51/kernel/external-module-compat.h the '#ifndef
>>>> CPU_TASKS_FROZEN' needs to have a case. For my purposes, I just moved
>>>> up the endif around what was defined.
>>>>         
>>> I committed a change which renders this unnecessary.  Will be part of
>>> kvm-52.
>>>
>>>    
>>>> With that change, kvm-51 compiles. I am still seeing 32-bit SMP guests
>>>> hang on boot for both 32-bit and 64-bit hosts (again running RHEL5.1).
>>>>         
>>> I still don't.  Can you test the attached patch?
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> -------------------------------------------------------------------------
>>>
>>> This SF.net email is sponsored by: Splunk Inc.
>>> Still grepping through log files to find problems?  Stop.
>>> Now Search log events and configuration files using AJAX and a browser.
>>> Download your FREE copy of Splunk now >> http://get.splunk.com/
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> kvm-devel mailing list
>>> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>>> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>>>     
> 
> 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/