qemu_mutex deadlock with -smp 2

qemu_mutex deadlock with -smp 2

[Darrick J. Wong]

Hi,

I'm attempting to run a KVM instance with 2 CPUs configured.  The VM
appears to get most of the way through booting Ubuntu and then kvm locks
up.  At that point there's nothing printed on stdout from the kvm
process and the monitor doesn't respond either.  The host's kernel is
2.6.24-rc7 and the kvm binary is built from git.

Here's the command line I use:

/usr/local/bin/qemu-system-x86_64 -serial telnet:0.0.0.0:7005,server,nodelay \
  -m 1024 -smp 2 -vnc :5 -std-vga \
  -net nic,macaddr=00:12:34:56:78:95,model=rtl8139 -net tap -boot c \
  -monitor telnet:0.0.0.0:8005,server,nowait,nodelay \
  -drive file=/u/abat0.img,if=scsi,boot=on -snapshot

This usually suffices to reproduce:

/usr/local/bin/qemu-system-x86_64 \
  -m 1024 -smp 2 \
  -net nic,macaddr=00:12:34:56:78:96,model=rtl8139 -net tap -boot c \
  -monitor telnet:0.0.0.0:8006,server,nowait,nodelay \
  -hda /u/abat0.img -snapshot

An strace of the process shows that it's stuck here:
futex(0x60281640, FUTEX_WAIT, 2, NULL)
and this is the gdb stack trace:

#0  0x00002ae19369eaf8 in __lll_mutex_lock_wait () from /lib/libpthread.so.0
#1  0x00002ae19369aada in _L_mutex_lock_100 () from /lib/libpthread.so.0
#2  0x00002ae19369a475 in pthread_mutex_lock () from /lib/libpthread.so.0
#3  0x000000006008c6f7 in kvm_eat_signal (env=0x60281640, timeout=-1795357456) at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:176
#4  0x000000006008c82e in kvm_eat_signals (env=0x625f15a0, timeout=10) at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:200
#5  0x000000006008c89b in kvm_main_loop_wait (env=0x625f15a0, timeout=10) at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:219
#6  0x000000006008d04d in kvm_main_loop_cpu (env=0x625f15a0) at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:335
#7  0x0000000060010b94 in main (argc=0, argv=0x7fff181eb248) at /usr/local/src/kvm-userspace/qemu/vl.c:7817

It appears that we're deadlocked on qemu_mutex; it would also appear
that disabling the network interface buys me a few more seconds until
the lockup occurs.  Most of the time it is the case that post_kvm_run()
locked qemu_mutex and then kvm_eat_signal() tries to lock it as well.

--D

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: qemu_mutex deadlock with -smp 2

[Avi Kivity]

Darrick J. Wong wrote:
> Hi,
>
> I'm attempting to run a KVM instance with 2 CPUs configured.  The VM
> appears to get most of the way through booting Ubuntu and then kvm locks
> up.  At that point there's nothing printed on stdout from the kvm
> process and the monitor doesn't respond either.  The host's kernel is
> 2.6.24-rc7 and the kvm binary is built from git.
>
> Here's the command line I use:
>
>   
[...]

> It appears that we're deadlocked on qemu_mutex; it would also appear
> that disabling the network interface buys me a few more seconds until
> the lockup occurs.  Most of the time it is the case that post_kvm_run()
> locked qemu_mutex and then kvm_eat_signal() tries to lock it as well.
>   

Are you using qcow?  If so, this may be the recently-committed smp aio 
deadlock fix.  See 6bcdef5c1157bd7b526491252d20f8007ff2e9dc.  Please try 
a fresh qemu-userspace.git.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: qemu_mutex deadlock with -smp 2

[Darrick J. Wong]

On Tue, Jan 15, 2008 at 05:07:46PM +0200, Avi Kivity wrote:

> Are you using qcow?  If so, this may be the recently-committed smp aio 
> deadlock fix.  See 6bcdef5c1157bd7b526491252d20f8007ff2e9dc.  Please try a 
> fresh qemu-userspace.git.

qcow2, actually.  qcow + fresh git seems to reduce the deadlocks, but
not eliminate them entirely.  Converting to raw fixes the problem.

--D

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: qemu_mutex deadlock with -smp 2

[Darrick J. Wong]

On Tue, Jan 15, 2008 at 12:13:40PM -0800, Darrick J. Wong wrote:
> On Tue, Jan 15, 2008 at 05:07:46PM +0200, Avi Kivity wrote:
> 
> > Are you using qcow?  If so, this may be the recently-committed smp aio 
> > deadlock fix.  See 6bcdef5c1157bd7b526491252d20f8007ff2e9dc.  Please try a 
> > fresh qemu-userspace.git.
> 
> qcow2, actually.  qcow + fresh git seems to reduce the deadlocks, but
> not eliminate them entirely.  Converting to raw fixes the problem.

Drat, raw doesn't really fix it either.  Deadlock on reboot; same stack
trace as before:

(gdb) where
#0  0x00002ab011f45af8 in __lll_mutex_lock_wait () from
/lib/libpthread.so.0
#1  0x00002ab011f41ada in _L_mutex_lock_100 () from /lib/libpthread.so.0
#2  0x00002ab011f41475 in pthread_mutex_lock () from
/lib/libpthread.so.0
#3  0x000000006008c767 in kvm_eat_signal (env=0x60281640,
timeout=327644400) at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:176
#4  0x000000006008c84f in kvm_eat_signals (env=0x625944a0, timeout=0) at
/usr/local/src/kvm-userspace/qemu/qemu-kvm.c:198
#5  0x000000006008c90b in kvm_main_loop_wait (env=0x625944a0, timeout=0)
at /usr/local/src/kvm-userspace/qemu/qemu-kvm.c:219
#6  0x000000006008d0cb in kvm_main_loop_cpu (env=0x625944a0) at
/usr/local/src/kvm-userspace/qemu/qemu-kvm.c:346
#7  0x0000000060010b94 in main (argc=0, argv=0x7fff99945418) at
/usr/local/src/kvm-userspace/qemu/vl.c:7817

--D

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: qemu_mutex deadlock with -smp 2

[Avi Kivity]

Darrick J. Wong wrote:
> On Tue, Jan 15, 2008 at 12:13:40PM -0800, Darrick J. Wong wrote:
>   
>> On Tue, Jan 15, 2008 at 05:07:46PM +0200, Avi Kivity wrote:
>>
>>     
>>> Are you using qcow?  If so, this may be the recently-committed smp aio 
>>> deadlock fix.  See 6bcdef5c1157bd7b526491252d20f8007ff2e9dc.  Please try a 
>>> fresh qemu-userspace.git.
>>>       
>> qcow2, actually.  qcow + fresh git seems to reduce the deadlocks, but
>> not eliminate them entirely.  Converting to raw fixes the problem.
>>     
>
> Drat, raw doesn't really fix it either.  Deadlock on reboot; same stack
> trace as before:
>   

Is this immediately after reboot, or during the boot process?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: qemu_mutex deadlock with -smp 2

[Darrick J. Wong]

On Sun, Jan 20, 2008 at 05:15:07PM +0200, Avi Kivity wrote:

> Is this immediately after reboot, or during the boot process?

After reboot.

--D

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: qemu_mutex deadlock with -smp 2

[Avi Kivity]

Darrick J. Wong wrote:
> On Tue, Jan 15, 2008 at 12:13:40PM -0800, Darrick J. Wong wrote:
>   
>> On Tue, Jan 15, 2008 at 05:07:46PM +0200, Avi Kivity wrote:
>>
>>     
>>> Are you using qcow?  If so, this may be the recently-committed smp aio 
>>> deadlock fix.  See 6bcdef5c1157bd7b526491252d20f8007ff2e9dc.  Please try a 
>>> fresh qemu-userspace.git.
>>>       
>> qcow2, actually.  qcow + fresh git seems to reduce the deadlocks, but
>> not eliminate them entirely.  Converting to raw fixes the problem.
>>     
>
> Drat, raw doesn't really fix it either.  Deadlock on reboot; same stack
> trace as before:
>   



How does current kvm-userspace.git fare?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/