[Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

Hi Guys,

I am building QEMU on Windows using MinGW + Msys. If I build QEMU with
Glib 2.28, qemu binary works fine for sdcard. I noticed that QEMU
binary has dependency on libgthread-2.0-0.dll in this case.
But when I build QEMU with Glib 2.33.8 version, its sdcard option
throw error at run time. Also generated QEMU binary has no dependency
on gthread DLL.
Below is the error thrown while using -sd:

vvfat C:\qemu_arm_win32\sdcard\files chs 1024,16,63
Co-routine is yielding to no one

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Thanks,
Taimoor

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Stefan Hajnoczi]

On Wed, Aug 28, 2013 at 10:14:04AM +0500, Taimoor Mirza wrote:
> I am building QEMU on Windows using MinGW + Msys. If I build QEMU with
> Glib 2.28, qemu binary works fine for sdcard. I noticed that QEMU
> binary has dependency on libgthread-2.0-0.dll in this case.
> But when I build QEMU with Glib 2.33.8 version, its sdcard option
> throw error at run time. Also generated QEMU binary has no dependency
> on gthread DLL.
> Below is the error thrown while using -sd:
> 
> vvfat C:\qemu_arm_win32\sdcard\files chs 1024,16,63
> Co-routine is yielding to no one

What is the output of:
$ grep CONFIG_COROUTINE_BACKEND config-host.mak

On Windows it should be "win32".

Can you capture a backtrace using gdb?

$ gdb --args qemu-system-arm.exe ...
(gdb) r
Co-routine is yielding to no one
(gdb) bt

Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

Hi Stefan,

Thanks for your reply. Below are answers

$ grep CONFIG_COROUTINE_BACKEND config-host.mak
CONFIG_COROUTINE_BACKEND=win32

(gdb) r
Starting program: C:\tools\qemu\qemu-system-arm.exe
-M realview-eb -m 256 -kernel Debug\\\\KD2.out -sd fat:16:rw:C:\\testCard
[New Thread 3836.0x404]
[New Thread 3836.0x87c]
[New Thread 3836.0xd14]
[New Thread 3836.0xc58]
[Inferior 1 (process 3836) exited with code 03]
(gdb) bt
No stack.
(gdb)

When I open stderr in QEMU binary directory, it contains:
vvfat C:\\testCard chs 1024,16,63
Co-routine is yielding to no one

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Another important point to note is that it happens when I add -DNDEBUG
in my QEMU's cflags. If I don't add this then I get following error:

vvfat C:\\testCard chs 1024,16,63 Assertion failed:
qemu_in_coroutine(), file qemu-coroutine-lock.c, line 148 This
application has requested the Runtime to terminate it in an unusual
way. Please contact the application's support team for more
information.

It looks to me that it is an issue with the QEMU block driver
following the coroutine protection.

Thanks,
Taimoor


On Wed, Aug 28, 2013 at 1:11 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Wed, Aug 28, 2013 at 10:14:04AM +0500, Taimoor Mirza wrote:
>> I am building QEMU on Windows using MinGW + Msys. If I build QEMU with
>> Glib 2.28, qemu binary works fine for sdcard. I noticed that QEMU
>> binary has dependency on libgthread-2.0-0.dll in this case.
>> But when I build QEMU with Glib 2.33.8 version, its sdcard option
>> throw error at run time. Also generated QEMU binary has no dependency
>> on gthread DLL.
>> Below is the error thrown while using -sd:
>>
>> vvfat C:\qemu_arm_win32\sdcard\files chs 1024,16,63
>> Co-routine is yielding to no one
>
> What is the output of:
> $ grep CONFIG_COROUTINE_BACKEND config-host.mak
>
> On Windows it should be "win32".
>
> Can you capture a backtrace using gdb?
>
> $ gdb --args qemu-system-arm.exe ...
> (gdb) r
> Co-routine is yielding to no one
> (gdb) bt
>
> Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Stefan Hajnoczi]

On Wed, Aug 28, 2013 at 04:17:24PM +0500, Taimoor Mirza wrote:
> Thanks for your reply. Below are answers
> 
> $ grep CONFIG_COROUTINE_BACKEND config-host.mak
> CONFIG_COROUTINE_BACKEND=win32
> 
> (gdb) r
> Starting program: C:\tools\qemu\qemu-system-arm.exe
> -M realview-eb -m 256 -kernel Debug\\\\KD2.out -sd fat:16:rw:C:\\testCard
> [New Thread 3836.0x404]
> [New Thread 3836.0x87c]
> [New Thread 3836.0xd14]
> [New Thread 3836.0xc58]
> [Inferior 1 (process 3836) exited with code 03]
> (gdb) bt
> No stack.

It seems that gdb on Windows does not catch abort(3) - perhaps because
SIGABRT is not used (on Linux gdb will intercept SIGABRT and return to
the gdb prompt).

So you may need to set a breakpoint on abort(3) instead:

 (gdb) b abort
 (gdb) r
 [...runs until abort(3) is called...]
 (gdb) bt

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

Hi Stefan,

Below is result of bt:

Breakpoint 1, 0x006ac304 in abort ()
(gdb) bt
#0  0x006ac304 in abort ()
#1  0x00553052 in _fu10846____stack_chk_guard () at qemu-coroutine.c:111
#2  0x0040d746 in _fu473____stack_chk_guard () at block.c:4294
#3  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
#4  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
#5  0x00414875 in bdrv_rw_co_entry (opaque=0xa90f8d8) at block.c:2172
#6  _fu836____stack_chk_guard () at block.c:2167
#7  0x004439f8 in _fu1936____stack_chk_guard () at coroutine-win32.c:57
#8  0x767dbff2 in KERNEL32!GetQueuedCompletionStatus ()
   from C:\windows\syswow64\kernel32.dll
#9  0x035e3be8 in ?? ()
#10 0x767dbfaa in KERNEL32!GetQueuedCompletionStatus ()
   from C:\windows\syswow64\kernel32.dll
#11 0x019c3c70 in ?? ()
Cannot access memory at address 0xa080000
(gdb)

Thanks,
Taimoor

On Wed, Aug 28, 2013 at 6:38 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Wed, Aug 28, 2013 at 04:17:24PM +0500, Taimoor Mirza wrote:
>> Thanks for your reply. Below are answers
>>
>> $ grep CONFIG_COROUTINE_BACKEND config-host.mak
>> CONFIG_COROUTINE_BACKEND=win32
>>
>> (gdb) r
>> Starting program: C:\tools\qemu\qemu-system-arm.exe
>> -M realview-eb -m 256 -kernel Debug\\\\KD2.out -sd fat:16:rw:C:\\testCard
>> [New Thread 3836.0x404]
>> [New Thread 3836.0x87c]
>> [New Thread 3836.0xd14]
>> [New Thread 3836.0xc58]
>> [Inferior 1 (process 3836) exited with code 03]
>> (gdb) bt
>> No stack.
>
> It seems that gdb on Windows does not catch abort(3) - perhaps because
> SIGABRT is not used (on Linux gdb will intercept SIGABRT and return to
> the gdb prompt).
>
> So you may need to set a breakpoint on abort(3) instead:
>
>  (gdb) b abort
>  (gdb) r
>  [...runs until abort(3) is called...]
>  (gdb) bt

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

I just tried it on Windows XP and its working fine there. So this
problem comes only on Windows 7.

-Taimoor

On Thu, Aug 29, 2013 at 12:10 AM, Taimoor Mirza <taimoor.mrza@gmail.com> wrote:
> Hi Stefan,
>
> Below is result of bt:
>
> Breakpoint 1, 0x006ac304 in abort ()
> (gdb) bt
> #0  0x006ac304 in abort ()
> #1  0x00553052 in _fu10846____stack_chk_guard () at qemu-coroutine.c:111
> #2  0x0040d746 in _fu473____stack_chk_guard () at block.c:4294
> #3  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
> #4  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
> #5  0x00414875 in bdrv_rw_co_entry (opaque=0xa90f8d8) at block.c:2172
> #6  _fu836____stack_chk_guard () at block.c:2167
> #7  0x004439f8 in _fu1936____stack_chk_guard () at coroutine-win32.c:57
> #8  0x767dbff2 in KERNEL32!GetQueuedCompletionStatus ()
>    from C:\windows\syswow64\kernel32.dll
> #9  0x035e3be8 in ?? ()
> #10 0x767dbfaa in KERNEL32!GetQueuedCompletionStatus ()
>    from C:\windows\syswow64\kernel32.dll
> #11 0x019c3c70 in ?? ()
> Cannot access memory at address 0xa080000
> (gdb)
>
> Thanks,
> Taimoor
>
> On Wed, Aug 28, 2013 at 6:38 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Wed, Aug 28, 2013 at 04:17:24PM +0500, Taimoor Mirza wrote:
>>> Thanks for your reply. Below are answers
>>>
>>> $ grep CONFIG_COROUTINE_BACKEND config-host.mak
>>> CONFIG_COROUTINE_BACKEND=win32
>>>
>>> (gdb) r
>>> Starting program: C:\tools\qemu\qemu-system-arm.exe
>>> -M realview-eb -m 256 -kernel Debug\\\\KD2.out -sd fat:16:rw:C:\\testCard
>>> [New Thread 3836.0x404]
>>> [New Thread 3836.0x87c]
>>> [New Thread 3836.0xd14]
>>> [New Thread 3836.0xc58]
>>> [Inferior 1 (process 3836) exited with code 03]
>>> (gdb) bt
>>> No stack.
>>
>> It seems that gdb on Windows does not catch abort(3) - perhaps because
>> SIGABRT is not used (on Linux gdb will intercept SIGABRT and return to
>> the gdb prompt).
>>
>> So you may need to set a breakpoint on abort(3) instead:
>>
>>  (gdb) b abort
>>  (gdb) r
>>  [...runs until abort(3) is called...]
>>  (gdb) bt

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

Kindly ignore last email.......it is reproducible on XP as well.

On Thu, Aug 29, 2013 at 11:31 AM, Taimoor Mirza <taimoor.mrza@gmail.com> wrote:
> I just tried it on Windows XP and its working fine there. So this
> problem comes only on Windows 7.
>
> -Taimoor
>
> On Thu, Aug 29, 2013 at 12:10 AM, Taimoor Mirza <taimoor.mrza@gmail.com> wrote:
>> Hi Stefan,
>>
>> Below is result of bt:
>>
>> Breakpoint 1, 0x006ac304 in abort ()
>> (gdb) bt
>> #0  0x006ac304 in abort ()
>> #1  0x00553052 in _fu10846____stack_chk_guard () at qemu-coroutine.c:111
>> #2  0x0040d746 in _fu473____stack_chk_guard () at block.c:4294
>> #3  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
>> #4  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
>> #5  0x00414875 in bdrv_rw_co_entry (opaque=0xa90f8d8) at block.c:2172
>> #6  _fu836____stack_chk_guard () at block.c:2167
>> #7  0x004439f8 in _fu1936____stack_chk_guard () at coroutine-win32.c:57
>> #8  0x767dbff2 in KERNEL32!GetQueuedCompletionStatus ()
>>    from C:\windows\syswow64\kernel32.dll
>> #9  0x035e3be8 in ?? ()
>> #10 0x767dbfaa in KERNEL32!GetQueuedCompletionStatus ()
>>    from C:\windows\syswow64\kernel32.dll
>> #11 0x019c3c70 in ?? ()
>> Cannot access memory at address 0xa080000
>> (gdb)
>>
>> Thanks,
>> Taimoor
>>
>> On Wed, Aug 28, 2013 at 6:38 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Wed, Aug 28, 2013 at 04:17:24PM +0500, Taimoor Mirza wrote:
>>>> Thanks for your reply. Below are answers
>>>>
>>>> $ grep CONFIG_COROUTINE_BACKEND config-host.mak
>>>> CONFIG_COROUTINE_BACKEND=win32
>>>>
>>>> (gdb) r
>>>> Starting program: C:\tools\qemu\qemu-system-arm.exe
>>>> -M realview-eb -m 256 -kernel Debug\\\\KD2.out -sd fat:16:rw:C:\\testCard
>>>> [New Thread 3836.0x404]
>>>> [New Thread 3836.0x87c]
>>>> [New Thread 3836.0xd14]
>>>> [New Thread 3836.0xc58]
>>>> [Inferior 1 (process 3836) exited with code 03]
>>>> (gdb) bt
>>>> No stack.
>>>
>>> It seems that gdb on Windows does not catch abort(3) - perhaps because
>>> SIGABRT is not used (on Linux gdb will intercept SIGABRT and return to
>>> the gdb prompt).
>>>
>>> So you may need to set a breakpoint on abort(3) instead:
>>>
>>>  (gdb) b abort
>>>  (gdb) r
>>>  [...runs until abort(3) is called...]
>>>  (gdb) bt

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Stefan Hajnoczi]

On Thu, Aug 29, 2013 at 12:10:48AM +0500, Taimoor Mirza wrote:
> Hi Stefan,
> 
> Below is result of bt:
> 
> Breakpoint 1, 0x006ac304 in abort ()
> (gdb) bt
> #0  0x006ac304 in abort ()
> #1  0x00553052 in _fu10846____stack_chk_guard () at qemu-coroutine.c:111
> #2  0x0040d746 in _fu473____stack_chk_guard () at block.c:4294
> #3  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
> #4  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
> #5  0x00414875 in bdrv_rw_co_entry (opaque=0xa90f8d8) at block.c:2172
> #6  _fu836____stack_chk_guard () at block.c:2167
> #7  0x004439f8 in _fu1936____stack_chk_guard () at coroutine-win32.c:57
> #8  0x767dbff2 in KERNEL32!GetQueuedCompletionStatus ()
>    from C:\windows\syswow64\kernel32.dll
> #9  0x035e3be8 in ?? ()
> #10 0x767dbfaa in KERNEL32!GetQueuedCompletionStatus ()
>    from C:\windows\syswow64\kernel32.dll
> #11 0x019c3c70 in ?? ()
> Cannot access memory at address 0xa080000
> (gdb)

Great, thanks.  Please post the exact git commit ID you have built, I'm
having trouble tracking down the line numbers in qemu.git/master:

  $ git rev-parse HEAD

Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Taimoor Mirza]

Hi Stefan,

I am using released 1.5.0 version from http://wiki.qemu.org/Download.
I think it should be same as commit ID
"295d81c62414a63c625fa2e78175573d4b3f5ba4"

I have observed some interesting behavior. This problem does not come
if I use MinGW with GCC version 4.7.2. I was originally using 4.6.2. I
spent some time looking at what can cause this problem and found out
an interesting thing. If I change optimization flag from O2 to O1
while building coroutine-win32 then it works fine even with 4.6.2.
For this I first built QEMU binary with default O2 flag, remove
coroutine-win32.o, changed CFLAG in makefile to O1 and rerun make.
Generated binary works fine without any problem.

Thanks,
Taimoor

On Thu, Aug 29, 2013 at 1:28 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Thu, Aug 29, 2013 at 12:10:48AM +0500, Taimoor Mirza wrote:
>> Hi Stefan,
>>
>> Below is result of bt:
>>
>> Breakpoint 1, 0x006ac304 in abort ()
>> (gdb) bt
>> #0  0x006ac304 in abort ()
>> #1  0x00553052 in _fu10846____stack_chk_guard () at qemu-coroutine.c:111
>> #2  0x0040d746 in _fu473____stack_chk_guard () at block.c:4294
>> #3  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
>> #4  0x00413cc7 in _fu805____stack_chk_guard () at block.c:2530
>> #5  0x00414875 in bdrv_rw_co_entry (opaque=0xa90f8d8) at block.c:2172
>> #6  _fu836____stack_chk_guard () at block.c:2167
>> #7  0x004439f8 in _fu1936____stack_chk_guard () at coroutine-win32.c:57
>> #8  0x767dbff2 in KERNEL32!GetQueuedCompletionStatus ()
>>    from C:\windows\syswow64\kernel32.dll
>> #9  0x035e3be8 in ?? ()
>> #10 0x767dbfaa in KERNEL32!GetQueuedCompletionStatus ()
>>    from C:\windows\syswow64\kernel32.dll
>> #11 0x019c3c70 in ?? ()
>> Cannot access memory at address 0xa080000
>> (gdb)
>
> Great, thanks.  Please post the exact git commit ID you have built, I'm
> having trouble tracking down the line numbers in qemu.git/master:
>
>   $ git rev-parse HEAD
>
> Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Stefan Hajnoczi]

On Thu, Sep 05, 2013 at 04:02:43PM +0500, Taimoor Mirza wrote:
> I am using released 1.5.0 version from http://wiki.qemu.org/Download.
> I think it should be same as commit ID
> "295d81c62414a63c625fa2e78175573d4b3f5ba4"
> 
> I have observed some interesting behavior. This problem does not come
> if I use MinGW with GCC version 4.7.2. I was originally using 4.6.2. I
> spent some time looking at what can cause this problem and found out
> an interesting thing. If I change optimization flag from O2 to O1
> while building coroutine-win32 then it works fine even with 4.6.2.
> For this I first built QEMU binary with default O2 flag, remove
> coroutine-win32.o, changed CFLAG in makefile to O1 and rerun make.
> Generated binary works fine without any problem.

Can you pastebin the output of "objdump -dr coroutine-win32.o" for both
-O2 and -O1?

Let's diff them and figure out why it breaks with -O2.

Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Stefan Hajnoczi]

On Thu, Sep 19, 2013 at 8:36 AM, Taimoor Mirza <taimoor.mrza@gmail.com> wrote:

Please use Reply-All to keep the CC list on your reply email.  This
way others on the mailing list can follow the discussion and
participate.

> O1 disassembly is at http://pastebin.com/yxWAmmmf
>
> O2 disassembly is at http://pastebin.com/v0uqHuYY

I diffed them.  The differences look reasonable, nothing obviously broken:
 * The broken -O2 version inlines qemu_coroutine_switch() into
coroutine_trampoline().
 * The broken -O2 version changes the ordering of basic blocks and
switches a few instructions for shorter/cheaper equivalents.

Taimoor: Can you try "make check" with the -O2 version?  In
particular, the test-coroutine tests may fail and that could help us
figure out which operation is broken.

Paolo: CCed you in case you want to diff the working -O1 against the
broken -O2 disassembly.  Perhaps Taimoor's gcc 4.6.2 -O2 subtly breaks
around the Windows Fiber API calls.

> On Wed, Sep 18, 2013 at 7:10 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Thu, Sep 05, 2013 at 04:02:43PM +0500, Taimoor Mirza wrote:
>>> I am using released 1.5.0 version from http://wiki.qemu.org/Download.
>>> I think it should be same as commit ID
>>> "295d81c62414a63c625fa2e78175573d4b3f5ba4"
>>>
>>> I have observed some interesting behavior. This problem does not come
>>> if I use MinGW with GCC version 4.7.2. I was originally using 4.6.2. I
>>> spent some time looking at what can cause this problem and found out
>>> an interesting thing. If I change optimization flag from O2 to O1
>>> while building coroutine-win32 then it works fine even with 4.6.2.
>>> For this I first built QEMU binary with default O2 flag, remove
>>> coroutine-win32.o, changed CFLAG in makefile to O1 and rerun make.
>>> Generated binary works fine without any problem.
>>
>> Can you pastebin the output of "objdump -dr coroutine-win32.o" for both
>> -O2 and -O1?
>>
>> Let's diff them and figure out why it breaks with -O2.
>>
>> Stefan

Re: [Qemu-devel] Problems with QEMU sdcard while using glib 2.33.8

[Paolo Bonzini]

Il 19/09/2013 10:11, Stefan Hajnoczi ha scritto:
> On Thu, Sep 19, 2013 at 8:36 AM, Taimoor Mirza <taimoor.mrza@gmail.com> wrote:
> 
> Please use Reply-All to keep the CC list on your reply email.  This
> way others on the mailing list can follow the discussion and
> participate.
> 
>> O1 disassembly is at http://pastebin.com/yxWAmmmf
>>
>> O2 disassembly is at http://pastebin.com/v0uqHuYY
> 
> I diffed them.  The differences look reasonable, nothing obviously broken:
>  * The broken -O2 version inlines qemu_coroutine_switch() into
> coroutine_trampoline().
>  * The broken -O2 version changes the ordering of basic blocks and
> switches a few instructions for shorter/cheaper equivalents.
> 
> Taimoor: Can you try "make check" with the -O2 version?  In
> particular, the test-coroutine tests may fail and that could help us
> figure out which operation is broken.
> 
> Paolo: CCed you in case you want to diff the working -O1 against the
> broken -O2 disassembly.  Perhaps Taimoor's gcc 4.6.2 -O2 subtly breaks
> around the Windows Fiber API calls.

Taimoor, can you please try compiling coroutine-win32.c at
-O2 but without the -fstack-protector-all compiler option?  If it works,
please try with -fstack-protector instead of -fstack-protector-all.

Also (separately from the above experiment), can you try this patch and
see if it breaks/works at -O1 and -O2?  Then, independent of the result,
please post -O1 and -O2 assembly outputs with the patch.

diff --git a/coroutine-win32.c b/coroutine-win32.c
index edc1f72..3b10189 100644
--- a/coroutine-win32.c
+++ b/coroutine-win32.c
@@ -55,7 +55,13 @@ static void CALLBACK coroutine_trampoline(void *co_)
 
     while (true) {
         co->entry(co->entry_arg);
-        qemu_coroutine_switch(co, co->caller, COROUTINE_TERMINATE);
+        Coroutine *to_ = co->caller;
+        CoroutineWin32 *from = DO_UPCAST(CoroutineWin32, base, co);
+        CoroutineWin32 *to = DO_UPCAST(CoroutineWin32, base, to_);
+
+        current = to_;
+        to->action = COROUTINE_TERMINATE;
+        SwitchToFiber(to->fiber);
     }
 }
 

Thanks,

Paolo