[Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Марк Коренберг]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Hello developers.

I'm developer of ideco software product (www.ideco-software.ru)
and our products is run perfectly on vmware, but strange behaviour in qemu.

I have qemu 0.9.0 for windows with kqemu accelerator used.

i have screenshots attached. I think race condition somewhere, as the
the bug appear in different places.

The scr01.bmp - string " === qwe ===" is specially inserted in script
just before
/sbin/hwclock --localtime --hctosys
command after which kernel panic occurs.
If i test this command just after kernel booting - it works fine.
Also. our kernel doesn't support modules.

The scr202.bmp appear when md5 check of our system scripts failed (as
the string "===qwe===" inserted by me)  and bash asks user to confirm
booting:

read answer
if [ "$answer" = "yes" ]; then ........

so i have no idea why kernel panic occurs.
PLEASE contact with me by email socketpair@gmail.com.
I can not use maillists and can not use kernel debugger.

Thanks.
-- 
Segmentation fault

[-- Attachment #2: scr.zip --]
[-- Type: application/zip, Size: 26118 bytes --]

Re: [Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Mulyadi Santosa]

Hi...

Trying to deliver a help...

On Jan 17, 2008 6:50 PM, Марк Коренберг <socketpair@gmail.com> wrote:
> Hello developers.
>
> I'm developer of ideco software product (www.ideco-software.ru)
> and our products is run perfectly on vmware, but strange behaviour in qemu.

OK, that could means anything...

> I have qemu 0.9.0 for windows with kqemu accelerator used.

1st rule, try to reproduce the problem without kqemu at all. 2nd,
upgrade to 0.9.1, even better, try CVS version.

> i have screenshots attached. I think race condition somewhere, as the
> the bug appear in different places.

Apart from that, interesting to see that in both screen capture, call
traces (EIP) are same. Registers...AFAICT, also same. So, probably
they hit the same function..and from the EIP, it happens when landed
into kernel space.

> The scr01.bmp - string " === qwe ===" is specially inserted in script
> just before
> /sbin/hwclock --localtime --hctosys
> command after which kernel panic occurs.
> If i test this command just after kernel booting - it works fine.
> Also. our kernel doesn't support modules.

I suspect implementation of get/settimeofday somehow conflicts with
kqemu assumptions. Or it could be a bug in rtc driver of qemu. But
like I have mentioned above, it might already been fixed in latest
release.

> The scr202.bmp appear when md5 check of our system scripts failed (as
> the string "===qwe===" inserted by me)  and bash asks user to confirm
> booting:

What do you mdsum against?

Ultimate debugging could be done by dumping resulting translation,
check qemu-user and qemu-tech doc first. We'll try to help further if
you are still stuck.

regards,

Mulyadi.

Re: [Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Mulyadi Santosa]

Hi..

On Jan 18, 2008 5:10 PM, Марк Коренберг <socketpair@gmail.com> wrote:
> Thanks.
>
> I reproduced the same bugs with kqemu disabled on md5checking
> but without kqemu hwclock work correctly!
At least we got suspect: kqemu.

> very strange... why md5checking generates bug ?

I really don't have any idea. Maybe md5sum is doing many integer
computation/hashing, thus hitting floating point bug? Everybody, have
a clue? or maybe it's doing map/unmapping in large volume?

> I don't think that there is defference in RTC driver between qemu
> version. i'm always read changelogs.

Hm, ok.

> md5sum check - is running /usr/bin/md5sum with list of files. nothing
> special. just massive IO operations. bug appear after md5sums
> complete. just after "echo" in bash and before "read answer" command.
>
> if ! md5sums ... ; then
>  echo "(yes/no)?"
>  #bug appear here
>  read answer
>  if [ "$answer" != yes ]; then
>   .....
>
>
> At a time writing this message, i found, that ksyms or ksymoops should
> give huge help. But we use 2.4.24 kernel without modularity enabled
> (our system is embedded and modules functionality is not needed in the
> kernel). so i can't get ksyms and ksymoops to work.
> some manuals say me to use /dev/ksyms, some of them /proc/ksyms and
> also /proc/kallsyms. Maybe /proc/ksyms should be enabled in kernel
> anyhow ?
> where to read about ksyms and _simple_ debugging ?

Here is what I used to do:
Compile the kernel with -g flags. You can do it manually by adding -g
to CFLAGS in Makefile. I remember there is other variant of CFLAGS
there...just add it to all of them. Also, don't enable
optimization..simply throw out -O flag

it will create vmlinux filled with debugging symbol. use it with gdb
to find out the address of a function/variable.

is it helpful for you?

regards,

Mulyadi.

Re: [Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Mulyadi Santosa]

Hi...

Pls don't forget to CC qemu-devel too...

On Jan 18, 2008 6:01 PM, Марк Коренберг <socketpair@gmail.com> wrote:
> I'm not familiar with gdb. but reading ksyms manpage i see that this
> is much simpler to get info from ksyms file, or using ksymoops tool
> rather using gdb.
basically using ksymoops, it will be easier for you to follow stack
trace. But, IMHO, you can do very much the same armed with vmlinux as
a result of gcc -g... just check the nearest symbol address compared
to EIP showed in stack trace.

> If I compile kernel with -g flag, will it print more detailed info in
> "oops!" output ?

Nope.

> Alos, i noticed, that even kqemu enabled, hwclock work fine if run in
> commandline. but not in automatic boot-script. i think some race
> condition or other unknown condition exist. how to debug ? It's very
> easy to reproduce bug. what can i do to detect the problem?

Ehm, I don't know about this one. How about booting in runlevel 1 ?

regards,

Mulyadi.

Re: [Qemu-devel] The linux-based system runs perfectly on vmware, but kernel panic on qemu

[Mulyadi Santosa]

Hi..

On Jan 18, 2008 6:39 PM, Марк Коренберг <socketpair@gmail.com> wrote:
> Ok. i'll try -g, gdb, ksyms and additional sleep command in script
> later. (tomorrow).
> When i detect source of the bug, i will write to qemu developers.
> I you know, who also can help me, please notify him. Now, before i
> found bug, i'll write to you exclusively. i don't want to bother
> developers by specifying unknown bug conditions.

Sorry if I am being honest...

Well, it's not a matter of who is bugging who, but the possibility
that it's really qemu bug. By reporting your progress, you increase
your chance for getting further help from the community. Also, at the
same time you contribute something for qemu development. In my side, I
try to help at my best, but don't expect too much from me. I am not so
knowledgeable about Qemu internals.

regards,

Mulyadi