[Qemu-devel] Pentium D with guest Ubuntu 6.06 server kernel panic with kqemu

[Qemu-devel] Pentium D with guest Ubuntu 6.06 server kernel panic with kqemu

[R. Armiento]

Hi,

Thanks to everyone involved in the qemu project, it is great software.

I have some issues with kqemu in kernel mode;

Host is running qemu-0.8.1 or qemu-cvs-2006-07-05_23, and 
kqemu-1.3.0pre9. Host OS is Ubuntu 6.06 server.

To reproduce: Start qemu as:
   qemu -kernel-kqemu -cdrom ubuntu-6.06-server-i386.iso -hda test.img 
-boot d
select 'Install to the hard disk'. Output is as follows:
------------------------------
Uncompressing Linux... Ok, booting the kernel
ACPI: Unable to locate RSDP
PCI: PIIX3: Enabling Passive Release on 0000:00:01.0
Invalid operand: 0000 [#1]
PREEMPT
Modules linked in:
CPU 0
EIP: 0060:[<c01011c6>] Not tainted VLI
EFLAGS: 00010246 (2.6.15-23-386)
EIP is at mwait_idle+0x16/0x30
[... register dump, etc ...]
<0>Kernel panic - not syncing: Attempted to kill the id
-----------------------------

Running qemu without '-kernel-kqemu' OR booting the guest install cd 
with kernel option 'idle=poll' or 'idle=halt' avoids the kernel panic.
(idle=halt is much faster than idle=poll, though).

The error looks very similar to the one reported here:
   http://www.mail-archive.com/qemu-devel@nongnu.org/msg03964.html
But I believe that reported issue should not appear in recent qemu, 
since SSE3 is now emulated (right?). (At least the patch in the end of 
that thread seems to already be included in the sources?)

So, my hypothesis is that there is some other feature that appears in my 
host CPUID, which the booting linux image tries to make use of, but 
which qemu does not emulate.

The host CPU is a dual-core Pentium D. Here is relevant lines from 
/proc/cpuinfo:
------------------------------
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Pentium(R) D CPU 3.00GHz
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
pni monitor ds_cpl vmx est cid cx16 xtpr lahf_lm
-----------------------------

Any ideas?

Best regards,
Rickard
http://www.dewmill.com

Re: [Qemu-devel] Pentium D with guest Ubuntu 6.06 server kernel panic with kqemu

[Joachim Henke]

Yes, this patch was included, but it doesn't solve that problem. As  
this message [http://www.mail-archive.com/qemu-devel@nongnu.org/ 
msg03972.html] states, the 'monitor' and the 'mwait' instructions  
have not been added. But your guest OS assumes them to be present,  
because your host cpu has the MONITOR flag set in CPUID.

Jo.

R. Armiento wrote:
> The error looks very similar to the one reported here:
>   http://www.mail-archive.com/qemu-devel@nongnu.org/msg03964.html
> But I believe that reported issue should not appear in recent qemu,  
> since SSE3 is now emulated (right?). (At least the patch in the end  
> of that thread seems to already be included in the sources?)
>
> So, my hypothesis is that there is some other feature that appears  
> in my host CPUID, which the booting linux image tries to make use  
> of, but which qemu does not emulate.

-- 
Joachim Henke
http://he-jo.net/

Re: [Qemu-devel] Pentium D with guest Ubuntu 6.06 server kernel panic with kqemu

[Jens Axboe]

On Fri, Jul 07 2006, Joachim Henke wrote:
> Yes, this patch was included, but it doesn't solve that problem. As  
> this message [http://www.mail-archive.com/qemu-devel@nongnu.org/ 
> msg03972.html] states, the 'monitor' and the 'mwait' instructions  
> have not been added. But your guest OS assumes them to be present,  
> because your host cpu has the MONITOR flag set in CPUID.
> 
> Jo.
> 
> R. Armiento wrote:
> >The error looks very similar to the one reported here:
> >  http://www.mail-archive.com/qemu-devel@nongnu.org/msg03964.html
> >But I believe that reported issue should not appear in recent qemu,  
> >since SSE3 is now emulated (right?). (At least the patch in the end  
> >of that thread seems to already be included in the sources?)
> >
> >So, my hypothesis is that there is some other feature that appears  
> >in my host CPUID, which the booting linux image tries to make use  
> >of, but which qemu does not emulate.

Until that gets fixed up, you can boot with idle=halt.

-- 
Jens Axboe

Re: [Qemu-devel] Pentium D with guest Ubuntu 6.06 server kernel panic with kqemu

[R. Armiento]

R. Armiento wrote:
>> The error looks very similar to the one reported here:
>>   http://www.mail-archive.com/qemu-devel@nongnu.org/msg03964.html
>> But I believe that reported issue should not appear in recent qemu, 
>> since SSE3 is now emulated (right?). (At least the patch in the end of 
>> that thread seems to already be included in the sources?)

Joachim Henke wrote:
> Yes, this patch was included, but it doesn't solve that problem. As this 
> message 
> [http://www.mail-archive.com/qemu-devel@nongnu.org/msg03972.html] 
> states, the 'monitor' and the 'mwait' instructions have not been added. 
> But your guest OS assumes them to be present, because your host cpu has 
> the MONITOR flag set in CPUID.

I see; so the issue I reported is in fact the exact same issue as the 
one reported in march. And the situation is that we are waiting for some 
kind soul to implement the monitor/mwait instructions (and according to 
Fabrice in that thread "doing nops should suffice").

(While I am able to boot with idle=halt, I'm still worried that some 
other software than Linux idle function will happen to also use the 
wrongly declared MONITOR feautre and crash the emulated host...)

Best regards,
Rickard

[Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction

[Joachim Henke]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

Could you please check, if the attached patch works for you? A quick  
test showed that Linux boots fine with the MONITOR flag set now.

This patch adds 'monitor' and 'mwait' as nops, as suggested by Fabrice.


Regards,
Jo.

Am 06.07.2006 um 17:13 schrieb R. Armiento:
> Uncompressing Linux... Ok, booting the kernel
> ACPI: Unable to locate RSDP
> PCI: PIIX3: Enabling Passive Release on 0000:00:01.0
> Invalid operand: 0000 [#1]
> PREEMPT
> Modules linked in:
> CPU 0
> EIP: 0060:[<c01011c6>] Not tainted VLI
> EFLAGS: 00010246 (2.6.15-23-386)
> EIP is at mwait_idle+0x16/0x30
> [... register dump, etc ...]
> <0>Kernel panic - not syncing: Attempted to kill the id

-- 
Joachim Henke
http://he-jo.net/

[-- Attachment #2: mwait.diff.gz --]
[-- Type: application/x-gzip, Size: 787 bytes --]

Re: [Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction

[maestro]

Am Freitag, den 07.07.2006, 14:30 +0200 schrieb Joachim Henke:
> Could you please check, if the attached patch works for you? A quick  
> test showed that Linux boots fine with the MONITOR flag set now.
> 
> This patch adds 'monitor' and 'mwait' as nops, as suggested by Fabrice.
> 
hello just tested the patch against 0.8.1 and current cvs and at least
here it does not work:
still 
Kernel panic - not syncing: Attempted to kill init!

im on a pentium D with ubuntu 6.06server as guest os (debian sarge host)

cheers
m

btw: when i patch < mwait.diff in the qemu-src directory patch cannot
find the files to patch and asks me for their location - did i do
anything wrong?

Re: [Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction

[Joachim Henke]

Try

zcat mwait.diff.gz | patch -p0

in the source directory.

Am 07.07.2006 um 14:57 schrieb maestro:
> btw: when i patch < mwait.diff in the qemu-src directory patch cannot
> find the files to patch and asks me for their location - did i do
> anything wrong?

-- 
Joachim Henke
http://he-jo.net/

Re: [Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction

[Joachim Henke]

Maybe there are some further issues in your setup. Could you please  
provide more detailed kernel messages (regarding the panic)?

Thanks,
Jo.

maestro wrote:
> hello just tested the patch against 0.8.1 and current cvs and at least
> here it does not work:
> still
> Kernel panic - not syncing: Attempted to kill init!

-- 
Joachim Henke
http://he-jo.net/

[Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction (update)

[Joachim Henke]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

The last patch I sent, enabled the MONITOR cpuid flag for testing  
purposes. So Linux guests won't execute the 'hlt' instruction  
anymore. Since we don't really emulate an 'mwait' instruction, this  
leads to high cpu usage on the host, even when the guest cpu is idle.

Please use the updated patch below.


Sorry,
Jo.


Joachim Henke wrote:
> Could you please check, if the attached patch works for you? A  
> quick test showed that Linux boots fine with the MONITOR flag set now.
>
> This patch adds 'monitor' and 'mwait' as nops, as suggested by  
> Fabrice.

-- 
Joachim Henke
http://he-jo.net/

[-- Attachment #2: mwait.diff.gz --]
[-- Type: application/x-gzip, Size: 631 bytes --]

[Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction (update)

[Joachim Henke]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

The last patch I sent, enabled the MONITOR cpuid flag for testing  
purposes. So Linux guests won't execute the 'hlt' instruction  
anymore. Since we don't really emulate an 'mwait' instruction, this  
leads to high cpu usage on the host, even when the guest cpu is idle.

Please use the updated patch attached below.


Sorry,
Jo.


Joachim Henke wrote:
> Could you please check, if the attached patch works for you? A  
> quick test showed that Linux boots fine with the MONITOR flag set now.
>
> This patch adds 'monitor' and 'mwait' as nops, as suggested by  
> Fabrice.

-- 
Joachim Henke
http://he-jo.net/

[-- Attachment #2: mwait.diff.gz --]
[-- Type: application/x-gzip, Size: 631 bytes --]

Re: [Qemu-devel] [PATCH] add 'monitor' and 'mwait' instruction (update)

[R. Armiento]

Hi,

Joachim Henke wrote:
> Please use the updated patch attached below.

Great work! The patch fixes the kernel panic for me. Thank you.

However, as you probably know, despite not declaring MONITOR in qemu, 
kqemu sees MONITOR on the host processor and Linux CPU usage will still 
be 100%, even with your updated patch. So even with your patch applied 
one should use the 'idle=halt' kernel parameter when booting Linux with 
-kernel-kqemu on newer processors.

While your patch resolves the problem and is great for my setup, I 
suppose it is problematic. If MONITOR is not declared by qemu then still 
serving mwait 'correctly' is not faithful emulation (right?). To make 
sure this does not break anything else, I would guess that either mwait 
has to be emulated better so that it is ok to let qemu declare MONITOR 
support; OR as a temporary workaround make the patch only serve mwait if 
-kernel-kqemu is enabled and host CPUID supports MONITOR.

The whole thing with CPUID not being trappable seems like a can of worms 
to me, partly breaking hardware abstraction. I am totally ignorant of 
the inner workings of kqemu; but if it was somehow possible to "rewrite" 
CPUID into something trappable and handling it correctly, that would 
give a much more satisfying solution to this problem.

Rickard

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Joachim Henke]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

R. Armiento wrote:
> However, as you probably know, despite not declaring MONITOR in  
> qemu, kqemu sees MONITOR on the host processor and Linux CPU usage  
> will still be 100%, even with your updated patch. So even with your  
> patch applied one should use the 'idle=halt' kernel parameter when  
> booting Linux with -kernel-kqemu on newer processors.

Yes, this is obviously true. To lower the cpu usage, you can  
additionally apply the attached patch, which makes 'mwait' similar to  
'hlt'. This ugly hack seems to be sufficient at least for Linux.

> While your patch resolves the problem and is great for my setup, I  
> suppose it is problematic. If MONITOR is not declared by qemu then  
> still serving mwait 'correctly' is not faithful emulation (right?).  
> To make sure this does not break anything else, I would guess that  
> either mwait has to be emulated better so that it is ok to let qemu  
> declare MONITOR support; OR as a temporary workaround make the  
> patch only serve mwait if -kernel-kqemu is enabled and host CPUID  
> supports MONITOR.

Yes, the emulation should be done in a cleaner way. But to me it  
seems impossible to do real mwait emulation in user space. Probably  
someone else will come up with a better solution. Anyway, your last  
suggestion looks good.

-- 
Joachim Henke
http://he-jo.net/

[-- Attachment #2: mwait_hlt.diff --]
[-- Type: application/octet-stream, Size: 439 bytes --]

--- target-i386/translate.c
+++ target-i386/translate.c
@@ -5587,3 +5587,9 @@
                 case 0: /* monitor */
+                    break;
                 case 1: /* mwait */
+                    if (s->cc_op != CC_OP_DYNAMIC)
+                        gen_op_set_cc_op(s->cc_op);
+                    gen_jmp_im(s->pc - s->cs_base);
+                    gen_op_hlt();
+                    s->is_jmp = 3;
                     break;

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[R. Armiento]

Hi,

Again, thank you for helping out with updated patches, it is much 
appreciated.

Joachim Henke wrote:
> R. Armiento wrote:
>> So even with your patch applied one should use the 'idle=halt'
>> kernel parameter when booting Linux with -kernel-kqemu on newer
>> processors. [...]
> To lower the cpu usage, you can additionally apply the attached
> patch, which makes 'mwait' similar to 'hlt'. This ugly hack seems to
> be sufficient at least for Linux.

Is this hack really 'safe'? I don't claim to know much about modern x86 
instructions, but some googling tells me that mwait is supposed to wake 
on a monitored memory write (but is allowed to wake up earlier, hence it 
is acceptable but CPU consuming to emulate it with a nop). Couldn't 
there be situations where someone depends on mwait waking up without 
there being an event that wakes hlt? Or are we sure qemu's hlt will 
happen to wake up anyway?

> Yes, the emulation should be done in a cleaner way. But to me it seems 
> impossible to do real mwait emulation in user space. 

Again, excuse my lack of knowledge of the internals of qemu and kqemu. 
If 'hlt' can be emulated to give CPU time back to the host OS until an 
interrupt occurs in the guest; then it is not obvious why mwait couldn't 
be simulated in a similar way, only with the addition that qemu also 
restarts guest CPU execution should there be writes in monitored memory. 
While I have no idea of how much work it would be, it would much 
surprise me if this is truly un-doable, at least for non-kqemu emulation.

Rickard

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Joachim Henke]

R. Armiento wrote:
> Is this hack really 'safe'? I don't claim to know much about modern  
> x86 instructions, but some googling tells me that mwait is supposed  
> to wake on a monitored memory write (but is allowed to wake up  
> earlier, hence it is acceptable but CPU consuming to emulate it  
> with a nop). Couldn't there be situations where someone depends on  
> mwait waking up without there being an event that wakes hlt? Or are  
> we sure qemu's hlt will happen to wake up anyway?

Currently the Linux kernel simply uses monitor/mwait as a faster  
'hlt' replacement, so it should be "safe" there. I don't know about  
other guest OSs. Anyway, I proposed this hack only as a quick  
"solution" for local usage.

> Again, excuse my lack of knowledge of the internals of qemu and  
> kqemu. If 'hlt' can be emulated to give CPU time back to the host  
> OS until an interrupt occurs in the guest; then it is not obvious  
> why mwait couldn't be simulated in a similar way, only with the  
> addition that qemu also restarts guest CPU execution should there  
> be writes in monitored memory. While I have no idea of how much  
> work it would be, it would much surprise me if this is truly un- 
> doable, at least for non-kqemu emulation.

Problem is, at the moment I've no idea, how we could achieve this  
memory monitoring in a safe and simple way in user space. But, as you  
already told, we only need monitor/mwait emulation, when using kernel- 
kqemu _and_ having the MONITOR flag set in the host cpuid. So I'm  
sure, Fabrice would be able to do the trick - maybe by using the  
hosts monitor/mwait instructions.

-- 
Joachim Henke
http://he-jo.net/

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[R. Armiento]

> R. Armiento wrote:
>> Couldn't there be situations where someone depends on mwait waking up 
>> without there being an event that wakes hlt? Or are we sure qemu's hlt 
>> will happen to wake up anyway?

Joachim Henke wrote:
> Currently the Linux kernel simply uses monitor/mwait as a faster 'hlt' 
> replacement, so it should be "safe" there. I don't know about other 
> guest OSs. Anyway, I proposed this hack only as a quick "solution" for 
> local usage.

I realize the last version of the patch was just a "quick hack" to solve 
the problem with Linux's idle function. But I just don't see how you can 
postulate that mwait is not used anywhere else in the Linux kernel? 
There are many kernel modules; some being closed source. And if an mwait 
lurks somewhere in your kernel, you might get a really obscure breakage 
that will be a pain to debug...

> Problem is, at the moment I've no idea, how we could achieve this memory 
> monitoring in a safe and simple way in user space. 

I'm trying to read up on monitor and mwait. Apparently mwait puts the 
processor in low-power wait mode, waiting for a memory write in some 
select area defined by monitor; and as I am new to this I'm not sure if 
I have understood all sources from where such a memory write can come 
from while the processor is asleep. One source, I suppose, is from other 
processors in an SMP setup? Another source may be DMA? Does this mean 
that it is safe to emulate wmait as hlt if neiter SMP or DMA is used? 
(Qemu hardware doesn't support DMA, right?)

Rickard

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Paul Brook]

> > Problem is, at the moment I've no idea, how we could achieve this memory
> > monitoring in a safe and simple way in user space.
>
> I'm trying to read up on monitor and mwait. Apparently mwait puts the
> processor in low-power wait mode, waiting for a memory write in some
> select area defined by monitor; and as I am new to this I'm not sure if
> I have understood all sources from where such a memory write can come
> from while the processor is asleep. One source, I suppose, is from other
> processors in an SMP setup? Another source may be DMA? Does this mean
> that it is safe to emulate wmait as hlt if neiter SMP or DMA is used?
> (Qemu hardware doesn't support DMA, right?)

qemu hardware does support DMA, but I don't think this matters.
By my reading DMA writes don't need to wake mwait.
The exact wording is "store operation", which I'd expect to mean execution of 
a store instruction (by a different CPU). So for UP systems mwait can be 
implemented the same as HLT, except that it raises an exception if state 
hasn't been initialized by monitor.

Paul

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Jamie Lokier]

Paul Brook wrote:
> qemu hardware does support DMA, but I don't think this matters.
> By my reading DMA writes don't need to wake mwait.
> The exact wording is "store operation", which I'd expect to mean
> execution of a store instruction (by a different CPU).

Hmm.  x86 CPUs snoop writes by DMA as well as other CPUs, and I'd
expect write snooping to be the mechanism used by 'mwait'.

It would be good to test on a real system whether DMA wakes 'mwait'.

I wouldn't expect many OSes to use this, but I have seen tuned
networking OSes which would benefit from 'mwait' on a network DMA ring
descriptor.

-- Jamie

Re: [Qemu-devel] 'monitor' and 'mwait' instruction

[Joachim Henke]

Ok, I forgot about external kernel modules or patches, as I don't use  
any. But in plain Linux 2.6.17 mwait is only used in the function  
mwait_idle()

R. Armiento wrote:
> But I just don't see how you can postulate that mwait is not used  
> anywhere else in the Linux kernel? There are many kernel modules;  
> some being closed source. And if an mwait lurks somewhere in your  
> kernel, you might get a really obscure breakage that will be a pain  
> to debug...

-- 
Joachim Henke
http://he-jo.net/

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Jamie Lokier]

R. Armiento wrote:
> I'm not sure if I have understood all sources from where such a
> memory write can come from while the processor is asleep. One
> source, I suppose, is from other processors in an SMP setup? Another
> source may be DMA? Does this mean that it is safe to emulate wmait
> as hlt if neiter SMP or DMA is used?

Yes, and yes.  More specifically, DMA to RAM (DMA from RAM to device
doesn't matter).

>  (Qemu hardware doesn't support DMA, right?)

I'm pretty sure it does.

-- Jamie

Re: [Qemu-devel] add 'monitor' and 'mwait' instruction (update)

[Jamie Lokier]

Joachim Henke wrote:
> Currently the Linux kernel simply uses monitor/mwait as a faster  
> 'hlt' replacement, so it should be "safe" there. I don't know about  
> other guest OSs. Anyway, I proposed this hack only as a quick  
> "solution" for local usage.

As long as there's only one CPU and 'mwait' isn't used to wait for a
_device_ writing to memory, then giving it the same behaviour as 'hlt'
will be fine.  In that case, the only way the memory could be written
is as a result of an interrupt, which wakes 'hlt' too.

If there's more than one CPU, or the OS uses 'mwait' to wait for a
device writing to memory, then you can emulate it accurately by
trapping access to the page containing the waited-on location(s) using
the MMU, although the performance may or may not be adequate.

-- Jamie