[Qemu-devel] Breakage with local APIC routing

[Qemu-devel] Breakage with local APIC routing

[Johannes Schindelin]

Hi,

due to the change in revision 3371 (well, at that time, CVS was used, 
which was no better than Subversion) installation of win64 is broken in 
QEmu.  The commit message reads like this:

	Don't route PIC interrupts through the local APIC if the local 
	APIC config says so. By Ari Kivity.

A bit of research showed that the patch was actually originally from Qing 
He, but he told me privately that the part that actually broke win64 (the 
removal of the call to cpu_reset_interrupt(), as opposed to moving that 
call into the "else" condition) was not part of his patch.

Unfortunately, a lot has been done to the APIC handling in the meantime, 
so it is not a simple matter of a revert.

Being a complete idiot when it comes to APICs, I have no clue how to fix 
the issue.

However, I am quite willing to test whatever patch is thrown at me.

Can somebody help?

Thanks,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> due to the change in revision 3371 (well, at that time, CVS was used, 
> which was no better than Subversion) installation of win64 is broken in 
> QEmu.  The commit message reads like this:
> 
> 	Don't route PIC interrupts through the local APIC if the local 
> 	APIC config says so. By Ari Kivity.
> 
> A bit of research showed that the patch was actually originally from Qing 
> He, but he told me privately that the part that actually broke win64 (the 
> removal of the call to cpu_reset_interrupt(), as opposed to moving that 
> call into the "else" condition) was not part of his patch.
> 
> Unfortunately, a lot has been done to the APIC handling in the meantime, 
> so it is not a simple matter of a revert.
> 
> Being a complete idiot when it comes to APICs, I have no clue how to fix 
> the issue.
> 
> However, I am quite willing to test whatever patch is thrown at me.
> 
> Can somebody help?

I recalled some earlier post on this which claimed to fix the issue and
found it in the archive:

http://permalink.gmane.org/gmane.comp.emulators.qemu/25415

However, no one replied on this, and I'm right now not sure if the fix
is OK for all cases. But it is a starting point for new discussion about
what is actually borken here.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

Jan Kiszka wrote:
> Johannes Schindelin wrote:
>> Hi,
>>
>> due to the change in revision 3371 (well, at that time, CVS was used, 
>> which was no better than Subversion) installation of win64 is broken in 
>> QEmu.  The commit message reads like this:
>>
>> 	Don't route PIC interrupts through the local APIC if the local 
>> 	APIC config says so. By Ari Kivity.
>>
>> A bit of research showed that the patch was actually originally from Qing 
>> He, but he told me privately that the part that actually broke win64 (the 
>> removal of the call to cpu_reset_interrupt(), as opposed to moving that 
>> call into the "else" condition) was not part of his patch.
>>
>> Unfortunately, a lot has been done to the APIC handling in the meantime, 
>> so it is not a simple matter of a revert.
>>
>> Being a complete idiot when it comes to APICs, I have no clue how to fix 
>> the issue.
>>
>> However, I am quite willing to test whatever patch is thrown at me.
>>
>> Can somebody help?
> 
> I recalled some earlier post on this which claimed to fix the issue and
> found it in the archive:
> 
> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> 
> However, no one replied on this, and I'm right now not sure if the fix
> is OK for all cases. But it is a starting point for new discussion about
> what is actually borken here.

This might not be the last word, but my feeling that it's closer to the
actual issue than the older fix. Please give it a try. Comments welcome.

--------

Ensure that PIC-delivered IRQs are properly de-asserted in case the APIC
is in EXTINT or FIXED mode (with level-triggering selected) on LINT0.
Should fix Win64 boot issues. This patch also cleans up a bit the
interface between PIC and APIC, making apic_local_deliver private again.

Signed-off-by: Jan Kiszka <jan.kiszka@web.de>
---
 hw/apic.c |   23 ++++++++++++++++++++++-
 hw/pc.c   |    5 +----
 hw/pc.h   |    4 +---
 3 files changed, 24 insertions(+), 8 deletions(-)

Index: b/hw/apic.c
===================================================================
--- a/hw/apic.c
+++ b/hw/apic.c
@@ -166,7 +166,7 @@ static inline void reset_bit(uint32_t *t
     tab[i] &= ~mask;
 }
 
-void apic_local_deliver(CPUState *env, int vector)
+static void apic_local_deliver(CPUState *env, int vector)
 {
     APICState *s = env->apic_state;
     uint32_t lvt = s->lvt[vector];
@@ -197,6 +197,27 @@ void apic_local_deliver(CPUState *env, i
     }
 }
 
+void apic_deliver_pic_intr(CPUState *env, int level)
+{
+    if (level)
+        apic_local_deliver(env, APIC_LVT_LINT0);
+    else {
+        APICState *s = env->apic_state;
+        uint32_t lvt = s->lvt[APIC_LVT_LINT0];
+
+        switch ((lvt >> 8) & 7) {
+        case APIC_DM_FIXED:
+            if (!(lvt & APIC_LVT_LEVEL_TRIGGER))
+                break;
+            reset_bit(s->irr, lvt & 0xff);
+            /* fall through */
+        case APIC_DM_EXTINT:
+            cpu_reset_interrupt(env, CPU_INTERRUPT_HARD);
+            break;
+        }
+    }
+}
+
 #define foreach_apic(apic, deliver_bitmask, code) \
 {\
     int __i, __j, __mask;\
Index: b/hw/pc.c
===================================================================
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -118,12 +118,9 @@ static void pic_irq_request(void *opaque
 {
     CPUState *env = first_cpu;
 
-    if (!level)
-        return;
-
     while (env) {
         if (apic_accept_pic_intr(env))
-            apic_local_deliver(env, APIC_LINT0);
+            apic_deliver_pic_intr(env, level);
         env = env->next_cpu;
     }
 }
Index: b/hw/pc.h
===================================================================
--- a/hw/pc.h
+++ b/hw/pc.h
@@ -40,11 +40,9 @@ void irq_info(void);
 /* APIC */
 typedef struct IOAPICState IOAPICState;
 
-#define APIC_LINT0	3
-
 int apic_init(CPUState *env);
 int apic_accept_pic_intr(CPUState *env);
-void apic_local_deliver(CPUState *env, int vector);
+void apic_deliver_pic_intr(CPUState *env, int level);
 int apic_get_interrupt(CPUState *env);
 IOAPICState *ioapic_init(void);
 void ioapic_set_irq(void *opaque, int vector, int level);

[Qemu-devel] Re: Breakage with local APIC routing

[Johannes Schindelin]

Hi,

On Wed, 13 Aug 2008, Jan Kiszka wrote:

> Jan Kiszka wrote:
> > Johannes Schindelin wrote:
> >>
> >> due to the change in revision 3371 (well, at that time, CVS was used, 
> >> which was no better than Subversion) installation of win64 is broken 
> >> in QEmu.  The commit message reads like this:
> >>
> >> 	Don't route PIC interrupts through the local APIC if the local 
> >> 	APIC config says so. By Ari Kivity.
> >>
> >> A bit of research showed that the patch was actually originally from 
> >> Qing He, but he told me privately that the part that actually broke 
> >> win64 (the removal of the call to cpu_reset_interrupt(), as opposed 
> >> to moving that call into the "else" condition) was not part of his 
> >> patch.
> >>
> >> Unfortunately, a lot has been done to the APIC handling in the 
> >> meantime, so it is not a simple matter of a revert.
> >>
> >> Being a complete idiot when it comes to APICs, I have no clue how to 
> >> fix the issue.
> >>
> >> However, I am quite willing to test whatever patch is thrown at me.
> >>
> >> Can somebody help?
> > 
> > I recalled some earlier post on this which claimed to fix the issue 
> > and found it in the archive:
> > 
> > http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> > 
> > However, no one replied on this, and I'm right now not sure if the fix 
> > is OK for all cases. But it is a starting point for new discussion 
> > about what is actually borken here.
> 
> This might not be the last word, but my feeling that it's closer to the
> actual issue than the older fix. Please give it a try. Comments welcome.

I could only quickly test your change, not work on it, but the issue is 
the same.  It seems to run into an endless loop, waiting to be woken up by 
an interrupt (although the latter is just my guess).

Thanks for your help,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Johannes Schindelin]

Hi,

On Wed, 13 Aug 2008, Jan Kiszka wrote:

> Johannes Schindelin wrote:
> > 
> > due to the change in revision 3371 (well, at that time, CVS was used, 
> > which was no better than Subversion) installation of win64 is broken 
> > in QEmu.  The commit message reads like this:
> > 
> > 	Don't route PIC interrupts through the local APIC if the local 
> > 	APIC config says so. By Ari Kivity.
> 
> I recalled some earlier post on this which claimed to fix the issue and 
> found it in the archive:
> 
> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415

I tried this, and it changes the symptoms, indeed.  Instead of an endless 
loop, it results in a bluescreen.

As the OP said that it worked for him, I guess it is either in commits 
that came after his post, or in my add-on patches.

Hopefully I will find some time to work more on this bug.

Ciao,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> On Wed, 13 Aug 2008, Jan Kiszka wrote:
> 
>> Johannes Schindelin wrote:
>>> due to the change in revision 3371 (well, at that time, CVS was used, 
>>> which was no better than Subversion) installation of win64 is broken 
>>> in QEmu.  The commit message reads like this:
>>>
>>> 	Don't route PIC interrupts through the local APIC if the local 
>>> 	APIC config says so. By Ari Kivity.
>> I recalled some earlier post on this which claimed to fix the issue and 
>> found it in the archive:
>>
>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> 
> I tried this, and it changes the symptoms, indeed.  Instead of an endless 
> loop, it results in a bluescreen.
> 
> As the OP said that it worked for him, I guess it is either in commits 
> that came after his post, or in my add-on patches.

So we are likely on the wrong path. Maybe we have to understand what
happens here first...

> 
> Hopefully I will find some time to work more on this bug.

Would be interesting to know
 - if pic_irq_request is continuously called or if it stops when windows
   hangs
 - what IRQ vectors are delivered
 - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]

Let me know if you can run some instrumentation patches.

Thanks,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Breakage with local APIC routing

[Stefan Weil]

Johannes Schindelin schrieb:
> Hi,
>
> due to the change in revision 3371 (well, at that time, CVS was used, 
> which was no better than Subversion) installation of win64 is broken in 
> QEmu.  The commit message reads like this:
>
> 	Don't route PIC interrupts through the local APIC if the local 
> 	APIC config says so. By Ari Kivity.
>
> A bit of research showed that the patch was actually originally from Qing 
> He, but he told me privately that the part that actually broke win64 (the 
> removal of the call to cpu_reset_interrupt(), as opposed to moving that 
> call into the "else" condition) was not part of his patch.
>
> Unfortunately, a lot has been done to the APIC handling in the meantime, 
> so it is not a simple matter of a revert.
>
> Being a complete idiot when it comes to APICs, I have no clue how to fix 
> the issue.
>
> However, I am quite willing to test whatever patch is thrown at me.
>
> Can somebody help?
>
> Thanks,
> Dscho
See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
EFI BIOS is affected by the same problem.

Regards,
Stefan

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

Stefan Weil wrote:
> Johannes Schindelin schrieb:
>> Hi,
>>
>> due to the change in revision 3371 (well, at that time, CVS was used, 
>> which was no better than Subversion) installation of win64 is broken in 
>> QEmu.  The commit message reads like this:
>>
>> 	Don't route PIC interrupts through the local APIC if the local 
>> 	APIC config says so. By Ari Kivity.
>>
>> A bit of research showed that the patch was actually originally from Qing 
>> He, but he told me privately that the part that actually broke win64 (the 
>> removal of the call to cpu_reset_interrupt(), as opposed to moving that 
>> call into the "else" condition) was not part of his patch.
>>
>> Unfortunately, a lot has been done to the APIC handling in the meantime, 
>> so it is not a simple matter of a revert.
>>
>> Being a complete idiot when it comes to APICs, I have no clue how to fix 
>> the issue.
>>
>> However, I am quite willing to test whatever patch is thrown at me.
>>
>> Can somebody help?
>>
>> Thanks,
>> Dscho
> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
> EFI BIOS is affected by the same problem.
> 

Can you describe a valid test case? I quickly tried
http://bellard.org/qemu/efi-bios.tar.bz2 with qemu revision #3370, but I
only get "Trying to execute code outside RAM or ROM at 0x000a0000".

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Breakage with local APIC routing

[Aurelien Jarno]

On Sun, Aug 17, 2008 at 07:32:28PM +0200, Stefan Weil wrote:
> Johannes Schindelin schrieb:
> > Hi,
> >
> > due to the change in revision 3371 (well, at that time, CVS was used, 
> > which was no better than Subversion) installation of win64 is broken in 
> > QEmu.  The commit message reads like this:
> >
> > 	Don't route PIC interrupts through the local APIC if the local 
> > 	APIC config says so. By Ari Kivity.
> >
> > A bit of research showed that the patch was actually originally from Qing 
> > He, but he told me privately that the part that actually broke win64 (the 
> > removal of the call to cpu_reset_interrupt(), as opposed to moving that 
> > call into the "else" condition) was not part of his patch.
> >
> > Unfortunately, a lot has been done to the APIC handling in the meantime, 
> > so it is not a simple matter of a revert.
> >
> > Being a complete idiot when it comes to APICs, I have no clue how to fix 
> > the issue.
> >
> > However, I am quite willing to test whatever patch is thrown at me.
> >
> > Can somebody help?
> >
> > Thanks,
> > Dscho
> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
> EFI BIOS is affected by the same problem.
> 

With the difference that the patch proposed in this thread actually
fixes the problem in case of the EFI BIOS.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32@debian.org         | aurelien@aurel32.net
   `-    people.debian.org/~aurel32 | www.aurel32.net

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Stefan Weil]

Jan Kiszka schrieb:
> Stefan Weil wrote:
>   
>> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
>> EFI BIOS is affected by the same problem.
>>     
> Can you describe a valid test case? I quickly tried
> http://bellard.org/qemu/efi-bios.tar.bz2 with qemu revision #3370, but I
> only get "Trying to execute code outside RAM or ROM at 0x000a0000".
>
> Jan

See http://article.gmane.org/gmane.comp.emulators.qemu/24738.

Stefan

Re: [Qemu-devel] Breakage with local APIC routing

[Stefan Weil]

Aurelien Jarno schrieb:
> On Sun, Aug 17, 2008 at 07:32:28PM +0200, Stefan Weil wrote:
>   
>> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
>> EFI BIOS is affected by the same problem.
> With the difference that the patch proposed in this thread actually
> fixes the problem in case of the EFI BIOS.
>   

Great. So that patch improves Qemu and should be applied to trunk?

Stefan

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

Aurelien Jarno wrote:
> On Sun, Aug 17, 2008 at 07:32:28PM +0200, Stefan Weil wrote:
>> Johannes Schindelin schrieb:
>>> Hi,
>>>
>>> due to the change in revision 3371 (well, at that time, CVS was used, 
>>> which was no better than Subversion) installation of win64 is broken in 
>>> QEmu.  The commit message reads like this:
>>>
>>> 	Don't route PIC interrupts through the local APIC if the local 
>>> 	APIC config says so. By Ari Kivity.
>>>
>>> A bit of research showed that the patch was actually originally from Qing 
>>> He, but he told me privately that the part that actually broke win64 (the 
>>> removal of the call to cpu_reset_interrupt(), as opposed to moving that 
>>> call into the "else" condition) was not part of his patch.
>>>
>>> Unfortunately, a lot has been done to the APIC handling in the meantime, 
>>> so it is not a simple matter of a revert.
>>>
>>> Being a complete idiot when it comes to APICs, I have no clue how to fix 
>>> the issue.
>>>
>>> However, I am quite willing to test whatever patch is thrown at me.
>>>
>>> Can somebody help?
>>>
>>> Thanks,
>>> Dscho
>> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
>> EFI BIOS is affected by the same problem.
>>
> 
> With the difference that the patch proposed in this thread actually
> fixes the problem in case of the EFI BIOS.

[ Got it working, was to dumb to feed in the right file as bios... ]

Yeah, indeed, my patch cures the EFI issue. Interesting. But I still
don't understand what problem Windows has.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Breakage with local APIC routing

[Anthony Liguori]

Stefan Weil wrote:
> Aurelien Jarno schrieb:
>   
>> On Sun, Aug 17, 2008 at 07:32:28PM +0200, Stefan Weil wrote:
>>   
>>     
>>> See also http://article.gmane.org/gmane.comp.emulators.qemu/24739.
>>> EFI BIOS is affected by the same problem.
>>>       
>> With the difference that the patch proposed in this thread actually
>> fixes the problem in case of the EFI BIOS.
>>   
>>     
>
> Great. So that patch improves Qemu and should be applied to trunk?
>   

If someone resubmits the patch, it will be reviewed and applied if 
appropriate.

I couldn't find the referenced patch (not sure which thread it's a part 
of).  In general, I don't like applying patches that are deep within a 
thread because it's hard to determine whether there was an updated patch 
somewhere else in the thread and it's even more difficult to identify 
which messages in the thread may contain a patch.

Top-level postings with a clear indication of which version of the patch 
(v2, v3, v4, etc.) and an indication of whether it's a resubmission (via 
[RESEND]) makes it much easier to find the right patch to apply.

Regards,

Anthony Liguori

> Stefan
>
>
>
>   

[Qemu-devel] Re: Breakage with local APIC routing

[Johannes Schindelin]

Hi,

On Sun, 17 Aug 2008, Jan Kiszka wrote:

> Johannes Schindelin wrote:
> 
> > On Wed, 13 Aug 2008, Jan Kiszka wrote:
> > 
> >> Johannes Schindelin wrote:
> >>> due to the change in revision 3371 (well, at that time, CVS was used, 
> >>> which was no better than Subversion) installation of win64 is broken 
> >>> in QEmu.  The commit message reads like this:
> >>>
> >>> 	Don't route PIC interrupts through the local APIC if the local 
> >>> 	APIC config says so. By Ari Kivity.
> >> I recalled some earlier post on this which claimed to fix the issue and 
> >> found it in the archive:
> >>
> >> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> > 
> > I tried this, and it changes the symptoms, indeed.  Instead of an endless 
> > loop, it results in a bluescreen.
> > 
> > As the OP said that it worked for him, I guess it is either in commits 
> > that came after his post, or in my add-on patches.
> 
> So we are likely on the wrong path. Maybe we have to understand what
> happens here first...
> 
> > 
> > Hopefully I will find some time to work more on this bug.
> 
> Would be interesting to know
>  - if pic_irq_request is continuously called or if it stops when windows 
>    hangs
>  - what IRQ vectors are delivered
>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]

Sorry for the long delay.  I just don't have time to take care of the 
issue, but I quickly verified that it still does not work, with aa0cba4 
(Aug 13 2009).

If you are still interested in this issue, could you give me a hint 
_where_ I should output _which_ values?  I'll gladly take time for that 
now.

Ciao,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> On Sun, 17 Aug 2008, Jan Kiszka wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>
>>>> Johannes Schindelin wrote:
>>>>> due to the change in revision 3371 (well, at that time, CVS was used, 
>>>>> which was no better than Subversion) installation of win64 is broken 
>>>>> in QEmu.  The commit message reads like this:
>>>>>
>>>>> 	Don't route PIC interrupts through the local APIC if the local 
>>>>> 	APIC config says so. By Ari Kivity.
>>>> I recalled some earlier post on this which claimed to fix the issue and 
>>>> found it in the archive:
>>>>
>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>> I tried this, and it changes the symptoms, indeed.  Instead of an endless 
>>> loop, it results in a bluescreen.
>>>
>>> As the OP said that it worked for him, I guess it is either in commits 
>>> that came after his post, or in my add-on patches.
>> So we are likely on the wrong path. Maybe we have to understand what
>> happens here first...
>>
>>> Hopefully I will find some time to work more on this bug.
>> Would be interesting to know
>>  - if pic_irq_request is continuously called or if it stops when windows 
>>    hangs
>>  - what IRQ vectors are delivered
>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
> 
> Sorry for the long delay.  I just don't have time to take care of the 
> issue, but I quickly verified that it still does not work, with aa0cba4 
> (Aug 13 2009).
> 
> If you are still interested in this issue, could you give me a hint 
> _where_ I should output _which_ values?  I'll gladly take time for that 
> now.

If some OS does not properly install due to a possible emulation bug, I
am interested, for sure. Let's restart this by specifying the test case
more precisely: What version of Windows are you trying to install? What
is your qemu command line? Where does the installation fail? Are there
specific steps required during the installation to reproduce the problem?

And one more question: Did you check that you were using the
corresponding BIOS to aa0cba4?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Mohammed Gamal]

On Tue, Aug 25, 2009 at 9:56 AM, Jan Kiszka<jan.kiszka@web.de> wrote:
> Johannes Schindelin wrote:
>> Hi,
>>
>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>
>>> Johannes Schindelin wrote:
>>>
>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>
>>>>> Johannes Schindelin wrote:
>>>>>> due to the change in revision 3371 (well, at that time, CVS was used,
>>>>>> which was no better than Subversion) installation of win64 is broken
>>>>>> in QEmu.  The commit message reads like this:
>>>>>>
>>>>>>   Don't route PIC interrupts through the local APIC if the local
>>>>>>   APIC config says so. By Ari Kivity.
>>>>> I recalled some earlier post on this which claimed to fix the issue and
>>>>> found it in the archive:
>>>>>
>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>> I tried this, and it changes the symptoms, indeed.  Instead of an endless
>>>> loop, it results in a bluescreen.
>>>>
>>>> As the OP said that it worked for him, I guess it is either in commits
>>>> that came after his post, or in my add-on patches.
>>> So we are likely on the wrong path. Maybe we have to understand what
>>> happens here first...
>>>
>>>> Hopefully I will find some time to work more on this bug.
>>> Would be interesting to know
>>>  - if pic_irq_request is continuously called or if it stops when windows
>>>    hangs
>>>  - what IRQ vectors are delivered
>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>
>> Sorry for the long delay.  I just don't have time to take care of the
>> issue, but I quickly verified that it still does not work, with aa0cba4
>> (Aug 13 2009).
>>
>> If you are still interested in this issue, could you give me a hint
>> _where_ I should output _which_ values?  I'll gladly take time for that
>> now.
>
> If some OS does not properly install due to a possible emulation bug, I
> am interested, for sure. Let's restart this by specifying the test case
> more precisely: What version of Windows are you trying to install? What
> is your qemu command line? Where does the installation fail? Are there
> specific steps required during the installation to reproduce the problem?
>
> And one more question: Did you check that you were using the
> corresponding BIOS to aa0cba4?
>
> Jan
>
>

In case this is considered relevant. I am also facing a bug with
installing amd64 version of ubuntu-9.04. I first thought it was a
possible bug in KVM, but even when using -no-kvm the guest still hangs
just when starting to load the kernel.

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2844 bytes --]

Mohammed Gamal wrote:
> On Tue, Aug 25, 2009 at 9:56 AM, Jan Kiszka<jan.kiszka@web.de> wrote:
>> Johannes Schindelin wrote:
>>> Hi,
>>>
>>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>>
>>>> Johannes Schindelin wrote:
>>>>
>>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>>
>>>>>> Johannes Schindelin wrote:
>>>>>>> due to the change in revision 3371 (well, at that time, CVS was used,
>>>>>>> which was no better than Subversion) installation of win64 is broken
>>>>>>> in QEmu.  The commit message reads like this:
>>>>>>>
>>>>>>>   Don't route PIC interrupts through the local APIC if the local
>>>>>>>   APIC config says so. By Ari Kivity.
>>>>>> I recalled some earlier post on this which claimed to fix the issue and
>>>>>> found it in the archive:
>>>>>>
>>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>>> I tried this, and it changes the symptoms, indeed.  Instead of an endless
>>>>> loop, it results in a bluescreen.
>>>>>
>>>>> As the OP said that it worked for him, I guess it is either in commits
>>>>> that came after his post, or in my add-on patches.
>>>> So we are likely on the wrong path. Maybe we have to understand what
>>>> happens here first...
>>>>
>>>>> Hopefully I will find some time to work more on this bug.
>>>> Would be interesting to know
>>>>  - if pic_irq_request is continuously called or if it stops when windows
>>>>    hangs
>>>>  - what IRQ vectors are delivered
>>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>> Sorry for the long delay.  I just don't have time to take care of the
>>> issue, but I quickly verified that it still does not work, with aa0cba4
>>> (Aug 13 2009).
>>>
>>> If you are still interested in this issue, could you give me a hint
>>> _where_ I should output _which_ values?  I'll gladly take time for that
>>> now.
>> If some OS does not properly install due to a possible emulation bug, I
>> am interested, for sure. Let's restart this by specifying the test case
>> more precisely: What version of Windows are you trying to install? What
>> is your qemu command line? Where does the installation fail? Are there
>> specific steps required during the installation to reproduce the problem?
>>
>> And one more question: Did you check that you were using the
>> corresponding BIOS to aa0cba4?
>>
>> Jan
>>
>>
> 
> In case this is considered relevant. I am also facing a bug with
> installing amd64 version of ubuntu-9.04. I first thought it was a
> possible bug in KVM, but even when using -no-kvm the guest still hangs
> just when starting to load the kernel.

Might be the same, might be a different issue. Same questions to you:
what is your command line, and are there specific steps required to
reproduce the issue? And I assume you also tried plain qemu, right?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Mohammed Gamal]

qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>

I was using qemu-kvm, but I assume that using -no-kvm would be
equivalent to using plain qemu, no?

On Tue, Aug 25, 2009 at 11:59 AM, Jan Kiszka<jan.kiszka@web.de> wrote:
> Mohammed Gamal wrote:
>> On Tue, Aug 25, 2009 at 9:56 AM, Jan Kiszka<jan.kiszka@web.de> wrote:
>>> Johannes Schindelin wrote:
>>>> Hi,
>>>>
>>>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>>>
>>>>> Johannes Schindelin wrote:
>>>>>
>>>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>>>
>>>>>>> Johannes Schindelin wrote:
>>>>>>>> due to the change in revision 3371 (well, at that time, CVS was used,
>>>>>>>> which was no better than Subversion) installation of win64 is broken
>>>>>>>> in QEmu.  The commit message reads like this:
>>>>>>>>
>>>>>>>>   Don't route PIC interrupts through the local APIC if the local
>>>>>>>>   APIC config says so. By Ari Kivity.
>>>>>>> I recalled some earlier post on this which claimed to fix the issue and
>>>>>>> found it in the archive:
>>>>>>>
>>>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>>>> I tried this, and it changes the symptoms, indeed.  Instead of an endless
>>>>>> loop, it results in a bluescreen.
>>>>>>
>>>>>> As the OP said that it worked for him, I guess it is either in commits
>>>>>> that came after his post, or in my add-on patches.
>>>>> So we are likely on the wrong path. Maybe we have to understand what
>>>>> happens here first...
>>>>>
>>>>>> Hopefully I will find some time to work more on this bug.
>>>>> Would be interesting to know
>>>>>  - if pic_irq_request is continuously called or if it stops when windows
>>>>>    hangs
>>>>>  - what IRQ vectors are delivered
>>>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>>> Sorry for the long delay.  I just don't have time to take care of the
>>>> issue, but I quickly verified that it still does not work, with aa0cba4
>>>> (Aug 13 2009).
>>>>
>>>> If you are still interested in this issue, could you give me a hint
>>>> _where_ I should output _which_ values?  I'll gladly take time for that
>>>> now.
>>> If some OS does not properly install due to a possible emulation bug, I
>>> am interested, for sure. Let's restart this by specifying the test case
>>> more precisely: What version of Windows are you trying to install? What
>>> is your qemu command line? Where does the installation fail? Are there
>>> specific steps required during the installation to reproduce the problem?
>>>
>>> And one more question: Did you check that you were using the
>>> corresponding BIOS to aa0cba4?
>>>
>>> Jan
>>>
>>>
>>
>> In case this is considered relevant. I am also facing a bug with
>> installing amd64 version of ubuntu-9.04. I first thought it was a
>> possible bug in KVM, but even when using -no-kvm the guest still hangs
>> just when starting to load the kernel.
>
> Might be the same, might be a different issue. Same questions to you:
> what is your command line, and are there specific steps required to
> reproduce the issue? And I assume you also tried plain qemu, right?
>
> Jan
>
>

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Mohammed Gamal wrote:
> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> 

I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
image I'm unable to reproduce. Will download and check standard ubuntu
later today.

> I was using qemu-kvm, but I assume that using -no-kvm would be
> equivalent to using plain qemu, no?

Generally yes, but not necessarily (e.g. the BIOSes are different). So
it's better to check such issues also against "clean" qemu, specifically
as we are on qemu-devel here.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Mohammed Gamal]

On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> Mohammed Gamal wrote:
>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
>>
>
> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> image I'm unable to reproduce. Will download and check standard ubuntu
> later today.
>
>> I was using qemu-kvm, but I assume that using -no-kvm would be
>> equivalent to using plain qemu, no?
>
> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> it's better to check such issues also against "clean" qemu, specifically
> as we are on qemu-devel here.
>
> Jan
>
>
Just tested this now on a vanilla qemu, I am still able to reproduce
the same issue.

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Mohammed Gamal]

On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
>> Mohammed Gamal wrote:
>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
>>>
>>
>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
>> image I'm unable to reproduce. Will download and check standard ubuntu
>> later today.
>>
>>> I was using qemu-kvm, but I assume that using -no-kvm would be
>>> equivalent to using plain qemu, no?
>>
>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
>> it's better to check such issues also against "clean" qemu, specifically
>> as we are on qemu-devel here.
>>
>> Jan
>>
>>
> Just tested this now on a vanilla qemu, I am still able to reproduce
> the same issue.
>

This bug might be related to the same problem
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

Mohammed Gamal wrote:
> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
>>> Mohammed Gamal wrote:
>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
>>>>
>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
>>> image I'm unable to reproduce. Will download and check standard ubuntu
>>> later today.
>>>
>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
>>>> equivalent to using plain qemu, no?
>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
>>> it's better to check such issues also against "clean" qemu, specifically
>>> as we are on qemu-devel here.
>>>
>>> Jan
>>>
>>>
>> Just tested this now on a vanilla qemu, I am still able to reproduce
>> the same issue.
>>
> 
> This bug might be related to the same problem
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000

I think at least those issues should be solved with recent qemu and
bioses. Note that this report refers to a fairly old qemu version
(0.10.0-derived).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Breakage with local APIC routing

[Johannes Schindelin]

Hi,

On Tue, 25 Aug 2009, Jan Kiszka wrote:

> Johannes Schindelin wrote:
> 
> > On Sun, 17 Aug 2008, Jan Kiszka wrote:
> > 
> >> Johannes Schindelin wrote:
> >>
> >>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
> >>>
> >>>> Johannes Schindelin wrote:
> >>>>> due to the change in revision 3371 (well, at that time, CVS was 
> >>>>> used, which was no better than Subversion) installation of win64 
> >>>>> is broken in QEmu.  The commit message reads like this:
> >>>>>
> >>>>> 	Don't route PIC interrupts through the local APIC if the local 
> >>>>> 	APIC config says so. By Ari Kivity.
> >>>> I recalled some earlier post on this which claimed to fix the issue 
> >>>> and found it in the archive:
> >>>>
> >>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> >>> I tried this, and it changes the symptoms, indeed.  Instead of an 
> >>> endless loop, it results in a bluescreen.
> >>>
> >>> As the OP said that it worked for him, I guess it is either in 
> >>> commits that came after his post, or in my add-on patches.
> >> So we are likely on the wrong path. Maybe we have to understand what
> >> happens here first...
> >>
> >>> Hopefully I will find some time to work more on this bug.
> >> Would be interesting to know
> >>  - if pic_irq_request is continuously called or if it stops when windows 
> >>    hangs
> >>  - what IRQ vectors are delivered
> >>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
> > 
> > Sorry for the long delay.  I just don't have time to take care of the 
> > issue, but I quickly verified that it still does not work, with aa0cba4 
> > (Aug 13 2009).
> > 
> > If you are still interested in this issue, could you give me a hint 
> > _where_ I should output _which_ values?  I'll gladly take time for that 
> > now.
> 
> If some OS does not properly install due to a possible emulation bug, I
> am interested, for sure. Let's restart this by specifying the test case
> more precisely: What version of Windows are you trying to install?

As far as I remember, it is a plain version of 64-bit XP Pro.  (Maybe it 
is a custom .iso for my day-job, but I think this is not the case).

> What is your qemu command line?

test -h pc-bios/keymaps || ln -s ../keymaps pc-bios/

./x86_64-softmmu/qemu-system-x86_64 \
        -L pc-bios/ \
        -m 1024 \
        -monitor stdio \
        -k en-us \
        -hda w64.img \
        -cdrom en_win_xp_pro_x64bit.iso \
        -fda fat:fat \
        -boot d \
        -net none \
        -localtime

> Where does the installation fail?

"Setup is starting Windows". (Just after "Setup is loading files (...)" 
phase.)

> Are there specific steps required during the installation to reproduce 
> the problem?

You need a 64-bit XP Pro, then call the command line as I did.  It hangs 
at

	(qemu) info cpus
	* CPU #0: pc=0xfffff800010cabeb

This is 100% reproducible.

> And one more question: Did you check that you were using the 
> corresponding BIOS to aa0cba4?

Yes, I always use -L pc-bios/ in the same Git working directory, and I 
just verified that indeed, the source is clean.

A tiny, gentle reminder: the revision which is now available as 0e21e12b 
introduced this particular breakage.

Ciao,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3620 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 25 Aug 2009, Jan Kiszka wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>>
>>>> Johannes Schindelin wrote:
>>>>
>>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>>
>>>>>> Johannes Schindelin wrote:
>>>>>>> due to the change in revision 3371 (well, at that time, CVS was 
>>>>>>> used, which was no better than Subversion) installation of win64 
>>>>>>> is broken in QEmu.  The commit message reads like this:
>>>>>>>
>>>>>>> 	Don't route PIC interrupts through the local APIC if the local 
>>>>>>> 	APIC config says so. By Ari Kivity.
>>>>>> I recalled some earlier post on this which claimed to fix the issue 
>>>>>> and found it in the archive:
>>>>>>
>>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>>> I tried this, and it changes the symptoms, indeed.  Instead of an 
>>>>> endless loop, it results in a bluescreen.
>>>>>
>>>>> As the OP said that it worked for him, I guess it is either in 
>>>>> commits that came after his post, or in my add-on patches.
>>>> So we are likely on the wrong path. Maybe we have to understand what
>>>> happens here first...
>>>>
>>>>> Hopefully I will find some time to work more on this bug.
>>>> Would be interesting to know
>>>>  - if pic_irq_request is continuously called or if it stops when windows 
>>>>    hangs
>>>>  - what IRQ vectors are delivered
>>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>> Sorry for the long delay.  I just don't have time to take care of the 
>>> issue, but I quickly verified that it still does not work, with aa0cba4 
>>> (Aug 13 2009).
>>>
>>> If you are still interested in this issue, could you give me a hint 
>>> _where_ I should output _which_ values?  I'll gladly take time for that 
>>> now.
>> If some OS does not properly install due to a possible emulation bug, I
>> am interested, for sure. Let's restart this by specifying the test case
>> more precisely: What version of Windows are you trying to install?
> 
> As far as I remember, it is a plain version of 64-bit XP Pro.  (Maybe it 
> is a custom .iso for my day-job, but I think this is not the case).
> 
>> What is your qemu command line?
> 
> test -h pc-bios/keymaps || ln -s ../keymaps pc-bios/
> 
> ./x86_64-softmmu/qemu-system-x86_64 \
>         -L pc-bios/ \
>         -m 1024 \
>         -monitor stdio \
>         -k en-us \
>         -hda w64.img \
>         -cdrom en_win_xp_pro_x64bit.iso \
>         -fda fat:fat \
>         -boot d \
>         -net none \
>         -localtime
> 
>> Where does the installation fail?
> 
> "Setup is starting Windows". (Just after "Setup is loading files (...)" 
> phase.)
> 
>> Are there specific steps required during the installation to reproduce 
>> the problem?
> 
> You need a 64-bit XP Pro, then call the command line as I did.  It hangs 
> at
> 
> 	(qemu) info cpus
> 	* CPU #0: pc=0xfffff800010cabeb
> 
> This is 100% reproducible.
> 
>> And one more question: Did you check that you were using the 
>> corresponding BIOS to aa0cba4?
> 
> Yes, I always use -L pc-bios/ in the same Git working directory, and I 
> just verified that indeed, the source is clean.
> 
> A tiny, gentle reminder: the revision which is now available as 0e21e12b 
> introduced this particular breakage.

OK, just found some 64-bit Windows ISO (Server 2003) that also makes no
progress at the point you described. Will play with it later today,
specifically with the LAPIC changes you referred to.

Thanks so far,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Breakage with local APIC routing

[Johannes Schindelin]

Hi,

On Tue, 25 Aug 2009, Jan Kiszka wrote:

> Johannes Schindelin wrote:
> 
> > On Tue, 25 Aug 2009, Jan Kiszka wrote:
> > 
> >> Johannes Schindelin wrote:
> >>
> >>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
> >>>
> >>>> Johannes Schindelin wrote:
> >>>>
> >>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
> >>>>>
> >>>>>> Johannes Schindelin wrote:
> >>>>>>> due to the change in revision 3371 (well, at that time, CVS was 
> >>>>>>> used, which was no better than Subversion) installation of win64 
> >>>>>>> is broken in QEmu.  The commit message reads like this:
> >>>>>>>
> >>>>>>> 	Don't route PIC interrupts through the local APIC if the local 
> >>>>>>> 	APIC config says so. By Ari Kivity.
> >>>>>> I recalled some earlier post on this which claimed to fix the issue 
> >>>>>> and found it in the archive:
> >>>>>>
> >>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
> >>>>> I tried this, and it changes the symptoms, indeed.  Instead of an 
> >>>>> endless loop, it results in a bluescreen.
> >>>>>
> >>>>> As the OP said that it worked for him, I guess it is either in 
> >>>>> commits that came after his post, or in my add-on patches.
> >>>> So we are likely on the wrong path. Maybe we have to understand what
> >>>> happens here first...
> >>>>
> >>>>> Hopefully I will find some time to work more on this bug.
> >>>> Would be interesting to know
> >>>>  - if pic_irq_request is continuously called or if it stops when windows 
> >>>>    hangs
> >>>>  - what IRQ vectors are delivered
> >>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
> >>> Sorry for the long delay.  I just don't have time to take care of the 
> >>> issue, but I quickly verified that it still does not work, with aa0cba4 
> >>> (Aug 13 2009).
> >>>
> >>> If you are still interested in this issue, could you give me a hint 
> >>> _where_ I should output _which_ values?  I'll gladly take time for that 
> >>> now.
> >> If some OS does not properly install due to a possible emulation bug, I
> >> am interested, for sure. Let's restart this by specifying the test case
> >> more precisely: What version of Windows are you trying to install?
> > 
> > As far as I remember, it is a plain version of 64-bit XP Pro.  (Maybe it 
> > is a custom .iso for my day-job, but I think this is not the case).
> > 
> >> What is your qemu command line?
> > 
> > test -h pc-bios/keymaps || ln -s ../keymaps pc-bios/
> > 
> > ./x86_64-softmmu/qemu-system-x86_64 \
> >         -L pc-bios/ \
> >         -m 1024 \
> >         -monitor stdio \
> >         -k en-us \
> >         -hda w64.img \
> >         -cdrom en_win_xp_pro_x64bit.iso \
> >         -fda fat:fat \
> >         -boot d \
> >         -net none \
> >         -localtime
> > 
> >> Where does the installation fail?
> > 
> > "Setup is starting Windows". (Just after "Setup is loading files (...)" 
> > phase.)
> > 
> >> Are there specific steps required during the installation to reproduce 
> >> the problem?
> > 
> > You need a 64-bit XP Pro, then call the command line as I did.  It hangs 
> > at
> > 
> > 	(qemu) info cpus
> > 	* CPU #0: pc=0xfffff800010cabeb
> > 
> > This is 100% reproducible.
> > 
> >> And one more question: Did you check that you were using the 
> >> corresponding BIOS to aa0cba4?
> > 
> > Yes, I always use -L pc-bios/ in the same Git working directory, and I 
> > just verified that indeed, the source is clean.
> > 
> > A tiny, gentle reminder: the revision which is now available as 0e21e12b 
> > introduced this particular breakage.
> 
> OK, just found some 64-bit Windows ISO (Server 2003) that also makes no
> progress at the point you described. Will play with it later today,
> specifically with the LAPIC changes you referred to.

Thank you very much!

If you need me to test something, just let me know; I'll try to squeeze 
that into my time schedule.

Ciao,
Dscho

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4626 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 25 Aug 2009, Jan Kiszka wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Tue, 25 Aug 2009, Jan Kiszka wrote:
>>>
>>>> Johannes Schindelin wrote:
>>>>
>>>>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>>>>
>>>>>> Johannes Schindelin wrote:
>>>>>>
>>>>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> Johannes Schindelin wrote:
>>>>>>>>> due to the change in revision 3371 (well, at that time, CVS was 
>>>>>>>>> used, which was no better than Subversion) installation of win64 
>>>>>>>>> is broken in QEmu.  The commit message reads like this:
>>>>>>>>>
>>>>>>>>> 	Don't route PIC interrupts through the local APIC if the local 
>>>>>>>>> 	APIC config says so. By Ari Kivity.
>>>>>>>> I recalled some earlier post on this which claimed to fix the issue 
>>>>>>>> and found it in the archive:
>>>>>>>>
>>>>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>>>>> I tried this, and it changes the symptoms, indeed.  Instead of an 
>>>>>>> endless loop, it results in a bluescreen.
>>>>>>>
>>>>>>> As the OP said that it worked for him, I guess it is either in 
>>>>>>> commits that came after his post, or in my add-on patches.
>>>>>> So we are likely on the wrong path. Maybe we have to understand what
>>>>>> happens here first...
>>>>>>
>>>>>>> Hopefully I will find some time to work more on this bug.
>>>>>> Would be interesting to know
>>>>>>  - if pic_irq_request is continuously called or if it stops when windows 
>>>>>>    hangs
>>>>>>  - what IRQ vectors are delivered
>>>>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>>>> Sorry for the long delay.  I just don't have time to take care of the 
>>>>> issue, but I quickly verified that it still does not work, with aa0cba4 
>>>>> (Aug 13 2009).
>>>>>
>>>>> If you are still interested in this issue, could you give me a hint 
>>>>> _where_ I should output _which_ values?  I'll gladly take time for that 
>>>>> now.
>>>> If some OS does not properly install due to a possible emulation bug, I
>>>> am interested, for sure. Let's restart this by specifying the test case
>>>> more precisely: What version of Windows are you trying to install?
>>> As far as I remember, it is a plain version of 64-bit XP Pro.  (Maybe it 
>>> is a custom .iso for my day-job, but I think this is not the case).
>>>
>>>> What is your qemu command line?
>>> test -h pc-bios/keymaps || ln -s ../keymaps pc-bios/
>>>
>>> ./x86_64-softmmu/qemu-system-x86_64 \
>>>         -L pc-bios/ \
>>>         -m 1024 \
>>>         -monitor stdio \
>>>         -k en-us \
>>>         -hda w64.img \
>>>         -cdrom en_win_xp_pro_x64bit.iso \
>>>         -fda fat:fat \
>>>         -boot d \
>>>         -net none \
>>>         -localtime
>>>
>>>> Where does the installation fail?
>>> "Setup is starting Windows". (Just after "Setup is loading files (...)" 
>>> phase.)
>>>
>>>> Are there specific steps required during the installation to reproduce 
>>>> the problem?
>>> You need a 64-bit XP Pro, then call the command line as I did.  It hangs 
>>> at
>>>
>>> 	(qemu) info cpus
>>> 	* CPU #0: pc=0xfffff800010cabeb
>>>
>>> This is 100% reproducible.
>>>
>>>> And one more question: Did you check that you were using the 
>>>> corresponding BIOS to aa0cba4?
>>> Yes, I always use -L pc-bios/ in the same Git working directory, and I 
>>> just verified that indeed, the source is clean.
>>>
>>> A tiny, gentle reminder: the revision which is now available as 0e21e12b 
>>> introduced this particular breakage.
>> OK, just found some 64-bit Windows ISO (Server 2003) that also makes no
>> progress at the point you described. Will play with it later today,
>> specifically with the LAPIC changes you referred to.
> 
> Thank you very much!
> 
> If you need me to test something, just let me know; I'll try to squeeze 
> that into my time schedule.

Very strange problem: I managed to get past this "Setup is starting
Windows", but only when using KVM mode (both -enable-kvm from upstream
as well as w/ or w/o -no-kvm-irqchip from qemu-kvm). It still takes ages
(several minutes), but it works. The version before the commit you
pointed out does not take so much time for this step.

So we may see two issues here: some x86 emulation problem in TCG mode
plus some IRQ delivery incorrectness.

Did anyone here recently try to install XP-64 (desktop or server) into
qemu or qemu-kvm recently? Can you confirm long delays or hangs during
"Setup is starting Windows" (early text-mode installer)?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Breakage with local APIC routing

[Avi Kivity]

On 08/26/2009 10:31 AM, Jan Kiszka wrote:
> Very strange problem: I managed to get past this "Setup is starting
> Windows", but only when using KVM mode (both -enable-kvm from upstream
> as well as w/ or w/o -no-kvm-irqchip from qemu-kvm). It still takes ages
> (several minutes), but it works. The version before the commit you
> pointed out does not take so much time for this step.
>
> So we may see two issues here: some x86 emulation problem in TCG mode
> plus some IRQ delivery incorrectness.
>
> Did anyone here recently try to install XP-64 (desktop or server) into
> qemu or qemu-kvm recently?

I install XP x64 regularly as part of kvm-autotest.  I don't recall any 
special slowness (well there was b4a2f5e723 hitting the text-mode 
installer, but it only affects smp).

> Can you confirm long delays or hangs during
> "Setup is starting Windows" (early text-mode installer)?
>    

 From the very early days of kvm, that screen was pretty hard to get past.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Juergen Lock]

On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
> Mohammed Gamal wrote:
> > On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> >> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> >>> Mohammed Gamal wrote:
> >>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> >>>>
> >>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> >>> image I'm unable to reproduce. Will download and check standard ubuntu
> >>> later today.
> >>>
> >>>> I was using qemu-kvm, but I assume that using -no-kvm would be
> >>>> equivalent to using plain qemu, no?
> >>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> >>> it's better to check such issues also against "clean" qemu, specifically
> >>> as we are on qemu-devel here.
> >>>
> >>> Jan
> >>>
> >>>
> >> Just tested this now on a vanilla qemu, I am still able to reproduce
> >> the same issue.
> >>
> > 
> > This bug might be related to the same problem
> > https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
> 
> I think at least those issues should be solved with recent qemu and
> bioses. Note that this report refers to a fairly old qemu version
> (0.10.0-derived).

Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
hosts both with 0.10.6 and 0.11.0rc1 before:
	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
As mentioned in that post I was able to work around the issue by passig
the linux guest kernels `no_timer_check' after which they seemed to boot up
just fine, so I suspect in that case its not actually an apic routing
problem but just guest timer irqs arriving late/irregularly which cause
the guest kernel timer checks to time out and fail.

 Just thought I'd mention... :)
	Juergen

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]

Juergen Lock wrote:
> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
>> Mohammed Gamal wrote:
>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
>>>>> Mohammed Gamal wrote:
>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
>>>>>>
>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
>>>>> image I'm unable to reproduce. Will download and check standard ubuntu
>>>>> later today.
>>>>>
>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
>>>>>> equivalent to using plain qemu, no?
>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
>>>>> it's better to check such issues also against "clean" qemu, specifically
>>>>> as we are on qemu-devel here.
>>>>>
>>>>> Jan
>>>>>
>>>>>
>>>> Just tested this now on a vanilla qemu, I am still able to reproduce
>>>> the same issue.
>>>>
>>> This bug might be related to the same problem
>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
>> I think at least those issues should be solved with recent qemu and
>> bioses. Note that this report refers to a fairly old qemu version
>> (0.10.0-derived).
> 
> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
> hosts both with 0.10.6 and 0.11.0rc1 before:
> 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
> As mentioned in that post I was able to work around the issue by passig
> the linux guest kernels `no_timer_check' after which they seemed to boot up
> just fine, so I suspect in that case its not actually an apic routing
> problem but just guest timer irqs arriving late/irregularly which cause
> the guest kernel timer checks to time out and fail.

Does this happen with git head and its corresponding bios, too? I cannot
imagine that the FreeBSD platform is so irregularly generating timer
events for qemu that Linux gets unhappy during this test loop (I think
to remember it needs 3 out of 10 ticks or so to be satisfied).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4464 bytes --]

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 25 Aug 2009, Jan Kiszka wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Tue, 25 Aug 2009, Jan Kiszka wrote:
>>>
>>>> Johannes Schindelin wrote:
>>>>
>>>>> On Sun, 17 Aug 2008, Jan Kiszka wrote:
>>>>>
>>>>>> Johannes Schindelin wrote:
>>>>>>
>>>>>>> On Wed, 13 Aug 2008, Jan Kiszka wrote:
>>>>>>>
>>>>>>>> Johannes Schindelin wrote:
>>>>>>>>> due to the change in revision 3371 (well, at that time, CVS was 
>>>>>>>>> used, which was no better than Subversion) installation of win64 
>>>>>>>>> is broken in QEmu.  The commit message reads like this:
>>>>>>>>>
>>>>>>>>> 	Don't route PIC interrupts through the local APIC if the local 
>>>>>>>>> 	APIC config says so. By Ari Kivity.
>>>>>>>> I recalled some earlier post on this which claimed to fix the issue 
>>>>>>>> and found it in the archive:
>>>>>>>>
>>>>>>>> http://permalink.gmane.org/gmane.comp.emulators.qemu/25415
>>>>>>> I tried this, and it changes the symptoms, indeed.  Instead of an 
>>>>>>> endless loop, it results in a bluescreen.
>>>>>>>
>>>>>>> As the OP said that it worked for him, I guess it is either in 
>>>>>>> commits that came after his post, or in my add-on patches.
>>>>>> So we are likely on the wrong path. Maybe we have to understand what
>>>>>> happens here first...
>>>>>>
>>>>>>> Hopefully I will find some time to work more on this bug.
>>>>>> Would be interesting to know
>>>>>>  - if pic_irq_request is continuously called or if it stops when windows 
>>>>>>    hangs
>>>>>>  - what IRQ vectors are delivered
>>>>>>  - in what state the apic is, namely the s->lvt[APIC_LVT_LINT0]
>>>>> Sorry for the long delay.  I just don't have time to take care of the 
>>>>> issue, but I quickly verified that it still does not work, with aa0cba4 
>>>>> (Aug 13 2009).
>>>>>
>>>>> If you are still interested in this issue, could you give me a hint 
>>>>> _where_ I should output _which_ values?  I'll gladly take time for that 
>>>>> now.
>>>> If some OS does not properly install due to a possible emulation bug, I
>>>> am interested, for sure. Let's restart this by specifying the test case
>>>> more precisely: What version of Windows are you trying to install?
>>> As far as I remember, it is a plain version of 64-bit XP Pro.  (Maybe it 
>>> is a custom .iso for my day-job, but I think this is not the case).
>>>
>>>> What is your qemu command line?
>>> test -h pc-bios/keymaps || ln -s ../keymaps pc-bios/
>>>
>>> ./x86_64-softmmu/qemu-system-x86_64 \
>>>         -L pc-bios/ \
>>>         -m 1024 \
>>>         -monitor stdio \
>>>         -k en-us \
>>>         -hda w64.img \
>>>         -cdrom en_win_xp_pro_x64bit.iso \
>>>         -fda fat:fat \
>>>         -boot d \
>>>         -net none \
>>>         -localtime
>>>
>>>> Where does the installation fail?
>>> "Setup is starting Windows". (Just after "Setup is loading files (...)" 
>>> phase.)
>>>
>>>> Are there specific steps required during the installation to reproduce 
>>>> the problem?
>>> You need a 64-bit XP Pro, then call the command line as I did.  It hangs 
>>> at
>>>
>>> 	(qemu) info cpus
>>> 	* CPU #0: pc=0xfffff800010cabeb
>>>
>>> This is 100% reproducible.
>>>
>>>> And one more question: Did you check that you were using the 
>>>> corresponding BIOS to aa0cba4?
>>> Yes, I always use -L pc-bios/ in the same Git working directory, and I 
>>> just verified that indeed, the source is clean.
>>>
>>> A tiny, gentle reminder: the revision which is now available as 0e21e12b 
>>> introduced this particular breakage.
>> OK, just found some 64-bit Windows ISO (Server 2003) that also makes no
>> progress at the point you described. Will play with it later today,
>> specifically with the LAPIC changes you referred to.
> 
> Thank you very much!
> 
> If you need me to test something, just let me know; I'll try to squeeze 
> that into my time schedule.

I'm starting to get clueless about this issue. It looks like a timing
issue as I was able to crash Windows when using qemu-kvm (in kvm mode)
and attaching a guest debugger to the "right" spot. As you may know,
this also happens today (after dyngen to TCG switch) when resetting the
CPU interrupt in pic_irq_request on !level. To exclude that Windows is
simply fragile here, I need a better test case, ideally some with source
code. Think I will look into Mohammed's Ubuntu case again.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Juergen Lock]

On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote:
> Juergen Lock wrote:
> > On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
> >> Mohammed Gamal wrote:
> >>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> >>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> >>>>> Mohammed Gamal wrote:
> >>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> >>>>>>
> >>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> >>>>> image I'm unable to reproduce. Will download and check standard ubuntu
> >>>>> later today.
> >>>>>
> >>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
> >>>>>> equivalent to using plain qemu, no?
> >>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> >>>>> it's better to check such issues also against "clean" qemu, specifically
> >>>>> as we are on qemu-devel here.
> >>>>>
> >>>>> Jan
> >>>>>
> >>>>>
> >>>> Just tested this now on a vanilla qemu, I am still able to reproduce
> >>>> the same issue.
> >>>>
> >>> This bug might be related to the same problem
> >>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
> >> I think at least those issues should be solved with recent qemu and
> >> bioses. Note that this report refers to a fairly old qemu version
> >> (0.10.0-derived).
> > 
> > Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
> > hosts both with 0.10.6 and 0.11.0rc1 before:
> > 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
> > As mentioned in that post I was able to work around the issue by passig
> > the linux guest kernels `no_timer_check' after which they seemed to boot up
> > just fine, so I suspect in that case its not actually an apic routing
> > problem but just guest timer irqs arriving late/irregularly which cause
> > the guest kernel timer checks to time out and fail.
> 
> Does this happen with git head and its corresponding bios, too? I cannot
> imagine that the FreeBSD platform is so irregularly generating timer
> events for qemu that Linux gets unhappy during this test loop (I think
> to remember it needs 3 out of 10 ticks or so to be satisfied).

Alright so I testwise updated the FreeBSD qemu-devel port to git head
(db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at
	http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch
) - and can report the problem still exists.  (btw pcap seems to be
broken in that patch which was stll working with the 0.11 snapshot,
will have to take care of that later...)

 Thanx,
	Juergen

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2897 bytes --]

Juergen Lock wrote:
> On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote:
>> Juergen Lock wrote:
>>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
>>>> Mohammed Gamal wrote:
>>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
>>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
>>>>>>> Mohammed Gamal wrote:
>>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
>>>>>>>>
>>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
>>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu
>>>>>>> later today.
>>>>>>>
>>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
>>>>>>>> equivalent to using plain qemu, no?
>>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
>>>>>>> it's better to check such issues also against "clean" qemu, specifically
>>>>>>> as we are on qemu-devel here.
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>>
>>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce
>>>>>> the same issue.
>>>>>>
>>>>> This bug might be related to the same problem
>>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
>>>> I think at least those issues should be solved with recent qemu and
>>>> bioses. Note that this report refers to a fairly old qemu version
>>>> (0.10.0-derived).
>>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
>>> hosts both with 0.10.6 and 0.11.0rc1 before:
>>> 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
>>> As mentioned in that post I was able to work around the issue by passig
>>> the linux guest kernels `no_timer_check' after which they seemed to boot up
>>> just fine, so I suspect in that case its not actually an apic routing
>>> problem but just guest timer irqs arriving late/irregularly which cause
>>> the guest kernel timer checks to time out and fail.
>> Does this happen with git head and its corresponding bios, too? I cannot
>> imagine that the FreeBSD platform is so irregularly generating timer
>> events for qemu that Linux gets unhappy during this test loop (I think
>> to remember it needs 3 out of 10 ticks or so to be satisfied).
> 
> Alright so I testwise updated the FreeBSD qemu-devel port to git head
> (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at
> 	http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch

You will have to help me isolating the reason as I don't have any BSD
host. And running recent qemu/-kvm on Linux hosts does not trigger
problems around booting ubuntu here.

> ) - and can report the problem still exists.  (btw pcap seems to be
> broken in that patch which was stll working with the 0.11 snapshot,
> will have to take care of that later...)
> 
>  Thanx,
> 	Juergen

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Juergen Lock]

On Mon, Aug 31, 2009 at 09:47:27AM +0200, Jan Kiszka wrote:
> Juergen Lock wrote:
> > On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote:
> >> Juergen Lock wrote:
> >>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
> >>>> Mohammed Gamal wrote:
> >>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> >>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> >>>>>>> Mohammed Gamal wrote:
> >>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> >>>>>>>>
> >>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> >>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu
> >>>>>>> later today.
> >>>>>>>
> >>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
> >>>>>>>> equivalent to using plain qemu, no?
> >>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> >>>>>>> it's better to check such issues also against "clean" qemu, specifically
> >>>>>>> as we are on qemu-devel here.
> >>>>>>>
> >>>>>>> Jan
> >>>>>>>
> >>>>>>>
> >>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce
> >>>>>> the same issue.
> >>>>>>
> >>>>> This bug might be related to the same problem
> >>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
> >>>> I think at least those issues should be solved with recent qemu and
> >>>> bioses. Note that this report refers to a fairly old qemu version
> >>>> (0.10.0-derived).
> >>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
> >>> hosts both with 0.10.6 and 0.11.0rc1 before:
> >>> 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
> >>> As mentioned in that post I was able to work around the issue by passig
> >>> the linux guest kernels `no_timer_check' after which they seemed to boot up
> >>> just fine, so I suspect in that case its not actually an apic routing
> >>> problem but just guest timer irqs arriving late/irregularly which cause
> >>> the guest kernel timer checks to time out and fail.
> >> Does this happen with git head and its corresponding bios, too? I cannot
> >> imagine that the FreeBSD platform is so irregularly generating timer
> >> events for qemu that Linux gets unhappy during this test loop (I think
> >> to remember it needs 3 out of 10 ticks or so to be satisfied).
> > 
> > Alright so I testwise updated the FreeBSD qemu-devel port to git head
> > (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at
> > 	http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch
> 
> You will have to help me isolating the reason as I don't have any BSD
> host. And running recent qemu/-kvm on Linux hosts does not trigger
> problems around booting ubuntu here.

Well I don't quite know where to start looking, of course I'd be
happy to test patches... :)

 Thanx,
	Juergen

Re: [Qemu-devel] Re: Breakage with local APIC routing

[Juergen Lock]

On Mon, Aug 31, 2009 at 11:27:23PM +0200, Juergen Lock wrote:
> On Mon, Aug 31, 2009 at 09:47:27AM +0200, Jan Kiszka wrote:
> > Juergen Lock wrote:
> > > On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote:
> > >> Juergen Lock wrote:
> > >>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
> > >>>> Mohammed Gamal wrote:
> > >>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> > >>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> > >>>>>>> Mohammed Gamal wrote:
> > >>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> > >>>>>>>>
> > >>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> > >>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu
> > >>>>>>> later today.
> > >>>>>>>
> > >>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
> > >>>>>>>> equivalent to using plain qemu, no?
> > >>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> > >>>>>>> it's better to check such issues also against "clean" qemu, specifically
> > >>>>>>> as we are on qemu-devel here.
> > >>>>>>>
> > >>>>>>> Jan
> > >>>>>>>
> > >>>>>>>
> > >>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce
> > >>>>>> the same issue.
> > >>>>>>
> > >>>>> This bug might be related to the same problem
> > >>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
> > >>>> I think at least those issues should be solved with recent qemu and
> > >>>> bioses. Note that this report refers to a fairly old qemu version
> > >>>> (0.10.0-derived).
> > >>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
> > >>> hosts both with 0.10.6 and 0.11.0rc1 before:
> > >>> 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
> > >>> As mentioned in that post I was able to work around the issue by passig
> > >>> the linux guest kernels `no_timer_check' after which they seemed to boot up
> > >>> just fine, so I suspect in that case its not actually an apic routing
> > >>> problem but just guest timer irqs arriving late/irregularly which cause
> > >>> the guest kernel timer checks to time out and fail.
> > >> Does this happen with git head and its corresponding bios, too? I cannot
> > >> imagine that the FreeBSD platform is so irregularly generating timer
> > >> events for qemu that Linux gets unhappy during this test loop (I think
> > >> to remember it needs 3 out of 10 ticks or so to be satisfied).
> > > 
> > > Alright so I testwise updated the FreeBSD qemu-devel port to git head
> > > (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at
> > > 	http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch
> > 
> > You will have to help me isolating the reason as I don't have any BSD
> > host. And running recent qemu/-kvm on Linux hosts does not trigger
> > problems around booting ubuntu here.
> 
> Well I don't quite know where to start looking, of course I'd be
> happy to test patches... :)

Ok I now found the #if 0 in vl.c:host_alarm_handler(), enabled it and got
this with a FreeBSD 7.2 iso (which defaults to HZ=1000) and -clock dynticks:

timer: min=2555 us max=32004 us avg=5150 us avg_freq=194.149 Hz
timer: min=2451 us max=7205 us avg=3012 us avg_freq=331.960 Hz
timer: min=2855 us max=7994 us avg=3019 us avg_freq=331.191 Hz
timer: min=2099 us max=20003 us avg=3057 us avg_freq=327.073 Hz
timer: min=2143 us max=11897 us avg=3081 us avg_freq=324.527 Hz
timer: min=2110 us max=80014 us avg=3165 us avg_freq=315.913 Hz
timer: min=2088 us max=51999 us avg=3206 us avg_freq=311.873 Hz

and this with the same guest and -clock unix:

timer: min=1614 us max=7121 us avg=2009 us avg_freq=497.692 Hz
timer: min=876 us max=11134 us avg=2016 us avg_freq=495.967 Hz
timer: min=753 us max=9245 us avg=2014 us avg_freq=496.455 Hz
timer: min=1798 us max=6211 us avg=2008 us avg_freq=497.941 Hz
timer: min=1772 us max=6142 us avg=2008 us avg_freq=497.943 Hz
timer: min=1767 us max=6157 us avg=2008 us avg_freq=497.942 Hz
timer: min=14 us max=6139 us avg=2008 us avg_freq=497.939 Hz
timer: min=736 us max=9969 us avg=2016 us avg_freq=495.962 Hz
timer: min=176 us max=11427 us avg=2076 us avg_freq=481.631 Hz
timer: min=229 us max=9827 us avg=2030 us avg_freq=492.546 Hz
timer: min=430 us max=11082 us avg=2078 us avg_freq=481.166 Hz
timer: min=226 us max=9871 us avg=2026 us avg_freq=493.517 Hz
timer: min=605 us max=10829 us avg=2082 us avg_freq=480.243 Hz
timer: min=57 us max=10976 us avg=2028 us avg_freq=493.031 Hz
timer: min=398 us max=11036 us avg=2080 us avg_freq=480.704 Hz

 `vmstat -i' in the guest (available e.g. from the fixit->cdrom/dvd shell
on a livefs or dvd1 iso) gives similar values as avg_freq above for the
`cpu0: timer' irq rate.  (Btw, FreeBSD 8 reduces its HZ to 100 when
running in a vm so the `vmstat -i' values will be a little lower if you
try that as a guest.)

 And `vmstat -i' on the host (also running with HZ=1000) reports a
rate of 1999 for each cpu's timer irq, so there are clearly quite a few
irqs missing there...

 (Btw I'm wondering if the poor usb performance I reported a long time
ago already could be related to this also...)

	Juergen

FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Juergen Lock]

[I'm copying freebsd-current@FreeBSD.org because ppl there might know
more about this...]

 qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest
with the same HZ as the host (like, 1000) with (mostly) proper timing
once, but no longer. :(  It seems there are two problems involved:

 a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ
(can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
only gets proper timing after setting hint.apic.0.disabled=1 via the
loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
or dvd1 iso.)

 b) qemu running on FreeBSD 8 hosts (and most likely head) has the
additional problem of running its timers only at HZ/2 when using
setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also
seen below, timer_settime(2) aka `-clock dynticks' in qemu behaves
even worse, but that is similarly true on FreeBSD 7 which is why
I removed the patch that enabled that from our qemu port(s) a few
days ago.)  And the only reason FreeBSD 8 guests are usually less
affected by these problems is they now reduce their HZ to 100 when
they detect being run in a VM.  (which makes sense for other reasons
as well, don't get me wrong...  but of course doesn't help when the
host is running with HZ=100 too.)

On Tue, Sep 01, 2009 at 10:12:48PM +0200, Juergen Lock wrote:
> On Mon, Aug 31, 2009 at 11:27:23PM +0200, Juergen Lock wrote:
> > On Mon, Aug 31, 2009 at 09:47:27AM +0200, Jan Kiszka wrote:
> > > Juergen Lock wrote:
> > > > On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote:
> > > >> Juergen Lock wrote:
> > > >>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote:
> > > >>>> Mohammed Gamal wrote:
> > > >>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005@gmail.com> wrote:
> > > >>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka@web.de> wrote:
> > > >>>>>>> Mohammed Gamal wrote:
> > > >>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image>
> > > >>>>>>>>
> > > >>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that
> > > >>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu
> > > >>>>>>> later today.
> > > >>>>>>>
> > > >>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be
> > > >>>>>>>> equivalent to using plain qemu, no?
> > > >>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So
> > > >>>>>>> it's better to check such issues also against "clean" qemu, specifically
> > > >>>>>>> as we are on qemu-devel here.
> > > >>>>>>>
> > > >>>>>>> Jan
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce
> > > >>>>>> the same issue.
> > > >>>>>>
> > > >>>>> This bug might be related to the same problem
> > > >>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000
> > > >>>> I think at least those issues should be solved with recent qemu and
> > > >>>> bioses. Note that this report refers to a fairly old qemu version
> > > >>>> (0.10.0-derived).
> > > >>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8
> > > >>> hosts both with 0.10.6 and 0.11.0rc1 before:
> > > >>> 	http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html
> > > >>> As mentioned in that post I was able to work around the issue by passig
> > > >>> the linux guest kernels `no_timer_check' after which they seemed to boot up
> > > >>> just fine, so I suspect in that case its not actually an apic routing
> > > >>> problem but just guest timer irqs arriving late/irregularly which cause
> > > >>> the guest kernel timer checks to time out and fail.
> > > >> Does this happen with git head and its corresponding bios, too? I cannot
> > > >> imagine that the FreeBSD platform is so irregularly generating timer
> > > >> events for qemu that Linux gets unhappy during this test loop (I think
> > > >> to remember it needs 3 out of 10 ticks or so to be satisfied).
> > > > 
> > > > Alright so I testwise updated the FreeBSD qemu-devel port to git head
> > > > (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at
> > > > 	http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch
> > > 
> > > You will have to help me isolating the reason as I don't have any BSD
> > > host. And running recent qemu/-kvm on Linux hosts does not trigger
> > > problems around booting ubuntu here.
> > 
> > Well I don't quite know where to start looking, of course I'd be
> > happy to test patches... :)
> 
> Ok I now found the #if 0 in vl.c:host_alarm_handler(), enabled it and got
> this with a FreeBSD 7.2 iso (which defaults to HZ=1000) and -clock dynticks:
> 
..and this was on a FreeBSD 8 host:

> timer: min=2555 us max=32004 us avg=5150 us avg_freq=194.149 Hz
> timer: min=2451 us max=7205 us avg=3012 us avg_freq=331.960 Hz
> timer: min=2855 us max=7994 us avg=3019 us avg_freq=331.191 Hz
> timer: min=2099 us max=20003 us avg=3057 us avg_freq=327.073 Hz
> timer: min=2143 us max=11897 us avg=3081 us avg_freq=324.527 Hz
> timer: min=2110 us max=80014 us avg=3165 us avg_freq=315.913 Hz
> timer: min=2088 us max=51999 us avg=3206 us avg_freq=311.873 Hz
> 
> and this with the same guest and -clock unix:
> 
> timer: min=1614 us max=7121 us avg=2009 us avg_freq=497.692 Hz
> timer: min=876 us max=11134 us avg=2016 us avg_freq=495.967 Hz
> timer: min=753 us max=9245 us avg=2014 us avg_freq=496.455 Hz
> timer: min=1798 us max=6211 us avg=2008 us avg_freq=497.941 Hz
> timer: min=1772 us max=6142 us avg=2008 us avg_freq=497.943 Hz
> timer: min=1767 us max=6157 us avg=2008 us avg_freq=497.942 Hz
> timer: min=14 us max=6139 us avg=2008 us avg_freq=497.939 Hz
> timer: min=736 us max=9969 us avg=2016 us avg_freq=495.962 Hz
> timer: min=176 us max=11427 us avg=2076 us avg_freq=481.631 Hz
> timer: min=229 us max=9827 us avg=2030 us avg_freq=492.546 Hz
> timer: min=430 us max=11082 us avg=2078 us avg_freq=481.166 Hz
> timer: min=226 us max=9871 us avg=2026 us avg_freq=493.517 Hz
> timer: min=605 us max=10829 us avg=2082 us avg_freq=480.243 Hz
> timer: min=57 us max=10976 us avg=2028 us avg_freq=493.031 Hz
> timer: min=398 us max=11036 us avg=2080 us avg_freq=480.704 Hz
> 
>  `vmstat -i' in the guest (available e.g. from the fixit->cdrom/dvd shell
> on a livefs or dvd1 iso) gives similar values as avg_freq above for the
> `cpu0: timer' irq rate.  (Btw, FreeBSD 8 reduces its HZ to 100 when
> running in a vm so the `vmstat -i' values will be a little lower if you
> try that as a guest.)
> 
>  And `vmstat -i' on the host (also running with HZ=1000) reports a
> rate of 1999 for each cpu's timer irq, so there are clearly quite a few
> irqs missing there...
> 
>  (Btw I'm wondering if the poor usb performance I reported a long time
> ago already could be related to this also...)

 And here is the same guest with -clock unix on a FreeBSD 7 host:
(same with and without apic, except of course that only without apic
timing in the guest is ok.)

timer: min=847 us max=6146 us avg=1005 us avg_freq=994.950 Hz
timer: min=866 us max=6130 us avg=1005 us avg_freq=994.960 Hz
timer: min=868 us max=6133 us avg=1005 us avg_freq=994.958 Hz
timer: min=287 us max=35199 us avg=1089 us avg_freq=918.216 Hz
timer: min=201 us max=13551 us avg=1029 us avg_freq=971.757 Hz
timer: min=792 us max=6211 us avg=1005 us avg_freq=994.961 Hz
timer: min=145 us max=6185 us avg=1007 us avg_freq=992.984 Hz
timer: min=852 us max=6152 us avg=1005 us avg_freq=994.961 Hz
timer: min=820 us max=6175 us avg=1005 us avg_freq=994.960 Hz
timer: min=654 us max=6242 us avg=1005 us avg_freq=994.966 Hz
timer: min=675 us max=6196 us avg=1005 us avg_freq=994.952 Hz
timer: min=829 us max=6162 us avg=1005 us avg_freq=994.960 Hz
timer: min=503 us max=6136 us avg=1005 us avg_freq=994.960 Hz
timer: min=859 us max=6138 us avg=1005 us avg_freq=994.960 Hz
timer: min=856 us max=6149 us avg=1005 us avg_freq=994.960 Hz
timer: min=832 us max=6164 us avg=1005 us avg_freq=994.961 Hz
timer: min=829 us max=6174 us avg=1005 us avg_freq=994.958 Hz
timer: min=826 us max=6173 us avg=1005 us avg_freq=994.960 Hz
timer: min=828 us max=6168 us avg=1005 us avg_freq=994.961 Hz
timer: min=841 us max=6160 us avg=1005 us avg_freq=994.960 Hz
timer: min=857 us max=6135 us avg=1005 us avg_freq=994.959 Hz
timer: min=857 us max=6139 us avg=1005 us avg_freq=994.959 Hz
timer: min=845 us max=6147 us avg=1005 us avg_freq=994.960 Hz
timer: min=854 us max=6143 us avg=1005 us avg_freq=994.960 Hz
timer: min=854 us max=6142 us avg=1005 us avg_freq=994.960 Hz
timer: min=857 us max=6142 us avg=1005 us avg_freq=994.960 Hz
timer: min=859 us max=6141 us avg=1005 us avg_freq=994.961 Hz
timer: min=849 us max=6141 us avg=1005 us avg_freq=994.962 Hz
timer: min=857 us max=6143 us avg=1005 us avg_freq=994.962 Hz
timer: min=864 us max=6138 us avg=1005 us avg_freq=994.958 Hz
timer: min=863 us max=6139 us avg=1005 us avg_freq=994.961 Hz
timer: min=869 us max=6136 us avg=1005 us avg_freq=994.960 Hz
timer: min=857 us max=6143 us avg=1005 us avg_freq=994.959 Hz
timer: min=865 us max=6133 us avg=1005 us avg_freq=994.960 Hz

 And the same guest/host with -clock dynticks:

timer: min=1657 us max=32164 us avg=6732 us avg_freq=148.534 Hz
timer: min=1675 us max=7158 us avg=2010 us avg_freq=497.480 Hz
timer: min=1773 us max=7200 us avg=2010 us avg_freq=497.480 Hz
timer: min=1798 us max=7148 us avg=2010 us avg_freq=497.480 Hz
timer: min=1851 us max=7144 us avg=2013 us avg_freq=496.739 Hz
timer: min=1821 us max=7136 us avg=2020 us avg_freq=495.017 Hz
timer: min=1823 us max=7173 us avg=2010 us avg_freq=497.480 Hz
timer: min=1825 us max=39002 us avg=2076 us avg_freq=481.664 Hz
timer: min=1851 us max=7145 us avg=2012 us avg_freq=496.985 Hz
timer: min=1855 us max=16998 us avg=2106 us avg_freq=474.803 Hz
timer: min=1855 us max=7138 us avg=2010 us avg_freq=497.480 Hz
timer: min=1858 us max=7136 us avg=2010 us avg_freq=497.480 Hz
timer: min=1856 us max=7138 us avg=2010 us avg_freq=497.480 Hz
timer: min=1855 us max=7138 us avg=2017 us avg_freq=495.754 Hz
timer: min=1818 us max=7143 us avg=2009 us avg_freq=497.728 Hz
timer: min=1851 us max=7141 us avg=2010 us avg_freq=497.480 Hz
timer: min=1859 us max=7140 us avg=2010 us avg_freq=497.480 Hz
timer: min=1851 us max=7138 us avg=2010 us avg_freq=497.480 Hz

 For people wanting to experiment, you can escape to the loader prompt
from the FreeBSD boot menu and then type:
	set hint.apic.0.disabled=1
and optionally:
	set kern.hz="1000"
and then:
	boot -v
(for verbose dmesg, otherwise `boot' without the -v.)

 Any enlightnig comments appreciated... :)
	Juergen

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Ryan Stone]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

I'm not entirely clear on why it's done this way, but the timer is run at
twice hz for statistics-gathering purposes*.  CPU usage statistics gathering
is driven off of the timer interrupt.  Running the timer at twice hz may be
an attempt to eliminate clock-aliasing problems; if so, it's a poor way of
doing so.  In any case, seeing interrupts come in at twice hz is expected
behaviour.  This means that the guest will be requesting a timer interrupt
rate of twice the granularity that the host's scheduler can support; this
may be the cause of your other timing problems(although I have a hard time
imagining how).

This timer is twice hz behaviour has existed at least since FreeBSD 6.1, so
I can't explain why you see the new behaviour between 7 and 8.  You do have
hz set to 1000 on both the guest and host when running 7?

* Actually, from looking at the code the behaviour is dynamic.  If hz >=
1500, the timer interrupt rate is set to hz.  If 750 <= hz < 1500, the timer
interrupt rate is set to 2 * hz.  If hz < 750, the timer interrupt rate is
set to 4 * hz.

[-- Attachment #2: Type: text/html, Size: 1121 bytes --]

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Juergen Lock]

On Mon, Sep 07, 2009 at 10:17:23PM -0400, Ryan Stone wrote:
> I'm not entirely clear on why it's done this way, but the timer is run at
> twice hz for statistics-gathering purposes*.  CPU usage statistics gathering
> is driven off of the timer interrupt.  Running the timer at twice hz may be
> an attempt to eliminate clock-aliasing problems; if so, it's a poor way of
> doing so.  In any case, seeing interrupts come in at twice hz is expected
> behaviour.  This means that the guest will be requesting a timer interrupt
> rate of twice the granularity that the host's scheduler can support; this
> may be the cause of your other timing problems(although I have a hard time
> imagining how).
> 
Yes, this is the first issue I reported, and it causes e.g. a FreeBSD 7
guest on a 7 host to sleep ~4s on a `time sleep 2'.  (unless when I
disable apic as I said.)

> This timer is twice hz behaviour has existed at least since FreeBSD 6.1, so
> I can't explain why you see the new behaviour between 7 and 8.  You do have
> hz set to 1000 on both the guest and host when running 7?
> 
 Yes, and the behaviour on 8 is in addition to the guest expecting
clock irqs at 2000 Hz, i.e. on 8 the guest gets only around 500 Hz
with -clock unix instead of ~1000 Hz on a 7 host, and `time sleep 2'
then consequently sleeps ~8s.

> * Actually, from looking at the code the behaviour is dynamic.  If hz >=
> 1500, the timer interrupt rate is set to hz.  If 750 <= hz < 1500, the timer
> interrupt rate is set to 2 * hz.  If hz < 750, the timer interrupt rate is
> set to 4 * hz.

 Aha, so another way to get around the first issue would be to run at
least the host with HZ=2000...  Worth a try, tho I suspect it won't help
the additional rate halving on 8, unless maybe to use even HZ=4000.
Oh well...

	Juergen

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Ryan Stone]

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

Luigi Rizzo recently posted a thread to current (
http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html)
that would explain the new problem on FreeBSD 8.  He posted a patch
that
you can try.

[-- Attachment #2: Type: text/html, Size: 313 bytes --]

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Luigi Rizzo]

On Mon, Sep 07, 2009 at 10:59:55PM +0200, Juergen Lock wrote:
> [I'm copying freebsd-current@FreeBSD.org because ppl there might know
> more about this...]
> 
>  qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest
> with the same HZ as the host (like, 1000) with (mostly) proper timing
> once, but no longer. :(  It seems there are two problems involved:
> 
>  a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ
> (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
> only gets proper timing after setting hint.apic.0.disabled=1 via the
> loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
> installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
> or dvd1 iso.)
> 
>  b) qemu running on FreeBSD 8 hosts (and most likely head) has the
> additional problem of running its timers only at HZ/2 when using
> setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also

this problem in 8.x is caused by the bug i described here yesterday:

http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html

In qeumu, the setitimer call (in file vl.c) has a timeout of 1 tick
which maps to callout_reset(..., 1, ...) and because (due to the bug)
8.x processes callouts 1 tick late, this effectively halves the clock rate.

> seen below, timer_settime(2) aka `-clock dynticks' in qemu behaves
> even worse, but that is similarly true on FreeBSD 7 which is why
> I removed the patch that enabled that from our qemu port(s) a few
> days ago.)  And the only reason FreeBSD 8 guests are usually less
> affected by these problems is they now reduce their HZ to 100 when
> they detect being run in a VM.  (which makes sense for other reasons
> as well, don't get me wrong...  but of course doesn't help when the
> host is running with HZ=100 too.)

cheers
luigi

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Juergen Lock]

On Wed, Sep 09, 2009 at 10:46:16PM +0200, Luigi Rizzo wrote:
> On Mon, Sep 07, 2009 at 10:59:55PM +0200, Juergen Lock wrote:
> > [I'm copying freebsd-current@FreeBSD.org because ppl there might know
> > more about this...]
> > 
> >  qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest
> > with the same HZ as the host (like, 1000) with (mostly) proper timing
> > once, but no longer. :(  It seems there are two problems involved:
> > 
> >  a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ
> > (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
> > only gets proper timing after setting hint.apic.0.disabled=1 via the
> > loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
> > installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
> > or dvd1 iso.)
> > 
> >  b) qemu running on FreeBSD 8 hosts (and most likely head) has the
> > additional problem of running its timers only at HZ/2 when using
> > setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also
> 
> this problem in 8.x is caused by the bug i described here yesterday:
> 
> http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html
> 
> In qeumu, the setitimer call (in file vl.c) has a timeout of 1 tick
> which maps to callout_reset(..., 1, ...) and because (due to the bug)
> 8.x processes callouts 1 tick late, this effectively halves the clock rate.
> 
Thanx for the pointer!

 The proposed patch in that post didn't make a different here tho,
guest still sees only half host HZ clock irq rate. (i.e. ~500 Hz.)

 Here is the patch I used, to make sure I patched what you meant...

Index: sys/kern/kern_timeout.c
@@ -323,7 +323,7 @@ softclock(void *arg)
 	steps = 0;
 	cc = (struct callout_cpu *)arg;
 	CC_LOCK(cc);
-	while (cc->cc_softticks != ticks) {
+	while (cc->cc_softticks-1 != ticks) {
 		/*
 		 * cc_softticks may be modified by hard clock, so cache
 		 * it while we work on a given bucket.

> > seen below, timer_settime(2) aka `-clock dynticks' in qemu behaves
> > even worse, but that is similarly true on FreeBSD 7 which is why
> > I removed the patch that enabled that from our qemu port(s) a few
> > days ago.)  And the only reason FreeBSD 8 guests are usually less
> > affected by these problems is they now reduce their HZ to 100 when
> > they detect being run in a VM.  (which makes sense for other reasons
> > as well, don't get me wrong...  but of course doesn't help when the
> > host is running with HZ=100 too.)
> 
> cheers
> luigi

 Cheers,
	Juergen

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Luigi Rizzo]

On Thu, Sep 10, 2009 at 07:46:40PM +0200, Juergen Lock wrote:
> On Wed, Sep 09, 2009 at 10:46:16PM +0200, Luigi Rizzo wrote:
> > On Mon, Sep 07, 2009 at 10:59:55PM +0200, Juergen Lock wrote:
> > > [I'm copying freebsd-current@FreeBSD.org because ppl there might know
> > > more about this...]
> > > 
> > >  qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest
> > > with the same HZ as the host (like, 1000) with (mostly) proper timing
> > > once, but no longer. :(  It seems there are two problems involved:
> > > 
> > >  a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ
> > > (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
> > > only gets proper timing after setting hint.apic.0.disabled=1 via the
> > > loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
> > > installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
> > > or dvd1 iso.)
> > > 
> > >  b) qemu running on FreeBSD 8 hosts (and most likely head) has the
> > > additional problem of running its timers only at HZ/2 when using
> > > setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also
> > 
> > this problem in 8.x is caused by the bug i described here yesterday:
> > 
> > http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html
> > 
> > In qeumu, the setitimer call (in file vl.c) has a timeout of 1 tick
> > which maps to callout_reset(..., 1, ...) and because (due to the bug)
> > 8.x processes callouts 1 tick late, this effectively halves the clock rate.
> > 
> Thanx for the pointer!
> 
>  The proposed patch in that post didn't make a different here tho,
> guest still sees only half host HZ clock irq rate. (i.e. ~500 Hz.)
> 
>  Here is the patch I used, to make sure I patched what you meant...
> 
> Index: sys/kern/kern_timeout.c
> @@ -323,7 +323,7 @@ softclock(void *arg)
>  	steps = 0;
>  	cc = (struct callout_cpu *)arg;
>  	CC_LOCK(cc);
> -	while (cc->cc_softticks != ticks) {
> +	while (cc->cc_softticks-1 != ticks) {
>  		/*
>  		 * cc_softticks may be modified by hard clock, so cache
>  		 * it while we work on a given bucket.
> 

as mentioned in the followup message in that thread,
you also need this change in callout_tick()

        mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET);
     -  for (; (cc->cc_softticks - ticks) < 0; cc->cc_softticks++) {
     +  for (; (cc->cc_softticks - ticks) <= 0; cc->cc_softticks++) {
                bucket = cc->cc_softticks & callwheelmask;

cheers
luigi

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Juergen Lock]

On Thu, Sep 10, 2009 at 09:08:00PM +0200, Luigi Rizzo wrote:
> On Thu, Sep 10, 2009 at 07:46:40PM +0200, Juergen Lock wrote:
> > On Wed, Sep 09, 2009 at 10:46:16PM +0200, Luigi Rizzo wrote:
> > > On Mon, Sep 07, 2009 at 10:59:55PM +0200, Juergen Lock wrote:
> > > > [I'm copying freebsd-current@FreeBSD.org because ppl there might know
> > > > more about this...]
> > > > 
> > > >  qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest
> > > > with the same HZ as the host (like, 1000) with (mostly) proper timing
> > > > once, but no longer. :(  It seems there are two problems involved:
> > > > 
> > > >  a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ
> > > > (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
> > > > only gets proper timing after setting hint.apic.0.disabled=1 via the
> > > > loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
> > > > installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
> > > > or dvd1 iso.)
> > > > 
> > > >  b) qemu running on FreeBSD 8 hosts (and most likely head) has the
> > > > additional problem of running its timers only at HZ/2 when using
> > > > setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also
> > > 
> > > this problem in 8.x is caused by the bug i described here yesterday:
> > > 
> > > http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html
> > > 
> > > In qeumu, the setitimer call (in file vl.c) has a timeout of 1 tick
> > > which maps to callout_reset(..., 1, ...) and because (due to the bug)
> > > 8.x processes callouts 1 tick late, this effectively halves the clock rate.
> > > 
> > Thanx for the pointer!
> > 
> >  The proposed patch in that post didn't make a different here tho,
> > guest still sees only half host HZ clock irq rate. (i.e. ~500 Hz.)
> > 
> >  Here is the patch I used, to make sure I patched what you meant...
> > 
> > Index: sys/kern/kern_timeout.c
> > @@ -323,7 +323,7 @@ softclock(void *arg)
> >  	steps = 0;
> >  	cc = (struct callout_cpu *)arg;
> >  	CC_LOCK(cc);
> > -	while (cc->cc_softticks != ticks) {
> > +	while (cc->cc_softticks-1 != ticks) {
> >  		/*
> >  		 * cc_softticks may be modified by hard clock, so cache
> >  		 * it while we work on a given bucket.
> > 
> 
> as mentioned in the followup message in that thread,
> you also need this change in callout_tick()
> 
>         mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET);
>      -  for (; (cc->cc_softticks - ticks) < 0; cc->cc_softticks++) {
>      +  for (; (cc->cc_softticks - ticks) <= 0; cc->cc_softticks++) {
>                 bucket = cc->cc_softticks & callwheelmask;
> 
Ah I missed that, now guest clock irqs are back to HZ indeed.

 Thanx, :)
	Juergen

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[John Baldwin]

On Thursday 10 September 2009 3:08:00 pm Luigi Rizzo wrote:
> On Thu, Sep 10, 2009 at 07:46:40PM +0200, Juergen Lock wrote:
> > On Wed, Sep 09, 2009 at 10:46:16PM +0200, Luigi Rizzo wrote:
> > > On Mon, Sep 07, 2009 at 10:59:55PM +0200, Juergen Lock wrote:
> > > > [I'm copying freebsd-current@FreeBSD.org because ppl there might know
> > > > more about this...]
> > > > 
> > > >  qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) 
guest
> > > > with the same HZ as the host (like, 1000) with (mostly) proper timing
> > > > once, but no longer. :(  It seems there are two problems involved:
> > > > 
> > > >  a) use of apic seems to cause the clock irq rate to be doubled to 2 * 
HZ
> > > > (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host
> > > > only gets proper timing after setting hint.apic.0.disabled=1 via the
> > > > loader.  (as can be verified by `vmstat -i' and `time sleep 2' in an
> > > > installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs
> > > > or dvd1 iso.)
> > > > 
> > > >  b) qemu running on FreeBSD 8 hosts (and most likely head) has the
> > > > additional problem of running its timers only at HZ/2 when using
> > > > setitimer(2) (called `-clock unix' in qemu), as seen below.  (as also
> > > 
> > > this problem in 8.x is caused by the bug i described here yesterday:
> > > 
> > > 
http://lists.freebsd.org/pipermail/freebsd-current/2009-September/011393.html
> > > 
> > > In qeumu, the setitimer call (in file vl.c) has a timeout of 1 tick
> > > which maps to callout_reset(..., 1, ...) and because (due to the bug)
> > > 8.x processes callouts 1 tick late, this effectively halves the clock 
rate.
> > > 
> > Thanx for the pointer!
> > 
> >  The proposed patch in that post didn't make a different here tho,
> > guest still sees only half host HZ clock irq rate. (i.e. ~500 Hz.)
> > 
> >  Here is the patch I used, to make sure I patched what you meant...
> > 
> > Index: sys/kern/kern_timeout.c
> > @@ -323,7 +323,7 @@ softclock(void *arg)
> >  	steps = 0;
> >  	cc = (struct callout_cpu *)arg;
> >  	CC_LOCK(cc);
> > -	while (cc->cc_softticks != ticks) {
> > +	while (cc->cc_softticks-1 != ticks) {
> >  		/*
> >  		 * cc_softticks may be modified by hard clock, so cache
> >  		 * it while we work on a given bucket.
> > 
> 
> as mentioned in the followup message in that thread,
> you also need this change in callout_tick()
> 
>         mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET);
>      -  for (; (cc->cc_softticks - ticks) < 0; cc->cc_softticks++) {
>      +  for (; (cc->cc_softticks - ticks) <= 0; cc->cc_softticks++) {
>                 bucket = cc->cc_softticks & callwheelmask;

I would fix the style in the first hunk (spaces around '-') but I think you 
should commit this and get it into 8.0.  I think a per-CPU ticks might prove 
very problematic as 'ticks' is rather widely used (though I would find that 
cleaner perhaps).

-- 
John Baldwin

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[John Baldwin]

On Friday 11 September 2009 1:03:17 pm Luigi Rizzo wrote:
> On Fri, Sep 11, 2009 at 11:22:59AM -0400, John Baldwin wrote:
> > On Thursday 10 September 2009 3:08:00 pm Luigi Rizzo wrote:
> ...
> > > > Index: sys/kern/kern_timeout.c
> > > > @@ -323,7 +323,7 @@ softclock(void *arg)
> > > >  	steps = 0;
> > > >  	cc = (struct callout_cpu *)arg;
> > > >  	CC_LOCK(cc);
> > > > -	while (cc->cc_softticks != ticks) {
> > > > +	while (cc->cc_softticks-1 != ticks) {
> > > >  		/*
> > > >  		 * cc_softticks may be modified by hard clock, so cache
> > > >  		 * it while we work on a given bucket.
> > > > 
> > > 
> > > as mentioned in the followup message in that thread,
> > > you also need this change in callout_tick()
> > > 
> > >         mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET);
> > >      -  for (; (cc->cc_softticks - ticks) < 0; cc->cc_softticks++) {
> > >      +  for (; (cc->cc_softticks - ticks) <= 0; cc->cc_softticks++) {
> > >                 bucket = cc->cc_softticks & callwheelmask;
> > 
> > I would fix the style in the first hunk (spaces around '-') but I think 
you 
> > should commit this and get it into 8.0.  I think a per-CPU ticks might 
prove 
> > very problematic as 'ticks' is rather widely used (though I would find 
that 
> > cleaner perhaps).
> 
> i will ask permission to re -- i was hoping to get some feedback
> on the thread on -current but no response so far :(
> 
> Note that the per-cpu ticks i was proposing were only visible to the
> timing wheels, which don't use absolute timeouts anyways.
> So i think the mechanism would be quite safe: right now, when you
> request a callout after x ticks, the code first picks a CPU
> (with some criteria), then puts the request in the timer wheel for
> that CPU using (now) the global 'ticks'. Replacing ticks with cc->cc_ticks,
> would completely remove the races in insertion and removal.
> 
> I actually find the per-cpu ticks even less intrusive than this change.

Well, it depends.  If TCP ever started using per-CPU callouts (i.e. 
callout_reset_on()) it would probably need to start using the per-CPU ticks 
instead of the global ticks, etc.  You could have 'ticks' just be == to CPU 
0's ticks perhaps.

-- 
John Baldwin

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Luigi Rizzo]

On Fri, Sep 11, 2009 at 11:22:59AM -0400, John Baldwin wrote:
> On Thursday 10 September 2009 3:08:00 pm Luigi Rizzo wrote:
...
> > > Index: sys/kern/kern_timeout.c
> > > @@ -323,7 +323,7 @@ softclock(void *arg)
> > >  	steps = 0;
> > >  	cc = (struct callout_cpu *)arg;
> > >  	CC_LOCK(cc);
> > > -	while (cc->cc_softticks != ticks) {
> > > +	while (cc->cc_softticks-1 != ticks) {
> > >  		/*
> > >  		 * cc_softticks may be modified by hard clock, so cache
> > >  		 * it while we work on a given bucket.
> > > 
> > 
> > as mentioned in the followup message in that thread,
> > you also need this change in callout_tick()
> > 
> >         mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET);
> >      -  for (; (cc->cc_softticks - ticks) < 0; cc->cc_softticks++) {
> >      +  for (; (cc->cc_softticks - ticks) <= 0; cc->cc_softticks++) {
> >                 bucket = cc->cc_softticks & callwheelmask;
> 
> I would fix the style in the first hunk (spaces around '-') but I think you 
> should commit this and get it into 8.0.  I think a per-CPU ticks might prove 
> very problematic as 'ticks' is rather widely used (though I would find that 
> cleaner perhaps).

i will ask permission to re -- i was hoping to get some feedback
on the thread on -current but no response so far :(

Note that the per-cpu ticks i was proposing were only visible to the
timing wheels, which don't use absolute timeouts anyways.
So i think the mechanism would be quite safe: right now, when you
request a callout after x ticks, the code first picks a CPU
(with some criteria), then puts the request in the timer wheel for
that CPU using (now) the global 'ticks'. Replacing ticks with cc->cc_ticks,
would completely remove the races in insertion and removal.

I actually find the per-cpu ticks even less intrusive than this change.

	cheers
	luigi

Re: FreeBSD timing issues and qemu (was: Re: [Qemu-devel] Re: Breakage with local APIC routing)

[Luigi Rizzo]

On Fri, Sep 11, 2009 at 01:01:54PM -0400, John Baldwin wrote:
> On Friday 11 September 2009 1:03:17 pm Luigi Rizzo wrote:
...
> > Note that the per-cpu ticks i was proposing were only visible to the
> > timing wheels, which don't use absolute timeouts anyways.
> > So i think the mechanism would be quite safe: right now, when you
> > request a callout after x ticks, the code first picks a CPU
> > (with some criteria), then puts the request in the timer wheel for
> > that CPU using (now) the global 'ticks'. Replacing ticks with cc->cc_ticks,
> > would completely remove the races in insertion and removal.
> > 
> > I actually find the per-cpu ticks even less intrusive than this change.
> 
> Well, it depends.  If TCP ever started using per-CPU callouts (i.e. 
> callout_reset_on())

It seems that this is already the case in practice.

callout_reset() is just #defined to callout_reset_on(c, ... c->cc_cpu)
so all calls end up there.
c->cc_cpu is initialized in callout_init as c->c_cpu = timeout_cpu;
(which is a static int variable; i still don't understand what is
the final value it gets, because the comment says that
kern_timeout_callwheel_alloc() can be called multiple times
and here is where timeout_cpu is initialized.)

cheers
luigi