[Qemu-devel] Network connections stalling (due to lost interrupts/ticks?)

[Qemu-devel] Network connections stalling (due to lost interrupts/ticks?)

[Charles Duffy]

I'm trying to use qemu to test an install process which involves quite a 
bit of downloading. Everything starts up fine (using either ne2k_pci or 
rtl8139 hardware), but the multi-GB download typically stalls out about 
100-400MB in. Is there anything I can do to prevent this?

There's a warning on startup that the system can't set a 1024Hz timer, 
which persists even after I set /proc/sys/dev/rtc/max-user-freq to 1024, 
and I occasionally get warnings at runtime ("Your time source seems to 
be instable or some driver is hogging interrupts").

[Qemu-devel] Re: Network connections stalling (due to lost interrupts/ticks?)

[Charles Duffy]

Charles Duffy wrote:
> There's a warning on startup that the system can't set a 1024Hz timer,
> which persists even after I set /proc/sys/dev/rtc/max-user-freq to 1024,
> and I occasionally get warnings at runtime ("Your time source seems to
> be instable or some driver is hogging interrupts").

This was happening because my host kernel was compiled with
CONFIG_HPET_RTC_IRQ=y. I've disabled this option, recompiled and
rebooted, and it resolved the RTC warning (and apparently, the unstable
time source messages) -- but my network connections are still stalling.

Re: [Qemu-devel] Re: Network connections stalling (due to lost interrupts/ticks?)

[Jason Wessel]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

Charles,

Are you willing to try an experimental patch?

Perhaps you could try the attached patch and post back if it happens to 
solve your problem.  There is most definitely a problem where qemu can 
get hung up indefinitely after an "interrupt storm".  I had not ever 
submitted it because there is no clean way to do this via the opaque 
information that is passed around.  It seems wrong to have to make the 
ioapic a global.  If this does fix the problem perhaps someone will 
decide to fix this up in a cleaner fashion via the opaque structures.

Jason.

Charles Duffy wrote:
> Charles Duffy wrote:
>   
>> There's a warning on startup that the system can't set a 1024Hz timer,
>> which persists even after I set /proc/sys/dev/rtc/max-user-freq to 1024,
>> and I occasionally get warnings at runtime ("Your time source seems to
>> be instable or some driver is hogging interrupts").
>>     
>
> This was happening because my host kernel was compiled with
> CONFIG_HPET_RTC_IRQ=y. I've disabled this option, recompiled and
> rebooted, and it resolved the RTC warning (and apparently, the unstable
> time source messages) -- but my network connections are still stalling.
>
>
>
>   


[-- Attachment #2: io_apic_eoi_fix.patch --]
[-- Type: text/x-patch, Size: 1706 bytes --]


Recover from an interupt flood by propagating the end of interrupt state. 

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 hw/apic.c |   23 +++++++++++++++++++++--
 hw/pc.c   |    2 +-
 2 files changed, 22 insertions(+), 3 deletions(-)

Index: qemu/hw/apic.c
===================================================================
--- qemu.orig/hw/apic.c
+++ qemu/hw/apic.c
@@ -332,6 +332,26 @@ static void apic_set_irq(APICState *s, i
     apic_update_irq(s);
 }
 
+struct IOAPICState *ioapic;
+/* XXX Multi IOAPIC support */
+static void apic_propogate_eoi(int vector) {
+    uint32_t irr;
+    int pin;
+
+    if ((vector < 0x10) || (vector > 0xfe))
+        return;
+
+    irr = ioapic->irr;
+    while (irr) {
+        pin = ffs_bit(irr);
+        irr &= ~(1 << pin);
+        if ((ioapic->ioredtbl[pin] & 0xff) == vector) {
+            ioapic->irr &= ~(1 << pin);
+            break;
+        }
+    }
+}
+
 static void apic_eoi(APICState *s)
 {
     int isrv;
@@ -339,8 +359,7 @@ static void apic_eoi(APICState *s)
     if (isrv < 0)
         return;
     reset_bit(s->isr, isrv);
-    /* XXX: send the EOI packet to the APIC bus to allow the I/O APIC to
-            set the remote IRR bit for level triggered interrupts. */
+    apic_propogate_eoi(isrv);
     apic_update_irq(s);
 }
 
Index: qemu/hw/pc.c
===================================================================
--- qemu.orig/hw/pc.c
+++ qemu/hw/pc.c
@@ -36,7 +36,7 @@
 static fdctrl_t *floppy_controller;
 static RTCState *rtc_state;
 static PITState *pit;
-static IOAPICState *ioapic;
+extern IOAPICState *ioapic;
 static PCIDevice *i440fx_state;
 
 static void ioport80_write(void *opaque, uint32_t addr, uint32_t data)

[Qemu-devel] Re: Network connections stalling (due to lost interrupts/ticks?)

[Charles Duffy]

Well, behavior with the patch applied is certainly different.

The large download I'm running still times out; however, it is now able
to resume without needing to bring the interface down and back up.
However, after the first timeout, subsequent timeouts occur with much
greater frequency -- still making this multi-GB download an
impracticality when using -net tap.

The flood ping is not killing the network connection, though it is
interrupted by frequent messages: "Warning: time of day goes back
(-23150us), taking countermeasures". (This is no the high end of the
time variances shown; the smallest are on the scale of 120us

[Qemu-devel] Re: Network connections stalling (due to lost interrupts/ticks?)

[Charles Duffy]

[Resending as the last copy seems to have been lost in the ether -- it
certainly isn't in the GMANE archive]

Well, behavior with the patch applied is certainly different.

The large download I'm running still times out; however, it is now able
to resume without needing to bring the interface down and back up.
However, after the first timeout, subsequent timeouts occur with much
greater frequency -- making this multi-GB download still an
impracticality when using -net tap.

The flood ping is not killing the network connection, though it is
interrupted by frequent messages: "Warning: time of day goes back
(-23150us), taking countermeasures". (This is no the high end of the
time variances shown; the smallest are on the scale of 120us).


[btw -- Wind River? heh -- 'yall were The Great Enemy back when I was at
MontaVista around 2000-2002 or so]

[Qemu-devel] Re: Network connections stalling

[Charles Duffy]

This loss of connectivity is still happening; at least within my
environment, it's quite reproducible. While I've eliminated the message
about being unable to set the RTC to 1024Hz by recompiling my host
kernel without CONFIG_HPET_RTC_IRQ, the host's kernel is frequently
complaining "rtc: lost some interrupts at 1024Hz".

I'm using tap-based networking (-net nic -net
tap,ifname=tap0,script=./tap0-start), where tap0 is bridged to my local
ethernet device, and outgoing network traffic from the qemu instance is
visible if I monitor traffic on the bridge.

Indeed, external responses to such traffic (ie. ARP responses) are
visible on the bridge, but evidently not to the virtual machine: the TX
counter rises when I try to ping the outside world, but the RX counter
does not. Resetting the device (ifconfig eth0 down; ifconfig eth0 up)
brings it back alive -- but this isn't an action which can trivially be
slotted into a trivial, single-threaded install script.

These same scripts have been tested on real hardware, and have not
turned up this issue. Certainly, a race condition unrelated to qemu is
possible, but I'd appreciate any insight 'yall can provide. Thank you!

[Qemu-devel] Re: Network connections stalling

[Charles Duffy]

This appears to only happen with -net tap; I cannot reproduce the issue
with -net user.