From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51241) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XPyi8-0002Vh-95 for qemu-devel@nongnu.org; Fri, 05 Sep 2014 14:58:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XPyi2-0006qM-3J for qemu-devel@nongnu.org; Fri, 05 Sep 2014 14:58:16 -0400 Received: from mxa.qnx.com ([72.1.200.108]:35075) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XPyi1-0006qH-UN for qemu-devel@nongnu.org; Fri, 05 Sep 2014 14:58:10 -0400 From: Richard Bilson Date: Fri, 5 Sep 2014 18:58:07 +0000 Message-ID: References: <5409ECC6.20808@redhat.com> In-Reply-To: <5409ECC6.20808@redhat.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [question] PIC and APIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini , "qemu-devel@nongnu.org" On 2014-09-05, 13:03, "Paolo Bonzini" wrote: >Il 05/09/2014 17:51, Richard Bilson ha scritto: >> Greetings folks, >>=20 >> I've been tracking down certain obscure failures that we see running our >> kernel on qemu x86 smp. Based on a few days diving into qemu internals >>it >> seems that the problems relate to mixing PIC and APIC-generated >> interrupts. I'd appreciate some help in understanding if this is a >> limitation of qemu, or if we are misconfiguring the system in some way. >>=20 >> Scenario: >> - All hardware interrupts come via the PIC, and are delivered by the >>cpu 0 >> LAPIC in ExtINT mode >> - IPIs are delivered by the LAPIC in fixed mode >>=20 >> Failure mode #1: >> - IPI sent to cpu 0 (bit set in APIC irr) >> - IPI accepted by cpu 0 (bit cleared in irr, set in isr) >> - IPI sent to cpu 0 (bit set in both irr and isr) >> - PIC interrupt sent to cpu 0 >>=20 >> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but >> apic_irq_pending observes that the highest pending APIC interrupt >>priority >> (the IPI) is the same as the processor priority (since the IPI is still >> being handled), so apic_get_interrupt returns a spurious interrupt >>rather >> than the pending PIC interrupt. The result is an endless sequence of >> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD. > >ExtINT interrupts should have ignored the processor priority, right? That's my understanding, yes. >> Failure mode #2: >> - cpu 0 masks a particular PIC interrupt >> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set) >> - before the IPI is accepted, the masked interrupt line is asserted by >>the >> device >>=20 >> Since the interrupt is masked, apic_deliver_pic_intr will clear >> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since >> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the >> scenario this can cause a system hang, i.e. if cpu 0 is expected to >>unmask >> the interrupt. >>=20 >> The commonality here seems to be that the APIC code is attempting to use >> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one >>of >> these purposes blocks the other. It seems that we should be able to >>solve >> the problem on our side by using a pure-APIC interrupt routing (and >>indeed >> we haven't observed these problems when so configured), but we've never >> noticed these kinds of problems on real hardware using legacy PIC mode. >>=20 >> Any help in confirming or refuting the above analysis is appreciated. > >It sounds plausible. Can you confirm that you are using TCG >(emulation)? Does it work with KVM? What about KVM with -machine >kernel_irqchip=3Doff? I am definitely using TCG. I will try the KVM variants asap. >It would be great if you could reproduce the issue using the harness at >http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just >as well without KVM). I'll look into this as well.