[Qemu-devel] [question] PIC and APIC

[Qemu-devel] [question] PIC and APIC

[Richard Bilson]

Greetings folks,

I've been tracking down certain obscure failures that we see running our
kernel on qemu x86 smp. Based on a few days diving into qemu internals it
seems that the problems relate to mixing PIC and APIC-generated
interrupts. I'd appreciate some help in understanding if this is a
limitation of qemu, or if we are misconfiguring the system in some way.

Scenario:
- All hardware interrupts come via the PIC, and are delivered by the cpu 0
LAPIC in ExtINT mode
- IPIs are delivered by the LAPIC in fixed mode

Failure mode #1:
- IPI sent to cpu 0 (bit set in APIC irr)
- IPI accepted by cpu 0 (bit cleared in irr, set in isr)
- IPI sent to cpu 0 (bit set in both irr and isr)
- PIC interrupt sent to cpu 0

The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
apic_irq_pending observes that the highest pending APIC interrupt priority
(the IPI) is the same as the processor priority (since the IPI is still
being handled), so apic_get_interrupt returns a spurious interrupt rather
than the pending PIC interrupt. The result is an endless sequence of
spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.

Failure mode #2:
- cpu 0 masks a particular PIC interrupt
- IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
- before the IPI is accepted, the masked interrupt line is asserted by the
device

Since the interrupt is masked, apic_deliver_pic_intr will clear
CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
scenario this can cause a system hang, i.e. if cpu 0 is expected to unmask
the interrupt.

The commonality here seems to be that the APIC code is attempting to use
the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one of
these purposes blocks the other. It seems that we should be able to solve
the problem on our side by using a pure-APIC interrupt routing (and indeed
we haven't observed these problems when so configured), but we've never
noticed these kinds of problems on real hardware using legacy PIC mode.

Any help in confirming or refuting the above analysis is appreciated.

Thanks,
Richard Bilson
QNX Software Systems Ltd.

Re: [Qemu-devel] [question] PIC and APIC

[Paolo Bonzini]

Il 05/09/2014 17:51, Richard Bilson ha scritto:
> Greetings folks,
> 
> I've been tracking down certain obscure failures that we see running our
> kernel on qemu x86 smp. Based on a few days diving into qemu internals it
> seems that the problems relate to mixing PIC and APIC-generated
> interrupts. I'd appreciate some help in understanding if this is a
> limitation of qemu, or if we are misconfiguring the system in some way.
> 
> Scenario:
> - All hardware interrupts come via the PIC, and are delivered by the cpu 0
> LAPIC in ExtINT mode
> - IPIs are delivered by the LAPIC in fixed mode
> 
> Failure mode #1:
> - IPI sent to cpu 0 (bit set in APIC irr)
> - IPI accepted by cpu 0 (bit cleared in irr, set in isr)
> - IPI sent to cpu 0 (bit set in both irr and isr)
> - PIC interrupt sent to cpu 0
> 
> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
> apic_irq_pending observes that the highest pending APIC interrupt priority
> (the IPI) is the same as the processor priority (since the IPI is still
> being handled), so apic_get_interrupt returns a spurious interrupt rather
> than the pending PIC interrupt. The result is an endless sequence of
> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.

ExtINT interrupts should have ignored the processor priority, right?

> Failure mode #2:
> - cpu 0 masks a particular PIC interrupt
> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
> - before the IPI is accepted, the masked interrupt line is asserted by the
> device
> 
> Since the interrupt is masked, apic_deliver_pic_intr will clear
> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
> scenario this can cause a system hang, i.e. if cpu 0 is expected to unmask
> the interrupt.
> 
> The commonality here seems to be that the APIC code is attempting to use
> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one of
> these purposes blocks the other. It seems that we should be able to solve
> the problem on our side by using a pure-APIC interrupt routing (and indeed
> we haven't observed these problems when so configured), but we've never
> noticed these kinds of problems on real hardware using legacy PIC mode.
> 
> Any help in confirming or refuting the above analysis is appreciated.

It sounds plausible.  Can you confirm that you are using TCG
(emulation)?  Does it work with KVM?  What about KVM with -machine
kernel_irqchip=off?

It would be great if you could reproduce the issue using the harness at
http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just
as well without KVM).

Paolo

Re: [Qemu-devel] [question] PIC and APIC

[Richard Bilson]

On 2014-09-05, 13:03, "Paolo Bonzini" <pbonzini@redhat.com> wrote:
>Il 05/09/2014 17:51, Richard Bilson ha scritto:
>> Greetings folks,
>> 
>> I've been tracking down certain obscure failures that we see running our
>> kernel on qemu x86 smp. Based on a few days diving into qemu internals
>>it
>> seems that the problems relate to mixing PIC and APIC-generated
>> interrupts. I'd appreciate some help in understanding if this is a
>> limitation of qemu, or if we are misconfiguring the system in some way.
>> 
>> Scenario:
>> - All hardware interrupts come via the PIC, and are delivered by the
>>cpu 0
>> LAPIC in ExtINT mode
>> - IPIs are delivered by the LAPIC in fixed mode
>> 
>> Failure mode #1:
>> - IPI sent to cpu 0 (bit set in APIC irr)
>> - IPI accepted by cpu 0 (bit cleared in irr, set in isr)
>> - IPI sent to cpu 0 (bit set in both irr and isr)
>> - PIC interrupt sent to cpu 0
>> 
>> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
>> apic_irq_pending observes that the highest pending APIC interrupt
>>priority
>> (the IPI) is the same as the processor priority (since the IPI is still
>> being handled), so apic_get_interrupt returns a spurious interrupt
>>rather
>> than the pending PIC interrupt. The result is an endless sequence of
>> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.
>
>ExtINT interrupts should have ignored the processor priority, right?

That's my understanding, yes.

>> Failure mode #2:
>> - cpu 0 masks a particular PIC interrupt
>> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
>> - before the IPI is accepted, the masked interrupt line is asserted by
>>the
>> device
>> 
>> Since the interrupt is masked, apic_deliver_pic_intr will clear
>> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
>> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
>> scenario this can cause a system hang, i.e. if cpu 0 is expected to
>>unmask
>> the interrupt.
>> 
>> The commonality here seems to be that the APIC code is attempting to use
>> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one
>>of
>> these purposes blocks the other. It seems that we should be able to
>>solve
>> the problem on our side by using a pure-APIC interrupt routing (and
>>indeed
>> we haven't observed these problems when so configured), but we've never
>> noticed these kinds of problems on real hardware using legacy PIC mode.
>> 
>> Any help in confirming or refuting the above analysis is appreciated.
>
>It sounds plausible.  Can you confirm that you are using TCG
>(emulation)?  Does it work with KVM?  What about KVM with -machine
>kernel_irqchip=off?

I am definitely using TCG. I will try the KVM variants asap.

>It would be great if you could reproduce the issue using the harness at
>http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just
>as well without KVM).

I'll look into this as well.