Re: IO-APIC interrupts getting stuck

["Roger Pau Monné" <roger.pau@citrix.com>]

El 18/12/14 a les 11.41, Jan Beulich ha escrit:
>>>> On 17.12.14 at 13:51, <roger.pau@citrix.com> wrote:
>> I've also added the following patch to Xen, and it reliably triggers on 
>> FreeBSD, while it seems to work fine on Linux:
>>
>> --- a/xen/arch/x86/io_apic.c
>> +++ b/xen/arch/x86/io_apic.c
>> @@ -1729,6 +1729,8 @@ static void end_level_ioapic_irq_new(struct irq_desc *desc, u8 vector)
>>   * The idea is from Manfred Spraul.  --macro
>>   */
>>      unsigned int v, i = desc->arch.vector;
>> +    struct IO_APIC_route_entry rte;
>> +    struct irq_pin_list *entry = irq_2_pin + desc->irq;
>>  
>>      /* Manually EOI the old vector if we are moving to the new */
>>      if ( vector && i != vector )
>> @@ -1751,6 +1753,9 @@ static void end_level_ioapic_irq_new(struct irq_desc *desc, u8 vector)
>>              __unmask_IO_APIC_irq(desc->irq);
>>          spin_unlock(&ioapic_lock);
>>      }
>> +
>> +    rte = ioapic_read_entry(entry->apic, entry->pin, 0);
>> +    ASSERT(rte.irr == 0 || rte.mask != 0);
> 
> Could you arrange for you to be able to determine which code
> paths were taken in the routine when the ASSERT() triggers? I.e.
> minimally make sure vector, i, and v can be determined from the
> printed registers and stack contents. Plus maybe also read the
> applicable APIC_I[RS]R bits.

OK, this was misleading. The ASSERT I've added triggers on Linux also
but it doesn't lead to the irr=1 mask=0 blocked state, so I think
returning from end_level_ioapic_irq_new with irr=1 and mask=0 is a valid
state, is this right?

I've also tested with the following patch applied:

http://lists.xenproject.org/archives/html/xen-devel/2014-12/msg01816.html

To make sure FreeBSD was not playing tricks behind Xen's back, and
AFAICT FreeBSD is not touching the IO APIC at all. Also Xen doesn't show
any pending EOI timers ('a' debug key).

> And you're also apparently not seeing this on a system where
> Linux'es IO-APIC re-route workaround might come into play (see
> drivers/pci/quirks.c:quirk_reroute_to_boot_interrupts_intel()),
> which then I would have suspected FreeBSD to not have such a
> workaround...

No, the same box I have with Linux doesn't use the IO-APIC re-route
quirk. I've also tested this on different hardware and when using
FreeBSD with the new IO APIC Ack method I always end up in this stuck
state, so I don't think it's a hardware errata/issue.

Roger.