Re: pvops-2.6.32 - Interrupt routing problem

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Mar 16, 2010 at 07:20:53PM +0100, Bastian Blank wrote:
> On Tue, Mar 16, 2010 at 11:32:16AM -0400, Konrad Rzeszutek Wilk wrote:
> > I wish you had included the whole serial console for Xen boot.
> 
> That was the intention but I had to run. Here is it (slightly
> annotated).
> 
> >                                                                I am
> > curious to at what stage the pin 14 of the IOAPIC is set. Is it set by Xen
> > hypervisor initially (don't think so),
> 
> No, the entries on the first IO-APIC for the devices in question are
> completely bogus in the initial printout. Unmasked but edge driven.

I am going to need your help here. I *think* the issue is that Xen
parses the MADT table, which makes the assumption that the 
ISA interrupt vectors will usually be identity-mapped into the first 16 I/O APIC
sources (and the INT_SRC_OVR are used to override those IO APIC
pins with new entries). I think it does this:

	acpi_parse_madt_ioapic_entries calls
		mp_config_acpi_legacy_irqs:
		.. snip ..
	         intsrc.mpc_irqflag = 0;       /* Conforming */

Which assumes that it is edge, high. I think if you fix this line:

1089                 Dprintk("Int: type %d, pol %d, trig %d, bus %d, irq
%d, "

to do a printk we will see that your pin 14 is programmed to edge(0),
high(0). Which by itself is exactly the same thing Linux kernel in
baremetal does.

But that does not explain why you can't reprogram it. There is that check
you subverted which sets this:

	mp_ioapic_routing[ioapic].pin_programmed[idx] |= (1<<bit);

but I only see that being set when you call 'mp_register_gsi'. And that
is only done by the Linux kernel, not by the hypervisor. Furthermore
there is a guard for that allows it to be called only for GSI above 15.

Can you instrument mp_register_gsi to see when that 'pin_programmed' is
set? Or maybe your grep magic is better than mine and can you see other
instances where the mp_ioapic_routing is modified and instrument that?

I think once we figure out when the 'pin_programmed' is set we can
back-trace why that is set.
> >                                        or is there another piece of code
> > in the pv-ops that iterates over the 0-15 IRQs and sets them to the
> > legacy trigger/level?
> 
> Before the initialisation of the console, the kernel seem to set pins 5
> to 15 on the first IO-APIC to all edge driven.

Yeah, the more I look up data sheets that seems to be right way.
> 
> > When you printed out the IOAPIC information, did the Xen one (in the
> > hypervisor, when you did apic=debug during boot) have pin 14 set to the
> > wrong mode?
> 
> Yes.

Which is OK, b/c when APIC turns on, then you should be able to program it.
That is what Linux on baremetal does.
> 
> > Or was it later when the pv-ops kernel booted that the pin 14 was set
> > incorrectly?
> 
> The kernel sets it to this values again.

You mean it sets it to level (1), low (1) for pin 14 and 5.