Re: xen 4 only seeing one keyboard and mouse

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Thu, Aug 26, 2010 at 11:24:48PM +0100, M A Young wrote:
> On Thu, 26 Aug 2010, M A Young wrote:
> 
> >On Thu, 26 Aug 2010, M A Young wrote:
> >
> >>Okay, here is my first attempt at dirty debugging. I have made a
> >>patch to try to track where vector_irq is being changed
> >>(attached) and have also attached. I have looked at it quickly,
> >>and I don't think some low IRQs are getting set on the second
> >>CPU.

Yeah, that definitly is the problem. Now to why it is happening.
> >
> >My next thoughts on this are that almost all IRQs allocated on the
> >first cpu before the second is started aren't initialized on the
> >second CPU. I presume that __setup_vector_irq from
> >xen/arch/x86/irq.c is where it is supposed to happen
> or perhaps it should happen in io_apic_set_pci_routing from
> xen/arch/x86/io_apic.c where the higher IRQs are set (it doesn't
> because __assign_irq_vector sees the IRQ is already in use
>      old_vector = irq_to_vector(irq);
>     if (old_vector) {
>         cpus_and(tmp_mask, mask, cpu_online_map);
>         cpus_and(tmp_mask, cfg->domain, tmp_mask);
>         if (!cpus_empty(tmp_mask)) {
>             cfg->vector = old_vector;
>             return 0;
>         }
>     }
> but seems to miss the fact that it is only actually configured for
> one cpu.

<nods> That code is actually copied from the Linux kernel .. so that begs
the question how does it work under baremetal?

Lets recap, on Xen we do this in three stages:

 1). Set up all the legacy IRQs. We don't know yet how many CPUs we have
     so we just set up the first sixteen IRQs of the IOAPIC to the first CPU (0).
     We also go through the IDT for CPU0 and set the IDT->IRQ for those legacy
     IRQs.

  2). Then we find out we got more CPUs, and for the other CPUs (1), we
      setup the IDT and we make the IDT->IRQ(-1).

  3). Then when Dom0 starts, we get called for those IRQs once more. And we
      set the IDT for both CPUs to point to the same IRQ:

(XEN) __assign_irq_vector: setting vector_irq[160]=16 for cpu=0
(XEN) __assign_irq_vector: setting vector_irq[160]=16 for cpu=1

	except we don't do it for those that have been already set.
    
So I wonder how this works on baremetal. I've an inkling, but looking at the
boolean logic it doesn't make much sense. So this is the file
arch/x86/kernel/apic/io_apic.c and the function is:  setup_IO_APIC_irq

1442          * For legacy irqs, cfg->domain starts with cpu 0 for legacy
1443          * controllers like 8259. Now that IO-APIC can handle this irq, update
1444          * the cfg->domain.
1445          */
1446         if (irq < legacy_pic->nr_legacy_irqs && cpumask_test_cpu(0, cfg->domain))
1447                 apic->vector_allocation_domain(0, cfg->domain);
1448 
1449         if (assign_irq_vector(irq, cfg, apic->target_cpus()))
1450                 return;
1451 
1452         dest = apic->cpu_mask_to_apicid_and(cfg->domain, apic->target_cpus());
1453 
1454         apic_printk(APIC_VERBOSE,KERN_DEBUG
1455                     "IOAPIC[%d]: Set routing entry (%d-%d -> 0x%x -> "
1456                     "IRQ %d Mode:%i Active:%i)\n",
1457                     apic_id, mp_ioapics[apic_id].apicid, pin, cfg->vector,
1458                     irq, trigger, polarity);

If you could, can you instrument it to print the cfg->domain, before the 'vector_allocation_domain'
is called, and as well instrument the assign_irq_vector similary to what you did with Xen?

And also instrument the 'dest' value. Basically the idea is to get an idea of what the
per_cpu(vector) gets set during the bootup for legacy IRQs. Similary to what you did
with Xen.