Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 04/27/2010 11:47 PM, John McCullough wrote:
> I did a little testing.
>
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet
> console=hvc0 nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS
> output, but unfortunately that doesn't change nr_irqs and I run into
> the same limit (36 domus on a less-beefy dual core machine).

Yes, NR_IRQS is the hard limit (for any statically defined irq arrays,
which are deprecated now), but nr_irqs is the amount it decides to
actually allocate for dynamic irq arrays, and so represents the actual
runtime limit.

nr_irqs is computed in arch_probe_nr_irqs(), and its a function of the
number of cpus, with a bump to deal with dynamically allocated MSI
interrupts.  I should probably add something to specifically add more if
we're running under Xen, at least as a workaround (ultimately the plan
is to make all irqs completely dynamically allocated so there is no hard
limit).

>
> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops
> kernel.

I'm pretty sure that's referring to Solaris dom0, so the fact that
there's a similarly named symbol is coincidence.  (But the root problem
is the same.)

    J