[PATCH] x86: fix probe_nr_irqs for xen

[PATCH] x86: fix probe_nr_irqs for xen

[Yinghai Lu]

otherwise Xen is _completely_ unusable with 5 or more VCPUs.
( when !CONFIG_HAVE_SPARSE_IRQ)

based on Alex's patch

also add +1 offset after redir_entries

Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
Cc: Alex Nixon <alex.nixon@citrix.com>

Index: linux-2.6/arch/x86/kernel/io_apic.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/io_apic.c
+++ linux-2.6/arch/x86/kernel/io_apic.c
@@ -3625,16 +3625,21 @@ int __init probe_nr_irqs(void)
 {
 	int idx;
 	int nr = 0;
+#ifndef CONFIG_XEN
+	int nr_min = 32;
+#else
+	int nr_min = NR_IRQS;
+#endif
 
 	for (idx = 0; idx < nr_ioapics; idx++)
-		nr += io_apic_get_redir_entries(idx);
+		nr += io_apic_get_redir_entries(idx) + 1;
 
 	/* double it for hotplug and msi and nmi */
 	nr <<= 1;
 
 	/* something wrong ? */
-	if (nr < 32)
-		nr = 32;
+	if (nr < nr_min)
+		nr = nr_min;
 
 	return nr;
 }

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Jeremy Fitzhardinge]

Yinghai Lu wrote:
> otherwise Xen is _completely_ unusable with 5 or more VCPUs.
> ( when !CONFIG_HAVE_SPARSE_IRQ)
>
> based on Alex's patch
>
> also add +1 offset after redir_entries
>
> Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
> Cc: Alex Nixon <alex.nixon@citrix.com>
>   

Looks good to me.  We can fix this in a better way later.

Acked-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

> Index: linux-2.6/arch/x86/kernel/io_apic.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/io_apic.c
> +++ linux-2.6/arch/x86/kernel/io_apic.c
> @@ -3625,16 +3625,21 @@ int __init probe_nr_irqs(void)
>  {
>  	int idx;
>  	int nr = 0;
> +#ifndef CONFIG_XEN
> +	int nr_min = 32;
> +#else
> +	int nr_min = NR_IRQS;
> +#endif
>  
>  	for (idx = 0; idx < nr_ioapics; idx++)
> -		nr += io_apic_get_redir_entries(idx);
> +		nr += io_apic_get_redir_entries(idx) + 1;
>  
>  	/* double it for hotplug and msi and nmi */
>  	nr <<= 1;
>  
>  	/* something wrong ? */
> -	if (nr < 32)
> -		nr = 32;
> +	if (nr < nr_min)
> +		nr = nr_min;
>  
>  	return nr;
>  }
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Yinghai Lu]

On Thu, Aug 21, 2008 at 4:33 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Yinghai Lu wrote:
>> otherwise Xen is _completely_ unusable with 5 or more VCPUs.
>> ( when !CONFIG_HAVE_SPARSE_IRQ)
>>
>> based on Alex's patch
>>
>> also add +1 offset after redir_entries
>>
>> Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
>> Cc: Alex Nixon <alex.nixon@citrix.com>
>>
>
> Looks good to me.  We can fix this in a better way later.

later could set CONFIG_HAVE_SPARSE_IRQ to true by default.

YH

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Jeremy Fitzhardinge]

Yinghai Lu wrote:
> On Thu, Aug 21, 2008 at 4:33 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>   
>> Yinghai Lu wrote:
>>     
>>> otherwise Xen is _completely_ unusable with 5 or more VCPUs.
>>> ( when !CONFIG_HAVE_SPARSE_IRQ)
>>>
>>> based on Alex's patch
>>>
>>> also add +1 offset after redir_entries
>>>
>>> Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
>>> Cc: Alex Nixon <alex.nixon@citrix.com>
>>>
>>>       
>> Looks good to me.  We can fix this in a better way later.
>>     
>
> later could set CONFIG_HAVE_SPARSE_IRQ to true by default.
>   

Well, perhaps.  But we can use vectors rather than IRQs for Xen event
channel mappings, which will save us from having to allocate lots of
per-cpu irqs.

Also, implementing probe_nr_irqs() in a more generic (rather than
io_apic-specific) way.  Using some interface that queries each irq chip
for how many irqs it supports, or something.

    J

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Yinghai Lu]

On Thu, Aug 21, 2008 at 4:52 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Yinghai Lu wrote:
>> On Thu, Aug 21, 2008 at 4:33 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>>
>>> Yinghai Lu wrote:
>>>
>>>> otherwise Xen is _completely_ unusable with 5 or more VCPUs.
>>>> ( when !CONFIG_HAVE_SPARSE_IRQ)
>>>>
>>>> based on Alex's patch
>>>>
>>>> also add +1 offset after redir_entries
>>>>
>>>> Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
>>>> Cc: Alex Nixon <alex.nixon@citrix.com>
>>>>
>>>>
>>> Looks good to me.  We can fix this in a better way later.
>>>
>>
>> later could set CONFIG_HAVE_SPARSE_IRQ to true by default.
>>
>
> Well, perhaps.  But we can use vectors rather than IRQs for Xen event
> channel mappings, which will save us from having to allocate lots of
> per-cpu irqs.

like system vectors with smp-ipi etc?

>
> Also, implementing probe_nr_irqs() in a more generic (rather than
> io_apic-specific) way.  Using some interface that queries each irq chip
> for how many irqs it supports, or something.

hope we can kill nr_irqs/NR_IRQS

YH

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Jeremy Fitzhardinge]

Yinghai Lu wrote:
> like system vectors with smp-ipi etc?
>   

More or less.  Xen currently uses 6 per-cpu event channels, and it maps
each to its own irq.  My plan is to allocate just 6 irqs, map 6 vectors
to them, and bind each event channel to a vector/cpu pair.  If nothing
else, it will make /proc/interrupts somewhat sane again.


>> Also, implementing probe_nr_irqs() in a more generic (rather than
>> io_apic-specific) way.  Using some interface that queries each irq chip
>> for how many irqs it supports, or something.
>>     
>
> hope we can kill nr_irqs/NR_IRQS
>   

When irqs are truely dynamically allocated?

    J

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Yinghai Lu]

On Thu, Aug 21, 2008 at 5:02 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Yinghai Lu wrote:
>> like system vectors with smp-ipi etc?
>>
>
> More or less.  Xen currently uses 6 per-cpu event channels, and it maps
> each to its own irq.  My plan is to allocate just 6 irqs, map 6 vectors
> to them, and bind each event channel to a vector/cpu pair.  If nothing
> else, it will make /proc/interrupts somewhat sane again.
>
>
>>> Also, implementing probe_nr_irqs() in a more generic (rather than
>>> io_apic-specific) way.  Using some interface that queries each irq chip
>>> for how many irqs it supports, or something.
>>>
>>
>> hope we can kill nr_irqs/NR_IRQS
>>
>
> When irqs are truely dynamically allocated?
>
need more test with sparse_irq

YH

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Ingo Molnar]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> > hope we can kill nr_irqs/NR_IRQS
> 
> When irqs are truely dynamically allocated?

they in essence already are truly dynamic in tip/master. Not yet 
unconditionally so on x86, we chose to also have a !SPARSEIRQ build 
model to make sure dense IRQ arrays still work fine as well. (that's far 
easier to test and validate than to keep a farm of cross compilers to 
other arches alive.)

But by all means SPARSEIRQ is the primary and only model on x86.

We could even plug in the dynamic vector allocator from Alan Mayer now, 
both on 32-bit and on 64-bit x86: Yinghai, any complications expected 
there? I guess we need to unify the local APIC code first.

	Ingo

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Yinghai Lu]

On Thu, Aug 21, 2008 at 9:26 PM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>
>> > hope we can kill nr_irqs/NR_IRQS
>>
>> When irqs are truely dynamically allocated?
>
> they in essence already are truly dynamic in tip/master. Not yet
> unconditionally so on x86, we chose to also have a !SPARSEIRQ build
> model to make sure dense IRQ arrays still work fine as well. (that's far
> easier to test and validate than to keep a farm of cross compilers to
> other arches alive.)
>
> But by all means SPARSEIRQ is the primary and only model on x86.
>
> We could even plug in the dynamic vector allocator from Alan Mayer now,
> both on 32-bit and on 64-bit x86: Yinghai, any complications expected
> there? I guess we need to unify the local APIC code first.

apic_32.c and apic_64.c
irqinit_32.c and irqinit_64.c
irq_32.c and irq_64.c

Cyrill, is still working on merging apic_32.c apic_64.c.?

YH

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Ingo Molnar]

* Yinghai Lu <yhlu.kernel@gmail.com> wrote:

> > But by all means SPARSEIRQ is the primary and only model on x86.
> >
> > We could even plug in the dynamic vector allocator from Alan Mayer 
> > now, both on 32-bit and on 64-bit x86: Yinghai, any complications 
> > expected there? I guess we need to unify the local APIC code first.
> 
> apic_32.c and apic_64.c
> irqinit_32.c and irqinit_64.c
> irq_32.c and irq_64.c
> 
> Cyrill, is still working on merging apic_32.c apic_64.c.?

there's a few weeks of inactivity expected in that area i think - so if 
you'd like to pick it up from there, feel free ...

but apic_32/64.c is looking already pretty good in tip/master:

 1 file changed, 391 insertions(+), 358 deletions(-)

the most significant bit of that delta seems to be around x2apic, 
calibration differences and some older 32-bit legacies which are not on 
the 64-bit side (but which we can add safely).

	Ingo

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Cyrill Gorcunov]

[Ingo Molnar - Fri, Aug 22, 2008 at 07:16:56AM +0200]
| 
| * Yinghai Lu <yhlu.kernel@gmail.com> wrote:
| 
| > > But by all means SPARSEIRQ is the primary and only model on x86.
| > >
| > > We could even plug in the dynamic vector allocator from Alan Mayer 
| > > now, both on 32-bit and on 64-bit x86: Yinghai, any complications 
| > > expected there? I guess we need to unify the local APIC code first.
| > 
| > apic_32.c and apic_64.c
| > irqinit_32.c and irqinit_64.c
| > irq_32.c and irq_64.c
| > 
| > Cyrill, is still working on merging apic_32.c apic_64.c.?
| 
| there's a few weeks of inactivity expected in that area i think - so if 
| you'd like to pick it up from there, feel free ...
| 
| but apic_32/64.c is looking already pretty good in tip/master:
| 
|  1 file changed, 391 insertions(+), 358 deletions(-)
| 
| the most significant bit of that delta seems to be around x2apic, 
| calibration differences and some older 32-bit legacies which are not on 
| the 64-bit side (but which we can add safely).
| 
| 	Ingo
| 

Yes - I will be busy for a few weeks with other duties, sorry for
that. There not that much diff rest:

- apic clock calibration
- x2apic operations
- interrupt handlers (different declarations)
- command line handling

ugh... maybe not that small heap of todo's :)

Anyway - before sending patches I'll fetch tip/x86/apic
branch so no problem if someone bring any changes into
the branch.

		- Cyrill -

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Ingo Molnar]

* Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> Yinghai Lu wrote:
> > otherwise Xen is _completely_ unusable with 5 or more VCPUs.
> > ( when !CONFIG_HAVE_SPARSE_IRQ)
> >
> > based on Alex's patch
> >
> > also add +1 offset after redir_entries
> >
> > Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
> > Cc: Alex Nixon <alex.nixon@citrix.com>
> >   
> 
> Looks good to me.  We can fix this in a better way later.
> 
> Acked-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

applied to tip/irq/sparseirq. 

Alex, does this approach work for you too?

	Ingo

Re: [PATCH] x86: fix probe_nr_irqs for xen

[Alex Nixon]

Ingo Molnar wrote:
> * Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> 
>> Yinghai Lu wrote:
>>> otherwise Xen is _completely_ unusable with 5 or more VCPUs.
>>> ( when !CONFIG_HAVE_SPARSE_IRQ)
>>>
>>> based on Alex's patch
>>>
>>> also add +1 offset after redir_entries
>>>
>>> Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
>>> Cc: Alex Nixon <alex.nixon@citrix.com>
>>>   
>> Looks good to me.  We can fix this in a better way later.
>>
>> Acked-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> 
> applied to tip/irq/sparseirq. 
> 
> Alex, does this approach work for you too?
> 
> 	Ingo

Yep.

- Alex