[PATCH] sparseirq: Increase nr_irqs if needed

[PATCH] sparseirq: Increase nr_irqs if needed

[Yinghai Lu]

only do that when we need more.

For x86_64 system:
when we have 128 cpus with 5 ioapics, will have nr_irqs = 3064
	120 + 8 * 128 + 120 * 16
systems could take 20 pcie,  when intel 10g are used with
sriov and ixgbevf, every vf will need 3 irqs, and one device
have 64 vf. so will need 20 * 3 * 64 = 3840.
some 6 ports Intel 10gb may need more.

with this patch, nr_irqs will increase 25% every time if nr_irqs is not
big enough.

Also later We could change nr_irqs initial value to
	 min(nr_irqs_gsi, 224*nr_cpu_ids).
for x86.

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 kernel/irq/irqdesc.c |   29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)

Index: linux-2.6/kernel/irq/irqdesc.c
===================================================================
--- linux-2.6.orig/kernel/irq/irqdesc.c
+++ linux-2.6/kernel/irq/irqdesc.c
@@ -301,6 +301,27 @@ void irq_free_descs(unsigned int from, u
 	mutex_unlock(&sparse_irq_lock);
 }
 
+static bool increase_nr_irqs(void)
+{
+#ifdef CONFIG_SPARSE_IRQ
+	int new;
+
+	if (nr_irqs == NR_IRQS)
+		return false;
+
+	new = nr_irqs;
+	new += nr_irqs >> 2;
+	if (new > NR_IRQS)
+		new = NR_IRQS;
+	if (new > nr_irqs) {
+		nr_irqs = new;
+		printk(KERN_INFO "nr_irqs increase to %d\n", nr_irqs);
+		return true;
+	}
+#endif
+	return false;
+}
+
 /**
  * irq_alloc_descs - allocate and initialize a range of irq descriptors
  * @irq:	Allocate for specific irq number if irq >= 0
@@ -320,14 +341,16 @@ irq_alloc_descs(int irq, unsigned int fr
 
 	mutex_lock(&sparse_irq_lock);
 
-	start = bitmap_find_next_zero_area(allocated_irqs, nr_irqs, from, cnt, 0);
+	start = bitmap_find_next_zero_area(allocated_irqs, NR_IRQS, from, cnt, 0);
 	ret = -EEXIST;
 	if (irq >=0 && start != irq)
 		goto err;
 
 	ret = -ENOMEM;
-	if (start >= nr_irqs)
-		goto err;
+	while ((start + cnt) >= nr_irqs) {
+		if (!increase_nr_irqs())
+			goto err;
+	}
 
 	bitmap_set(allocated_irqs, start, cnt);
 	mutex_unlock(&sparse_irq_lock);

[PATCH] x86, sparseirq: let nr_irqs equal to NR_IRQS

[Yinghai Lu]

For x86_64 system:
when we have 128 cpus with 5 ioapics, will have nr_irqs = 3064
	120 + 8 * 128 + 120 * 16
systems could take 20 pcie,  when intel 10g are used with
sriov and ixgbevf, every vf will need 3 irqs, and one device
have 64 vf. so will need 20 * 3 * 64 = 3840.
some 6 ports Intel 10gb may need more.

Just remove that function for x86, and let nr_irqs to NR_IRQS
because We already have radix-tree and bit_map for searching desc for irq.

Notes: long before same vresion cause one of Ingo's setup udev hang...

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/kernel/apic/io_apic.c |   22 ----------------------
 1 file changed, 22 deletions(-)

Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
+++ linux-2.6/arch/x86/kernel/apic/io_apic.c
@@ -3637,28 +3637,6 @@ int get_nr_irqs_gsi(void)
 	return nr_irqs_gsi;
 }
 
-#ifdef CONFIG_SPARSE_IRQ
-int __init arch_probe_nr_irqs(void)
-{
-	int nr;
-
-	if (nr_irqs > (NR_VECTORS * nr_cpu_ids))
-		nr_irqs = NR_VECTORS * nr_cpu_ids;
-
-	nr = nr_irqs_gsi + 8 * nr_cpu_ids;
-#if defined(CONFIG_PCI_MSI) || defined(CONFIG_HT_IRQ)
-	/*
-	 * for MSI and HT dyn irq
-	 */
-	nr += nr_irqs_gsi * 16;
-#endif
-	if (nr < nr_irqs)
-		nr_irqs = nr;
-
-	return NR_IRQS_LEGACY;
-}
-#endif
-
 static int __io_apic_set_pci_routing(struct device *dev, int irq,
 				struct io_apic_irq_attr *irq_attr)
 {

Re: [PATCH] x86, sparseirq: let nr_irqs equal to NR_IRQS

[Konrad Rzeszutek Wilk]

On Mon, Dec 27, 2010 at 04:54:11PM -0800, Yinghai Lu wrote:
> 
> 
> For x86_64 system:
> when we have 128 cpus with 5 ioapics, will have nr_irqs = 3064
> 	120 + 8 * 128 + 120 * 16
> systems could take 20 pcie,  when intel 10g are used with
> sriov and ixgbevf, every vf will need 3 irqs, and one device
> have 64 vf. so will need 20 * 3 * 64 = 3840.
> some 6 ports Intel 10gb may need more.
> 
> Just remove that function for x86, and let nr_irqs to NR_IRQS
> because We already have radix-tree and bit_map for searching desc for irq.

Won't that make the arch_probe_nr_irqs be NR_IRQS_LEGACY?

I am looking at Linus's today tree, or is this patch based on some other tree?

> 
> Notes: long before same vresion cause one of Ingo's setup udev hang...
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/kernel/apic/io_apic.c |   22 ----------------------
>  1 file changed, 22 deletions(-)
> 
> Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
> +++ linux-2.6/arch/x86/kernel/apic/io_apic.c
> @@ -3637,28 +3637,6 @@ int get_nr_irqs_gsi(void)
>  	return nr_irqs_gsi;
>  }
>  
> -#ifdef CONFIG_SPARSE_IRQ
> -int __init arch_probe_nr_irqs(void)
> -{
> -	int nr;
> -
> -	if (nr_irqs > (NR_VECTORS * nr_cpu_ids))
> -		nr_irqs = NR_VECTORS * nr_cpu_ids;
> -
> -	nr = nr_irqs_gsi + 8 * nr_cpu_ids;
> -#if defined(CONFIG_PCI_MSI) || defined(CONFIG_HT_IRQ)
> -	/*
> -	 * for MSI and HT dyn irq
> -	 */
> -	nr += nr_irqs_gsi * 16;
> -#endif
> -	if (nr < nr_irqs)
> -		nr_irqs = nr;
> -
> -	return NR_IRQS_LEGACY;
> -}
> -#endif
> -
>  static int __io_apic_set_pci_routing(struct device *dev, int irq,
>  				struct io_apic_irq_attr *irq_attr)
>  {
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] x86, sparseirq: let nr_irqs equal to NR_IRQS

[Yinghai Lu]

On Tue, Dec 28, 2010 at 9:42 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Mon, Dec 27, 2010 at 04:54:11PM -0800, Yinghai Lu wrote:
>>
>>
>> For x86_64 system:
>> when we have 128 cpus with 5 ioapics, will have nr_irqs = 3064
>>       120 + 8 * 128 + 120 * 16
>> systems could take 20 pcie,  when intel 10g are used with
>> sriov and ixgbevf, every vf will need 3 irqs, and one device
>> have 64 vf. so will need 20 * 3 * 64 = 3840.
>> some 6 ports Intel 10gb may need more.
>>
>> Just remove that function for x86, and let nr_irqs to NR_IRQS
>> because We already have radix-tree and bit_map for searching desc for irq.
>
> Won't that make the arch_probe_nr_irqs be NR_IRQS_LEGACY?

int nr_irqs = NR_IRQS;

and

int __init __weak arch_probe_nr_irqs(void)
{
        return NR_IRQS_LEGACY;
}

int __init early_irq_init(void)
{
        int i, initcnt, node = first_online_node;
        struct irq_desc *desc;

        init_irq_default_affinity();

        /* Let arch update nr_irqs and return the nr of preallocated irqs */
        initcnt = arch_probe_nr_irqs();
        printk(KERN_INFO "NR_IRQS:%d nr_irqs:%d %d\n", NR_IRQS,
nr_irqs, initcnt);


that arch_probe_nr_irqs() is return for initcnt...

should change nr_irqs in the functions...

some confusing...


>
> I am looking at Linus's today tree, or is this patch based on some other tree?
>
>>
>> Notes: long before same vresion cause one of Ingo's setup udev hang...
>>
>> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>>
>> ---
>>  arch/x86/kernel/apic/io_apic.c |   22 ----------------------
>>  1 file changed, 22 deletions(-)
>>
>> Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
>> +++ linux-2.6/arch/x86/kernel/apic/io_apic.c
>> @@ -3637,28 +3637,6 @@ int get_nr_irqs_gsi(void)
>>       return nr_irqs_gsi;
>>  }
>>
>> -#ifdef CONFIG_SPARSE_IRQ
>> -int __init arch_probe_nr_irqs(void)
>> -{
>> -     int nr;
>> -
>> -     if (nr_irqs > (NR_VECTORS * nr_cpu_ids))
>> -             nr_irqs = NR_VECTORS * nr_cpu_ids;
>> -
>> -     nr = nr_irqs_gsi + 8 * nr_cpu_ids;
>> -#if defined(CONFIG_PCI_MSI) || defined(CONFIG_HT_IRQ)
>> -     /*
>> -      * for MSI and HT dyn irq
>> -      */
>> -     nr += nr_irqs_gsi * 16;
>> -#endif
>> -     if (nr < nr_irqs)
>> -             nr_irqs = nr;
>> -
>> -     return NR_IRQS_LEGACY;
>> -}
>> -#endif
>> -
>>  static int __io_apic_set_pci_routing(struct device *dev, int irq,
>>                               struct io_apic_irq_attr *irq_attr)
>>  {
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>