[PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[yhlu]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

in phys flat mode, when using set_xxx_irq_affinity to irq balance from
one cpu to another,  _assign_irq_vector will get to increase last used
vector and get new vector. this will use up the vector if enough
set_xxx_irq_affintiy are called. and end with using same vector in
different cpu for different irq. (that is not what we want, we only
want to use same vector in different cpu for different irq when more
than 0x240 irq needed). To keep it simple, the vector should be reused
instead of getting new vector.

Also according to Eric's review, make it more generic to be used with
flat mode too.

This patch need to be applied over Eric's irq: cpu_online_map patch.


Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: Muli Ben-Yehuda <muli@il.ibm.com>
Signed-off-by: Yinghai Lu <yinghai.lu@amd.com>

[-- Attachment #2: io_apic_reuse_vector.diff --]
[-- Type: text/x-patch, Size: 2453 bytes --]

--- linux-2.6/arch/x86_64/kernel/io_apic.c	2006-10-23 18:44:05.000000000 -0700
+++ linux-2.6.xx/arch/x86_64/kernel/io_apic.c	2006-10-23 18:10:08.000000000 -0700
@@ -613,8 +613,9 @@
 	 * 0x80, because int 0x80 is hm, kind of importantish. ;)
 	 */
 	static int current_vector = FIRST_DEVICE_VECTOR, current_offset = 0;
-	int old_vector = -1;
-	int cpu;
+	int vector = -1, old_vector = -1;
+	cpumask_t domain, new_mask;
+	int cpu, new_cpu;
 
 	BUG_ON((unsigned)irq >= NR_IRQ_VECTORS);
 
@@ -624,15 +625,30 @@
 	if (irq_vector[irq] > 0)
 		old_vector = irq_vector[irq];
 	if (old_vector > 0) {
-		cpus_and(*result, irq_domain[irq], mask);
-		if (!cpus_empty(*result))
-			return old_vector;
+		/* try to reuse vector */
+		for_each_cpu_mask(cpu, mask) {
+			int can_reuse = 1;
+			domain = vector_allocation_domain(cpu);
+			cpus_and(new_mask, domain, cpu_online_map);
+			for_each_cpu_mask(new_cpu, new_mask) {
+				int old_irq;
+				old_irq = per_cpu(vector_irq, new_cpu)[old_vector];
+				if ( (old_irq != irq) && (old_irq != -1)) {
+					can_reuse = 0;
+					break;
+				}
+			}
+
+			if(!can_reuse) continue;
+
+			vector = old_vector;
+			goto found_one;	
+		}
+
 	}
 
 	for_each_cpu_mask(cpu, mask) {
-		cpumask_t domain, new_mask;
-		int new_cpu;
-		int vector, offset;
+		int offset;
 
 		domain = vector_allocation_domain(cpu);
 		cpus_and(new_mask, domain, cpu_online_map);
@@ -656,21 +672,27 @@
 		/* Found one! */
 		current_vector = vector;
 		current_offset = offset;
-		if (old_vector >= 0) {
-			cpumask_t old_mask;
-			int old_cpu;
-			cpus_and(old_mask, irq_domain[irq], cpu_online_map);
-			for_each_cpu_mask(old_cpu, old_mask)
-				per_cpu(vector_irq, old_cpu)[old_vector] = -1;
-		}
-		for_each_cpu_mask(new_cpu, new_mask)
-			per_cpu(vector_irq, new_cpu)[vector] = irq;
-		irq_vector[irq] = vector;
-		irq_domain[irq] = domain;
-		cpus_and(*result, domain, mask);
-		return vector;
+		
+		goto found_one;
 	}
+
 	return -ENOSPC;
+
+found_one:
+	if (old_vector >= 0) {
+		cpumask_t old_mask;
+		int old_cpu;
+		cpus_and(old_mask, irq_domain[irq], cpu_online_map);
+		for_each_cpu_mask(old_cpu, old_mask)
+			per_cpu(vector_irq, old_cpu)[old_vector] = -1;
+	}
+	for_each_cpu_mask(new_cpu, new_mask)
+		per_cpu(vector_irq, new_cpu)[vector] = irq;
+	irq_vector[irq] = vector;
+	irq_domain[irq] = domain;
+	cpus_and(*result, domain, mask);
+	return vector;
+
 }
 
 static int assign_irq_vector(int irq, cpumask_t mask, cpumask_t *result)

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Muli Ben-Yehuda]

On Mon, Oct 23, 2006 at 09:15:31PM -0700, yhlu wrote:
> in phys flat mode, when using set_xxx_irq_affinity to irq balance from
> one cpu to another,  _assign_irq_vector will get to increase last used
> vector and get new vector. this will use up the vector if enough
> set_xxx_irq_affintiy are called. and end with using same vector in
> different cpu for different irq. (that is not what we want, we only
> want to use same vector in different cpu for different irq when more
> than 0x240 irq needed). To keep it simple, the vector should be reused
> instead of getting new vector.
> 
> Also according to Eric's review, make it more generic to be used with
> flat mode too.
> 
> This patch need to be applied over Eric's irq: cpu_online_map patch.
> 
> 
> Cc: Eric W. Biederman <ebiederm@xmission.com>
> Cc: Muli Ben-Yehuda <muli@il.ibm.com>
> Signed-off-by: Yinghai Lu <yinghai.lu@amd.com>

Boots fine and survives a quick stress test.

Cheers,
Muli

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Andi Kleen]

On Mon, Oct 23, 2006 at 09:15:31PM -0700, yhlu wrote:
> in phys flat mode, when using set_xxx_irq_affinity to irq balance from
> one cpu to another,  _assign_irq_vector will get to increase last used
> vector and get new vector. this will use up the vector if enough
> set_xxx_irq_affintiy are called. and end with using same vector in
> different cpu for different irq. (that is not what we want, we only
> want to use same vector in different cpu for different irq when more
> than 0x240 irq needed). To keep it simple, the vector should be reused
> instead of getting new vector.
> 
> Also according to Eric's review, make it more generic to be used with
> flat mode too.

Is that still needed with Eric's latest patches? I suppose not?

-Andi

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Lu, Yinghai]

>From: Andi Kleen [mailto:ak@suse.de]
>Is that still needed with Eric's latest patches? I suppose not?

It needs Eric's

x86_64-irq-simplify-the-vector-allocator.patch
x86_64-irq-only-look-at-per_cpu-data-for-online-cpus.patch

Those two are in -mm tree now.

Otherwise it can not be applied without FAIL.

YH

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Yinghai Lu]

Andi,

I think this patch could be in your tree after Eric's tree is getting
into mainstream.

YH

On 10/24/06, Lu, Yinghai <yinghai.lu@amd.com> wrote:
> >From: Andi Kleen [mailto:ak@suse.de]
> >Is that still needed with Eric's latest patches? I suppose not?
>
> It needs Eric's
>
> x86_64-irq-simplify-the-vector-allocator.patch
> x86_64-irq-only-look-at-per_cpu-data-for-online-cpus.patch
>
> Those two are in -mm tree now.
>
> Otherwise it can not be applied without FAIL.
>
> YH
>

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

Andi,
please check the revised patch: It can be appied to current linus't tree.

in phys flat mode, when using set_xxx_irq_affinity to irq balance from
one cpu to another,  _assign_irq_vector will get to increase last used
vector and get new vector. this will use up the vector if enough
set_xxx_irq_affintiy are called. and end with using same vector in
different cpu for different irq. (that is not what we want, we only
want to use same vector in different cpu for different irq when more
than 0x240 irq needed). To keep it simple, the vector should be reused
instead of getting new vector.

Also according to Eric's review, make it more generic to be used with
flat mode too.

It also check if new domain and old domain is equal, to avoid extra operation.


Cc: Eric W. Biederman <ebiederm@xmission.com>
Cc: Muli Ben-Yehuda <muli@il.ibm.com>
Signed-off-by: Yinghai Lu <yinghai.lu@amd.com>

[-- Attachment #2: io_apic_reuse_vector_1026.diff --]
[-- Type: text/x-patch, Size: 2724 bytes --]

diff --git a/arch/x86_64/kernel/io_apic.c b/arch/x86_64/kernel/io_apic.c
index fe429e5..51535e8 100644
--- a/arch/x86_64/kernel/io_apic.c
+++ b/arch/x86_64/kernel/io_apic.c
@@ -613,8 +613,9 @@ static int __assign_irq_vector(int irq, 
 	 * 0x80, because int 0x80 is hm, kind of importantish. ;)
 	 */
 	static int current_vector = FIRST_DEVICE_VECTOR, current_offset = 0;
-	int old_vector = -1;
-	int cpu;
+	int vector = -1, old_vector = -1;
+	cpumask_t domain, new_mask;
+	int cpu, new_cpu;
 
 	BUG_ON((unsigned)irq >= NR_IRQ_VECTORS);
 
@@ -624,15 +625,38 @@ static int __assign_irq_vector(int irq, 
 	if (irq_vector[irq] > 0)
 		old_vector = irq_vector[irq];
 	if (old_vector > 0) {
-		cpus_and(*result, irq_domain[irq], mask);
-		if (!cpus_empty(*result))
-			return old_vector;
+		/* try to reuse vector */
+		for_each_cpu_mask(cpu, mask) {
+			int can_reuse = 1;
+
+			domain = vector_allocation_domain(cpu);
+			cpus_and(new_mask, domain, cpu_online_map);
+
+			if (cpus_equal(domain, irq_domain[irq])) {
+				cpus_and(*result, irq_domain[irq], mask);
+				if (!cpus_empty(*result))  
+					return old_vector;
+			}
+			
+			for_each_cpu_mask(new_cpu, new_mask) {
+				int old_irq;
+				old_irq = per_cpu(vector_irq, new_cpu)[old_vector];
+				if ( (old_irq != irq) && (old_irq != -1)) {
+					can_reuse = 0;
+					break;
+				}
+			}
+
+			if(!can_reuse) 
+				continue;
+
+			vector = old_vector;
+			goto found_one;	
+		}
 	}
 
 	for_each_cpu_mask(cpu, mask) {
-		cpumask_t domain, new_mask;
-		int new_cpu;
-		int vector, offset;
+		int offset;
 
 		domain = vector_allocation_domain(cpu);
 		cpus_and(new_mask, domain, cpu_online_map);
@@ -656,21 +680,27 @@ next:
 		/* Found one! */
 		current_vector = vector;
 		current_offset = offset;
-		if (old_vector >= 0) {
-			cpumask_t old_mask;
-			int old_cpu;
-			cpus_and(old_mask, irq_domain[irq], cpu_online_map);
-			for_each_cpu_mask(old_cpu, old_mask)
-				per_cpu(vector_irq, old_cpu)[old_vector] = -1;
-		}
-		for_each_cpu_mask(new_cpu, new_mask)
-			per_cpu(vector_irq, new_cpu)[vector] = irq;
-		irq_vector[irq] = vector;
-		irq_domain[irq] = domain;
-		cpus_and(*result, domain, mask);
-		return vector;
+		
+		goto found_one;
 	}
+
 	return -ENOSPC;
+
+found_one:
+	if (old_vector >= 0) {
+		cpumask_t old_mask;
+		int old_cpu;
+		cpus_and(old_mask, irq_domain[irq], cpu_online_map);
+		for_each_cpu_mask(old_cpu, old_mask)
+			per_cpu(vector_irq, old_cpu)[old_vector] = -1;
+	}
+	for_each_cpu_mask(new_cpu, new_mask)
+		per_cpu(vector_irq, new_cpu)[vector] = irq;
+	irq_vector[irq] = vector;
+	irq_domain[irq] = domain;
+	cpus_and(*result, domain, mask);
+	return vector;
+
 }
 
 static int assign_irq_vector(int irq, cpumask_t mask, cpumask_t *result)

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Muli Ben-Yehuda]

On Thu, Oct 26, 2006 at 02:04:57AM -0700, Yinghai Lu wrote:
> Andi,
> please check the revised patch: It can be appied to current linus't tree.
> 
> in phys flat mode, when using set_xxx_irq_affinity to irq balance from
> one cpu to another,  _assign_irq_vector will get to increase last used
> vector and get new vector. this will use up the vector if enough
> set_xxx_irq_affintiy are called. and end with using same vector in
> different cpu for different irq. (that is not what we want, we only
> want to use same vector in different cpu for different irq when more
> than 0x240 irq needed). To keep it simple, the vector should be reused
> instead of getting new vector.
> 
> Also according to Eric's review, make it more generic to be used with
> flat mode too.
> 
> It also check if new domain and old domain is equal, to avoid extra 
> operation.

Works fine for mee.

Cheers,
Muli

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Andi Kleen]

On Thu, Oct 26, 2006 at 02:04:57AM -0700, Yinghai Lu wrote:
> Andi,
> please check the revised patch: It can be appied to current linus't tree.
> 
> in phys flat mode, when using set_xxx_irq_affinity to irq balance from
> one cpu to another,  _assign_irq_vector will get to increase last used
> vector and get new vector. this will use up the vector if enough
> set_xxx_irq_affintiy are called. and end with using same vector in
> different cpu for different irq. (that is not what we want, we only
> want to use same vector in different cpu for different irq when more
> than 0x240 irq needed). To keep it simple, the vector should be reused
> instead of getting new vector.
> 
> Also according to Eric's review, make it more generic to be used with
> flat mode too.

Eric, do you ack the patch? 

-andi

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Andi Kleen]

On Mon, Oct 23, 2006 at 09:15:31PM -0700, yhlu wrote:
> in phys flat mode, when using set_xxx_irq_affinity to irq balance from
> one cpu to another,  _assign_irq_vector will get to increase last used
> vector and get new vector. this will use up the vector if enough
> set_xxx_irq_affintiy are called. and end with using same vector in
> different cpu for different irq. (that is not what we want, we only
> want to use same vector in different cpu for different irq when more
> than 0x240 irq needed). To keep it simple, the vector should be reused
> instead of getting new vector.
> 
> Also according to Eric's review, make it more generic to be used with
> flat mode too.
> 
Added thanks

> This patch need to be applied over Eric's irq: cpu_online_map patch.

Hmm, i'm not sure I got that. Which was patch was it exactly

Or can you please double check it is ok in 2.6.19rc3 +
ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/x86_64-2.6.19-rc3-061028-1.bz2

Thanks,

-Andi

Re: [PATCH] x86_64 irq: reuse vector for __assign_irq_vector

[Yinghai Lu]

On 28 Oct 2006 19:46:22 +0200, Andi Kleen <ak@muc.de> wrote:
> On Mon, Oct 23, 2006 at 09:15:31PM -0700, yhlu wrote:

> Hmm, i'm not sure I got that. Which was patch was it exactly


Eric's online cpus patch is already in the maintream.

Please use the diff in

http://lkml.org/lkml/2006/10/26/38

Thanks

YH