Patch: implementing least priority interrupt routing

Patch: implementing least priority interrupt routing

[Juergen Gross]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Hi,

the attached patch implements interrupt routing to least priority processor
for HVM domains.
Instead of round robin the vcpu with the lowest processor priority is selected
for the interrupt. If multiple vcpus share the same low priority interrupts
are distributed between those round robin.
Tested with BS2000 domain (3 vcpus, idle processors are now preferred
interrupt target).

Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html

[-- Attachment #2: lpp_int.patch --]
[-- Type: text/x-patch, Size: 2114 bytes --]

Implementing interrupt routing to least priority processor.
Instead of round robin the vcpu with the lowest processor
priority is selected for the interrupt. If multiple vcpus
share the same low priority interrupts are distributed between
those round robin.

Signed-off-by: juergen.gross@fujitsu-siemens.com

# HG changeset patch
# User juergen.gross@fujitsu-siemens.com
# Date 1227001650 -3600
# Node ID 370b052ca213bb03f234c102d72d774df6a5aa19
# Parent  7640fba355fac233773d183077d7d01c8fd591f3
implementing least priority interrupt routing

diff -r 7640fba355fa -r 370b052ca213 xen/arch/x86/hvm/vlapic.c
--- a/xen/arch/x86/hvm/vlapic.c	Tue Oct 28 09:47:35 2008 +0100
+++ b/xen/arch/x86/hvm/vlapic.c	Tue Nov 18 10:47:30 2008 +0100
@@ -380,25 +380,35 @@
 struct vlapic *apic_round_robin(
     struct domain *d, uint8_t vector, uint32_t bitmap)
 {
-    int next, old;
-    struct vlapic *target = NULL;
+    int old;
+    uint32_t ppr, ppr_min;
+    struct vlapic *target;
+    struct vlapic *target_min = NULL;
+    struct vcpu *v;
 
-    old = next = d->arch.hvm_domain.irq.round_robin_prev_vcpu;
+    old = d->arch.hvm_domain.irq.round_robin_prev_vcpu;
+    ppr_min = 256;
+    v = d->vcpu[old];
 
     do {
-        if ( ++next == MAX_VIRT_CPUS ) 
-            next = 0;
-        if ( (d->vcpu[next] == NULL) || !test_bit(next, &bitmap) )
+        v = v->next_in_list;
+	if ( v == NULL )
+	    v = d->vcpu[0];
+        if ( !test_bit(v->vcpu_id, &bitmap) )
             continue;
-        target = vcpu_vlapic(d->vcpu[next]);
-        if ( vlapic_enabled(target) )
-            break;
-        target = NULL;
-    } while ( next != old );
+        target = vcpu_vlapic(v);
+        if ( !vlapic_enabled(target) )
+            continue;
+        ppr = vlapic_get_ppr(target);
+	if ( ppr < ppr_min )
+	{
+	    target_min = target;
+	    ppr_min = ppr;
+	    d->arch.hvm_domain.irq.round_robin_prev_vcpu = v->vcpu_id;
+	}
+    } while ( v->vcpu_id != old );
 
-    d->arch.hvm_domain.irq.round_robin_prev_vcpu = next;
-
-    return target;
+    return target_min;
 }
 
 void vlapic_EOI_set(struct vlapic *vlapic)

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Patch: implementing least priority interrupt routing

[Keir Fraser]

On 18/11/08 09:59, "Juergen Gross" <juergen.gross@fujitsu-siemens.com>
wrote:

> the attached patch implements interrupt routing to least priority processor
> for HVM domains.
> Instead of round robin the vcpu with the lowest processor priority is selected
> for the interrupt. If multiple vcpus share the same low priority interrupts
> are distributed between those round robin.
> Tested with BS2000 domain (3 vcpus, idle processors are now preferred
> interrupt target).

Where idle means 'not processing an interrupt'. Which ought to be by far the
most common case even for a non-idle CPU. Does this really improve load
balancing all that much? Does BS2000 spend lots of time in IRQ context?

My fear is that extra complexity here slows down dest_lowprio for all OSes
(and it's used by a lot of OSes) for every ExtInt delivered.

 -- Keir

Re: Patch: implementing least priority interrupt routing

[Keir Fraser]

On 18/11/08 10:10, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> Where idle means 'not processing an interrupt'. Which ought to be by far the
> most common case even for a non-idle CPU. Does this really improve load
> balancing all that much? Does BS2000 spend lots of time in IRQ context?
> 
> My fear is that extra complexity here slows down dest_lowprio for all OSes
> (and it's used by a lot of OSes) for every ExtInt delivered.

That fear is probably unfounded actually, given we scan a vcpu's IRR bitmap
on *every* vmentry currently. Still it would be nice to know the motivation
behind this patch (beyond 'it's nice to behave like native hardware'). We
might still find a cheaper method with similar or better benefit (e.g.,
check vcpu_runnable() to find idle VCPUs is cheaper and may be more
accurate).

 -- Keir

Re: Patch: implementing least priority interrupt routing

[Juergen Gross]

Keir Fraser wrote:
> On 18/11/08 10:10, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
>> Where idle means 'not processing an interrupt'. Which ought to be by far the
>> most common case even for a non-idle CPU. Does this really improve load
>> balancing all that much? Does BS2000 spend lots of time in IRQ context?
>>
>> My fear is that extra complexity here slows down dest_lowprio for all OSes
>> (and it's used by a lot of OSes) for every ExtInt delivered.
> 
> That fear is probably unfounded actually, given we scan a vcpu's IRR bitmap
> on *every* vmentry currently. Still it would be nice to know the motivation
> behind this patch (beyond 'it's nice to behave like native hardware'). We
> might still find a cheaper method with similar or better benefit (e.g.,
> check vcpu_runnable() to find idle VCPUs is cheaper and may be more
> accurate).

We are using the lower 4 bits of the task priority register in the LAPIC to
control the interrupt routing. So "idle" really means idle.
We have other priorities as well. In our system we support /390 user
applications via a JIT-compiler which has to ensure /390 instruction
semantics even in case of interrupts (this means interrupts have to be
delayed until a /390 instruction boundary has been reached). This results
in a rather high interrupt latency when a /390 application is just running
on the processor. If however the processor is just executing BS2000 system
code this restriction no longer applies. So we are using a higher task
priority when executing /390 code resulting in less interruptions and thus
better performance.
I hope things are clearer now.

Juergen

-- 
Juergen Gross                             Principal Developer
IP SW OS6                      Telephone: +49 (0) 89 636 47950
Fujitsu Siemens Computers         e-mail: juergen.gross@fujitsu-siemens.com
Otto-Hahn-Ring 6                Internet: www.fujitsu-siemens.com
D-81739 Muenchen         Company details: www.fujitsu-siemens.com/imprint.html