[RFC]degradation on IPF due to hypercall set irq

[RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir,

After moving from sharing PIC to hypercall set irq.
KB on UP VTI domain incurs > 10% degradation.

The root cause is hypercall is very expensive on IPF side
due to huge processor context.

I revert to sharing PIC in lastest Cset of IPF side,
Then We can get performance back.


I prepare to use shared IOSAPIC to deliver interrupt from 
Qemu to VTI domain.
In IPF side, PIC is not needed,
In the same time, we can assign more interrupt pins(24) to qemu.


What's your opinion?


Thanks,
Anthony

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 3:33 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> After moving from sharing PIC to hypercall set irq.
> KB on UP VTI domain incurs > 10% degradation.
> 
> The root cause is hypercall is very expensive on IPF side
> due to huge processor context.
> 
> I revert to sharing PIC in lastest Cset of IPF side,
> Then We can get performance back.

We may well have similar degradation on x86 too. The cause is lots of
unnecessary calls to the set_level hypercall (when the level hasn't actually
changed). Qemu *definitely* needs to keep shadow wire state and only notify
Xen on transitions. If the rate of hypercalls is still too high (which I
think is unlikely) we can use batching multicalls.

> I prepare to use shared IOSAPIC to deliver interrupt from
> Qemu to VTI domain.
> In IPF side, PIC is not needed,
> In the same time, we can assign more interrupt pins(24) to qemu.

I moved x86 away from this on purpose, to obtain a clean abstraction. I
don't think it's a good idea for ia64 to step backwards here.

 -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月22日 15:07:
> On 22/11/06 3:33 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
>> After moving from sharing PIC to hypercall set irq.
>> KB on UP VTI domain incurs > 10% degradation.
>> 
>> The root cause is hypercall is very expensive on IPF side
>> due to huge processor context.
>> 
>> I revert to sharing PIC in lastest Cset of IPF side,
>> Then We can get performance back.
> 
> We may well have similar degradation on x86 too. The cause is lots of
> unnecessary calls to the set_level hypercall (when the level hasn't
> actually changed). Qemu *definitely* needs to keep shadow wire state
> and only notify Xen on transitions. If the rate of hypercalls is
> still too high (which I think is unlikely) we can use batching
> multicalls. 



I have tried shadow wire state in IPF side, filtered out unnecessary set_level hypercalls,
But I can only get about 50% performance degradation back.

In IPF side, I set all interrupt edge( there is no sharing interrupt in my environment), so 1->0, 0->0 and 1->1
 is not Passed to xen by hypercall, only 0->1 is passed to xen by hypercall, then about half of set_level
hypercalls are saved. But this can only get ~50% performance degradation back.

In previous shared PIC method, it is likely that interrupt and IO finish are passed to xen by only one
hypercall xc_evtchn_notify.

But now we may need to use two hypercalls xc_evtchn_notify and set_level hypercalls,
I think this is reason of another 50% performance degradation.

Batching multicall may be a good idea,
The only question is how and when we batch xc_evtchn_notify and set_level hypercall,

When xc_evtchn_notify is called, there may be several set_level hypercall should be called,
But set_level hypercall is based on irq line level, how to "remember" several set_level hypercalls?
Maybe we need to change set_level hypercall interface.

> 
>> I prepare to use shared IOSAPIC to deliver interrupt from
>> Qemu to VTI domain.
>> In IPF side, PIC is not needed,
>> In the same time, we can assign more interrupt pins(24) to qemu.
> 
> I moved x86 away from this on purpose, to obtain a clean abstraction.
> I don't think it's a good idea for ia64 to step backwards here.

Actually I don't want to do this, if there are better solution to pull back performance.

-- Anthony

> 
>  -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 7:55 am, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> Have an array of set_level hypercall structures, and an array of multicall
> structures. Fill them in at the point we currently do the hypercall. Flush
> when:
>  A) the array is already full; or
>  B) when qemu passes through its event loop.
> 
> Make the arrays 16 entries large, for example, will be plenty.
> 
> Use the same mechanism for the notification (i.e., add to the multicall
> array, to be flushed by qemu's main loop).

To clarify, by event/main loop I mean: Flush just before qemu blocks
(otherwise multicall can be held for unbounded time, unless we set a
batching timeout which hopefully we can avoid needing to do).

 -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月22日 15:59:
> On 22/11/06 7:55 am, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> Have an array of set_level hypercall structures, and an array of
>> multicall structures. Fill them in at the point we currently do the
>>  hypercall. Flush when: A) the array is already full; or
>>  B) when qemu passes through its event loop.
>> 
>> Make the arrays 16 entries large, for example, will be plenty.
>> 
>> Use the same mechanism for the notification (i.e., add to the
>> multicall array, to be flushed by qemu's main loop).
> 
> To clarify, by event/main loop I mean: Flush just before qemu blocks
> (otherwise multicall can be held for unbounded time, unless we set a
> batching timeout which hopefully we can avoid needing to do).

Seems good,

Is it possible that we add some file descriptors for some interrupts in qemu block (select)?
Then if there are any interrupts, qemu will be woke up.
In this case, we can only call multi-call just after qemu block.

--Anthony.




> 
>  -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 08:16, "Xu, Anthony" <anthony.xu@intel.com> wrote:

>> To clarify, by event/main loop I mean: Flush just before qemu blocks
>> (otherwise multicall can be held for unbounded time, unless we set a
>> batching timeout which hopefully we can avoid needing to do).
> 
> Seems good,
> 
> Is it possible that we add some file descriptors for some interrupts in qemu
> block (select)?
> Then if there are any interrupts, qemu will be woke up.
> In this case, we can only call multi-call just after qemu block.

I'm not sure what you mean.

 -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月22日 17:22:
> On 22/11/06 08:16, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
>>> To clarify, by event/main loop I mean: Flush just before qemu blocks
>>> (otherwise multicall can be held for unbounded time, unless we set a
>>> batching timeout which hopefully we can avoid needing to do).

Why otherwise multicall can be held for unbounded time?

--Anthony


> 
>  -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 09:24, "Xu, Anthony" <anthony.xu@intel.com> wrote:

>>>> To clarify, by event/main loop I mean: Flush just before qemu blocks
>>>> (otherwise multicall can be held for unbounded time, unless we set a
>>>> batching timeout which hopefully we can avoid needing to do).
> 
> Why otherwise multicall can be held for unbounded time?

Qemu only wakes up for device-model accesses. We don't know when the next of
those will be. So we should flush multicalls before the potentially blocking
select().

 -- Keir

RE: [Xen-devel][RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月22日 17:26:
> On 22/11/06 09:24, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
>>>>> To clarify, by event/main loop I mean: Flush just before qemu
>>>>> blocks (otherwise multicall can be held for unbounded time,
>>>>> unless we set a batching timeout which hopefully we can avoid
>>>>> needing to do). 
>> 
>> Why otherwise multicall can be held for unbounded time?
> 
> Qemu only wakes up for device-model accesses. We don't know when the
> next of those will be. So we should flush multicalls before the
> potentially blocking select().

There are two threads, one is qemu thread, the other is IDE DMA thread,
In IDE DMA thread, when it finishing DMA opereration, it will set irq, but it
doesn't  try to wakeup qemu thread. So if qemu thread is sleeping at the same time,
this interrupt may be delivered until qemu thread wakes up, the time may be
10 msec.
So we need a mechanism for IDE DMA thread to wake up Qemu thread.

What's your opinion?

Thanks,
Anthony




> 
>  -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 09:38, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> There are two threads, one is qemu thread, the other is IDE DMA thread,
> In IDE DMA thread, when it finishing DMA opereration, it will set irq, but it
> doesn't  try to wakeup qemu thread. So if qemu thread is sleeping at the same
> time,
> this interrupt may be delivered until qemu thread wakes up, the time may be
> 10 msec.
> So we need a mechanism for IDE DMA thread to wake up Qemu thread.
> 
> What's your opinion?

Did the IDE code really need to made multithreaded? I suppose it's a better
model for the stub domain plans...  Anyway, it's a pain here because it will
require the shadow wire bitmap to be updated with atomic accesses and the
multicall state to be per-thread or to be protected with a mutex. Each
thread should flush multicall state before it blocks.

 -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com> wrote:

>> because it will require the shadow wire bitmap to be updated with
>> atomic accesses and the multicall state to be per-thread or to be
>> protected with a mutex. Each thread should flush multicall state
>> before it blocks.
> 
> I prefer atomic access, we used it in shared PIC.
> If each thread flush multicall seperately,
> There are some extra hypercalls.

Since the threads run independently there seems little choice but for each
to be able to flush. If the IDE DMA support had been properly integrated
into the qemu select() event loop this would not be an issue.

 -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月22日 18:28:
> On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
>> I prefer atomic access, we used it in shared PIC.
>> If each thread flush multicall seperately, There are some extra
>> hypercalls.
> 
> Since the threads run independently there seems little choice but for
> each to be able to flush. If the IDE DMA support had been properly
> integrated into the qemu select() event loop this would not be an
> issue.


Agree, we can use pipe to integrate IDE DMA support into select.
But we still need to use atomic access to shawdow line.

--Anthony
   
> 
>  -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

[-- Attachment #1: Type: text/plain, Size: 2708 bytes --]

Keir,

This patch is for comments, and it is based on IPF, it may not apply to IA32 side.

This patch delivers interrupt and IO finish in the same hypercall xc_hvm_set_irq_and_wakeup_cpu,
This eliminate all unnecessary hypercalls.
In the meantime, I add a mechanism for IDE DMA thread to wakeup qemu main block(select) to
Deliver IDE DMA interrupt.

Yes all this can improve KB performance greatly,
But KB is still incurring 2%~3% degradation compared to shared PIC irq line.

I guest the reason as following,

In shared PIC irq line method,
IDE DMA thread doesn't wake up Qemu thread, after it has set irq line,
Select is only back when there is IO operation or other interrupt like keyborad, mouse.
This is not likely to delay the IDE interrupt deliverring, because every time xen returns to guest OS,
It will sync share irq line, and IDE interrupt can be delivered without dom0 waking up hvm domain.

There is no xc_evtchn_notify called due to IDE DMA interrupt deliver.


But In currently hypercall method,
There are many hypercalls called due to IDE DMA interrupt deliver.

So there are still 2%~3% performance degradation.


What's your opinion?


Thanks,
Anthony




Xu, Anthony write on 2006年11月24日 10:00:
> Keir Fraser write on 2006年11月22日 18:28:
>> On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com> wrote:
>> 
>> Since the threads run independently there seems little choice but for
>> each to be able to flush. If the IDE DMA support had been properly
>> integrated into the qemu select() event loop this would not be an
>> issue.
> 
> Hi keir,
> 
> I'm trying to use multi hypercall to resolve this issue.
> But I found it is difficult to use multi hypercall in qemu.
> Because a hypercall have different argument in qemu, libxc and dom0
> kernel. 
> 
> For example,
> In qemu,
> xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
> In libxc
> ioctl(xce_handle, IOCTL_EVTCHN_NOTIFY, &notify);
> In dom0 kernel
> HYPERVISOR_event_channel_op(EVTCHNOP_send, &send);
> 
> If we want to use multi hypercall in qemu, we need to get the
> interface information 
> of dom0 kernel. We may need to add several more hypercall.
> 
> I propose to change xc_hvm_set_irq_level functionality to deliver
> interrupt and wakeup cpu at the same time.
> For example,
>  xc_hvm_set_irq_and_wakeup_cpu(int xc_handle, domid_t dom, int irq,
> int level, int cpu) This hypercall will deliver interrupt irq and
> wake up cpu. 
> 
> What's your opinion?
> 
> 
> Thanks,
> Anthony
> 
> 
> 
> 
> 
> 
> 
>> 
>>  -- Keir
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


[-- Attachment #2: qemu_interrupt_deliver200611241336.patch --]
[-- Type: application/octet-stream, Size: 10087 bytes --]

diff -r ac5330d4945a Makefile
--- a/Makefile	Wed Nov 15 12:15:34 2006 -0700
+++ b/Makefile	Thu Nov 23 08:25:00 2006 +0800
@@ -40,7 +40,7 @@ test:
 # build and install everything into local dist directory
 .PHONY: dist
 dist: DESTDIR=$(DISTDIR)/install
-dist: dist-xen dist-kernels dist-tools dist-docs
+dist: dist-xen dist-tools 
 	$(INSTALL_DIR) $(DISTDIR)/check
 	$(INSTALL_DATA) ./COPYING $(DISTDIR)
 	$(INSTALL_DATA) ./README $(DISTDIR)
diff -r ac5330d4945a tools/ioemu/hw/ide.c
--- a/tools/ioemu/hw/ide.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/ioemu/hw/ide.c	Fri Nov 24 02:21:28 2006 +0800
@@ -397,9 +397,18 @@ typedef struct PCIIDEState {
 #ifdef DMA_MULTI_THREAD
 
 static int file_pipes[2];
+static int ide_dma_pipes[2];
 
 static void ide_dma_loop(BMDMAState *bm);
 static void dma_thread_loop(BMDMAState *bm);
+static inline void ide_set_irq(IDEState *s);
+
+static void ide_dma_set_irq(void *opaque)
+{
+    IDEState *s; 
+    read(ide_dma_pipes[0], &s, sizeof(s));
+    ide_set_irq(s);
+}
 
 static void *dma_thread_func(void* opaque)
 {
@@ -2181,7 +2190,8 @@ static void ide_dma_finish(BMDMAState *b
     bm->dma_cb = NULL;
     bm->ide_if = NULL;
 #ifdef DMA_MULTI_THREAD
-    ide_set_irq(s);
+    write(ide_dma_pipes[1],&s,sizeof(s));
+//    ide_set_irq(s);
 #endif /* DMA_MULTI_THREAD */
 }
 
@@ -2462,6 +2472,7 @@ void pci_cmd646_ide_init(PCIBus *bus, Bl
 #endif /* DMA_MULTI_THREAD */
 }
 
+
 /* hd_table must contain 4 block drivers */
 /* NOTE: for the PIIX3, the IRQs and IOports are hardcoded */
 void pci_piix3_ide_init(PCIBus *bus, BlockDriverState **hd_table, int devfn)
@@ -2495,7 +2506,12 @@ void pci_piix3_ide_init(PCIBus *bus, Blo
               pic_set_irq_new, isa_pic, 15);
     ide_init_ioport(&d->ide_if[0], 0x1f0, 0x3f6);
     ide_init_ioport(&d->ide_if[2], 0x170, 0x376);
-#ifdef DMA_MULTI_THREAD    
+#ifdef DMA_MULTI_THREAD
+    if (pipe(ide_dma_pipes) != 0) {
+         fprintf(stderr, "create ide dma pipe failed\n");
+         exit(1);
+    }
+    qemu_set_fd_handler(ide_dma_pipes[0],ide_dma_set_irq,NULL,NULL);   
     dma_create_thread();
 #endif //DMA_MULTI_THREAD    
 }
diff -r ac5330d4945a tools/ioemu/target-i386-dm/cpu.h
--- a/tools/ioemu/target-i386-dm/cpu.h	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/ioemu/target-i386-dm/cpu.h	Thu Nov 23 06:36:36 2006 +0800
@@ -51,10 +51,17 @@ typedef float64 CPU86_LDouble;
 /* Empty for now */
 typedef struct CPUX86State {
     uint32_t a20_mask;
+//these two fields must be continued for atomic access
+    uint16_t irr;
+
+    uint16_t level;
 
     int interrupt_request;
 
     CPU_COMMON
+
+    int send_event;
+
 } CPUX86State;
 
 CPUX86State *cpu_x86_init(void);
diff -r ac5330d4945a tools/ioemu/target-i386-dm/helper2.c
--- a/tools/ioemu/target-i386-dm/helper2.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/ioemu/target-i386-dm/helper2.c	Fri Nov 24 05:15:04 2006 +0800
@@ -49,6 +49,7 @@
 
 #include <xenctrl.h>
 #include <xen/hvm/ioreq.h>
+#include <asm/system.h>
 
 #include "cpu.h"
 #include "exec-all.h"
@@ -517,7 +518,8 @@ void cpu_handle_ioreq(void *opaque)
 
         wmb(); /* Update ioreq contents /then/ update state. */
         req->state = STATE_IORESP_READY;
-        xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
+        env->send_event = 1;
+//        xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
     }
 }
 
@@ -525,6 +527,7 @@ int main_loop(void)
 {
     extern int vm_running;
     extern int shutdown_requested;
+    int deliver_vcpu = -1;
     CPUState *env = cpu_single_env;
     int evtchn_fd = xc_evtchn_fd(xce_handle);
 
@@ -533,6 +536,10 @@ int main_loop(void)
     qemu_mod_timer(buffered_io_timer, qemu_get_clock(rt_clock));
 
     qemu_set_fd_handler(evtchn_fd, cpu_handle_ioreq, NULL, env);
+
+    env->send_event = 0;
+    env->irr = 0;
+    env->level = 0;
 
     while (1) {
         if (vm_running) {
@@ -546,7 +553,25 @@ int main_loop(void)
 
         /* Wait up to 10 msec. */
         main_loop_wait(10);
-    }
+        if (env->irr){
+            if(env->send_event){
+                env->send_event = 0;
+                deliver_vcpu = send_vcpu;
+            }else{
+                deliver_vcpu = -1;
+            }
+            xc_hvm_set_irq_and_wakeup_cpu(xc_handle, domid, env->irr, env->level, deliver_vcpu);
+            env->irr = 0;
+            env->level = 0;
+        }
+        if(env->send_event){
+            env->send_event = 0;
+            xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
+        }
+
+    }
+
+
     destroy_hvm_domain();
     return 0;
 }
diff -r ac5330d4945a tools/ioemu/target-i386-dm/i8259-dm.c
--- a/tools/ioemu/target-i386-dm/i8259-dm.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/ioemu/target-i386-dm/i8259-dm.c	Fri Nov 24 02:42:25 2006 +0800
@@ -29,11 +29,41 @@
 #include "cpu-all.h"
 
 struct PicState2 {
+    uint16_t last_irr;
+    uint16_t elcr; /* PIIX edge/trigger selection*/
 };
 
 void pic_set_irq_new(void *opaque, int irq, int level)
 {
-    xc_hvm_set_irq_level(xc_handle, domid, irq, level);
+    int mask = 1 << irq;
+    CPUState *env = cpu_single_env;
+    PicState2 *s = opaque;
+    if (s->elcr & mask) {
+        /* level triggered */
+        if (level) {
+            if((s->last_irr&mask)==0){
+                env->irr |= mask;
+                env->level |= mask;
+                s->last_irr |= mask;
+            }
+        } else {
+            if(s->last_irr&mask){
+                env->irr |= mask;
+                env->level &= ~mask;
+                s->last_irr &= ~mask;
+            }
+        }
+    } else {
+       /* edge triggered */
+        if (level) {
+            if((s->last_irr&mask)==0){
+                env->irr |= mask;
+                env->level |= mask;
+                s->last_irr |= mask;
+            }
+        } else 
+            s->last_irr &= ~mask;
+    }
 }
 
 /* obsolete function */
@@ -58,6 +88,9 @@ PicState2 *pic_init(IRQRequestFunc *irq_
     s = qemu_mallocz(sizeof(PicState2));
     if (!s)
         return NULL;
+    s->last_irr=0;
+    //pci interrupt is level triger
+    s->elcr=0xc00;
     return s;
 }
 
diff -r ac5330d4945a tools/ioemu/vl.c
--- a/tools/ioemu/vl.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/ioemu/vl.c	Thu Nov 23 05:08:24 2006 +0800
@@ -5146,6 +5146,8 @@ void main_loop_wait(int timeout)
             }
         }
     }
+
+
 #if defined(CONFIG_SLIRP)
     if (slirp_inited) {
         if (ret < 0) {
diff -r ac5330d4945a tools/libxc/xc_misc.c
--- a/tools/libxc/xc_misc.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/libxc/xc_misc.c	Thu Nov 23 06:41:34 2006 +0800
@@ -90,7 +90,7 @@ int xc_perfc_control(int xc_handle,
     return rc;
 }
 
-int xc_hvm_set_irq_level(int xc_handle, domid_t dom, int irq, int level)
+int xc_hvm_set_irq_and_wakeup_cpu(int xc_handle, domid_t dom, int irq, int level, int cpu)
 {
     DECLARE_HYPERCALL;
     struct xen_hvm_set_irq_level arg;
@@ -103,6 +103,7 @@ int xc_hvm_set_irq_level(int xc_handle, 
     arg.domid = dom;
     arg.irq   = irq;
     arg.level = level;
+    arg.cpu   = cpu;
 
     if ( mlock(&arg, sizeof(arg)) != 0 )
     {
diff -r ac5330d4945a tools/libxc/xenctrl.h
--- a/tools/libxc/xenctrl.h	Wed Nov 15 12:15:34 2006 -0700
+++ b/tools/libxc/xenctrl.h	Thu Nov 23 07:03:26 2006 +0800
@@ -666,6 +666,6 @@ evtchn_port_t xc_evtchn_pending(int xce_
  */
 int xc_evtchn_unmask(int xce_handle, evtchn_port_t port);
 
-int xc_hvm_set_irq_level(int xce_handle, domid_t dom, int irq, int level);
+int xc_hvm_set_irq_and_wakeup_cpu(int xce_handle, domid_t dom, int irq, int level, int cpu);
 
 #endif
diff -r ac5330d4945a xen/arch/ia64/vmx/vlsapic.c
--- a/xen/arch/ia64/vmx/vlsapic.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/xen/arch/ia64/vmx/vlsapic.c	Fri Nov 24 05:25:21 2006 +0800
@@ -670,11 +670,18 @@ void vmx_vexirq(VCPU *vcpu)
 
 void vmx_vioapic_set_irq(struct domain *d, int irq, int level)
 {
-    unsigned long flags;
-
-    spin_lock_irqsave(&d->arch.arch_vmx.virq_assist_lock, flags);
-    vioapic_set_irq(d, irq, level);
-    spin_unlock_irqrestore(&d->arch.arch_vmx.virq_assist_lock, flags);
+//    unsigned long flags;
+    int i;
+//    spin_lock_irqsave(&d->arch.arch_vmx.virq_assist_lock, flags);
+    for(i=0;i<16;i++){
+        if(irq==0) break;
+        if(irq&(1<<i)){
+            irq &= ~(1<<i);
+            vioapic_set_irq(d, i, (level&(1<<i)));
+        }
+    }
+
+//    spin_unlock_irqrestore(&d->arch.arch_vmx.virq_assist_lock, flags);
 }
 
 int vmx_vlapic_set_irq(VCPU *v, uint8_t vec, uint8_t trig)
diff -r ac5330d4945a xen/arch/ia64/vmx/vmx_hypercall.c
--- a/xen/arch/ia64/vmx/vmx_hypercall.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/xen/arch/ia64/vmx/vmx_hypercall.c	Thu Nov 23 10:05:24 2006 +0800
@@ -96,6 +96,11 @@ do_hvm_op(unsigned long op, XEN_GUEST_HA
 
         rc = -EINVAL;
         if (is_hvm_domain(d)) {
+            if(op.cpu!=-1){
+                if ( test_and_clear_bit(_VCPUF_blocked_in_xen,
+                            &d->vcpu[op.cpu]->vcpu_flags) )
+                    vcpu_wake(d->vcpu[op.cpu]);
+            }            
             vmx_vioapic_set_irq(d, op.irq, op.level);
             rc = 0;
         }
diff -r ac5330d4945a xen/arch/x86/hvm/vioapic.c
--- a/xen/arch/x86/hvm/vioapic.c	Wed Nov 15 12:15:34 2006 -0700
+++ b/xen/arch/x86/hvm/vioapic.c	Thu Nov 23 06:43:56 2006 +0800
@@ -541,6 +541,7 @@ void vioapic_update_EOI(struct domain *d
                  redir_num, vector);
         return;
     }
+    service_ioapic(vioapic);
 }
 
 void vioapic_init(struct domain *d)
diff -r ac5330d4945a xen/include/public/hvm/hvm_op.h
--- a/xen/include/public/hvm/hvm_op.h	Wed Nov 15 12:15:34 2006 -0700
+++ b/xen/include/public/hvm/hvm_op.h	Thu Nov 23 06:57:31 2006 +0800
@@ -18,6 +18,7 @@ struct xen_hvm_set_irq_level {
     domid_t  domid;    /* Domain to be updated.          */
     uint16_t level;    /* New level of the IRQ (0 or 1). */
     uint32_t irq;      /* IRQ to be updated.             */
+    int cpu;      /* cpu to be waken up.            */
 };
 typedef struct xen_hvm_set_irq_level xen_hvm_set_irq_level_t;
 DEFINE_XEN_GUEST_HANDLE(xen_hvm_set_irq_level_t);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 24/11/06 07:08, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> This patch is for comments, and it is based on IPF, it may not apply to IA32
> side.
> 
> This patch delivers interrupt and IO finish in the same hypercall
> xc_hvm_set_irq_and_wakeup_cpu,
> This eliminate all unnecessary hypercalls.
> In the meantime, I add a mechanism for IDE DMA thread to wakeup qemu main
> block(select) to
> Deliver IDE DMA interrupt.

Firstly, this patch is not against unstable tip.

Secondly, we should make multicalls work rather than kludge a
set_irq_and_notify_io_done type of hypercall. Applications are free to
include any of the Xen public headers. We really just need a xc_multicall()
API function.

Thirdly, either we should keep the independent IDE-DMA thread or it should
be entirely incorporated into the main qemu thread. Are pipe writes much
faster than just doing a hypercall? If much slower, why is that? Could you
work out a way of generically making IPC hypercalls faster (particularly
from privileged user space -- could you trap straight to the hypervisor from
user space rather than bouncing through guest kernel?).

 -- Keir

Re: [RFC]degradation on IPF due to hypercall set irq

[Keir Fraser]

On 24/11/06 14:45, "Xu, Anthony" <anthony.xu@intel.com> wrote:

> Until now, I can't see hypercall is faster than share PIC line.
> 
> Can you enlighten me why we use hypercall?

I didn't change it for speed but because it's cleaner. I believe it can be
made no slower, with a bit of effort. x86 certainly won't be reverting to
the shared state.

 -- Keir

RE: [RFC]degradation on IPF due to hypercall set irq

[Xu, Anthony]

Keir Fraser write on 2006年11月24日 23:29:
> On 24/11/06 14:45, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> 
>> Until now, I can't see hypercall is faster than share PIC line.
>> 
>> Can you enlighten me why we use hypercall?
> 
> I didn't change it for speed but because it's cleaner. I believe it
> can be made no slower, with a bit of effort. x86 certainly won't be
> reverting to the shared state.  

Can't share PIC line be done in a clean way?
I don't think so.
I think which one get better performance wins.

--Anthony


> 
>  -- Keir