From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:58377) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TDxGQ-0004sk-Jw for qemu-devel@nongnu.org; Tue, 18 Sep 2012 08:50:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TDxGJ-0006py-IP for qemu-devel@nongnu.org; Tue, 18 Sep 2012 08:50:54 -0400 Received: from mail-lpp01m010-f45.google.com ([209.85.215.45]:55643) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TDxGI-0006om-VE for qemu-devel@nongnu.org; Tue, 18 Sep 2012 08:50:47 -0400 Received: by lagz14 with SMTP id z14so4703016lag.4 for ; Tue, 18 Sep 2012 05:50:45 -0700 (PDT) MIME-Version: 1.0 Date: Tue, 18 Sep 2012 20:50:45 +0800 Message-ID: From: GaoYi Content-Type: multipart/alternative; boundary=f46d04088c85af906d04c9f9565a Subject: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: qemu-devel@nongnu.org --f46d04088c85af906d04c9f9565a Content-Type: text/plain; charset=ISO-8859-1 Hi Jan, I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. I wonder whether this mechanism will be incorporated in KVM someday. Thanks, Yi --f46d04088c85af906d04c9f9565a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Jan,

=A0=A0 I have followed a previous thread about ELI proposed = by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html.
=A0=A0 I wonder whether this mechanism will be incorporated in KVM someday.=
=A0=A0 Thanks,

Yi
=A0=A0
--f46d04088c85af906d04c9f9565a-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:51184) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TDxXn-00032S-OZ for qemu-devel@nongnu.org; Tue, 18 Sep 2012 09:09:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TDxXf-0006OT-OU for qemu-devel@nongnu.org; Tue, 18 Sep 2012 09:08:51 -0400 Received: from david.siemens.de ([192.35.17.14]:22178) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TDxXf-0006OC-F2 for qemu-devel@nongnu.org; Tue, 18 Sep 2012 09:08:43 -0400 Message-ID: <50587258.9090303@siemens.com> Date: Tue, 18 Sep 2012 15:08:40 +0200 From: Jan Kiszka MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: GaoYi Cc: "qemu-devel@nongnu.org" On 2012-09-18 14:50, GaoYi wrote: > Hi Jan, > > I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. > I wonder whether this mechanism will be incorporated in KVM someday. Likely not. Both Intel and AMD will soon ship hardware that obsoletes this invasive and imperfect software solution, see also [1]. Jan [1] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/97715 -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:32836) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEKze-0002Rh-Al for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:11:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEKza-0000uo-09 for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:11:10 -0400 Received: from goliath.siemens.de ([192.35.17.28]:27409) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEKzZ-0000tt-MX for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:11:05 -0400 Message-ID: <5059D26E.2050808@siemens.com> Date: Wed, 19 Sep 2012 16:10:54 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> In-Reply-To: <20120919133720.GB22659@snow> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Muli Ben-Yehuda Cc: Nadav Har'el , GaoYi , "qemu-devel@nongnu.org" , Avi Kivity , Nadav Amit , Abel Gordon [putting Avi on CC as the final decision maker] On 2012-09-19 15:37, Muli Ben-Yehuda wrote: > On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote: >> On 2012-09-18 14:50, GaoYi wrote: >>> Hi Jan, >>> >>> I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. >>> I wonder whether this mechanism will be incorporated in KVM someday. >> >> Likely not. Both Intel and AMD will soon ship hardware that >> obsoletes this invasive and imperfect software solution, see also >> [1]. > > Hi Jan, > > I think this decision may be short-sighted. First, it remains to be > seen how good the upcoming hardware support for interrupt > virtualization will be once it ships. Second, history has been pretty > consistent in that it takes several years for hardware to catch up to > software, and even when it does, software sometimes still wins. (There > are workloads where shadow page tables still outperform EPT/NPT, to > give a concrete example). Although ELI works today, I'd be happy to > hear what parts of it you find "invasive" and "imperfect" (what > software ever is perfect?) It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. Back then in our discussion, it was unclear if and when real hardware will actually show up. Now the spec is already there, hardware should follow within months (typically). I'm not aware of limitations regarding zero-exit virtualization so far. But please share any insights you collected from studying the specs or recent patches! Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:56511) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEL9U-0007IF-Sd for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:21:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEL9L-0007BJ-A0 for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:21:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:13768) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEL9L-0007B9-1V for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:21:11 -0400 Message-ID: <5059D4C8.7080904@redhat.com> Date: Wed, 19 Sep 2012 17:20:56 +0300 From: Avi Kivity MIME-Version: 1.0 References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> In-Reply-To: <5059D26E.2050808@siemens.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Nadav Har'el , GaoYi , "qemu-devel@nongnu.org" , Muli Ben-Yehuda , Nadav Amit , Abel Gordon On 09/19/2012 05:10 PM, Jan Kiszka wrote: >> Although ELI works today, I'd be happy to >> hear what parts of it you find "invasive" and "imperfect" (what >> software ever is perfect?) > > It's imperfect as you need to dedicate a core to pure guest-mode load > and cannot run userspace on that core (cannot walk through > userspace-based device models e.g.). And it requires that magic bar to > map the shadow IDT into the guest (hmm, I think Hitachi avoided this). > > It's invasive as it has to change Linux to maintain those isolated slave > CPUs. That is, of course, based on code that was published by Hitachi. > Yours may differ but will still have to solve the same problems. > > Back then in our discussion, it was unclear if and when real hardware > will actually show up. Now the spec is already there, hardware should > follow within months (typically). I'm not aware of limitations regarding > zero-exit virtualization so far. But please share any insights you > collected from studying the specs or recent patches! This matches my thoughts on the matter exactly. I would add that ELI adds a maintenance burden of supporting rarely used code, as ELI can hardly be called mainstream. On the other hand, the new hardware support will be used by every guest running on new hardware, for IPIs, virtual interrupts, and assigned device interrupts. -- error compiling committee.c: too many arguments to function From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:36028) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TELY4-0001jO-IZ for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:46:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TELXy-0005pU-M0 for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:46:44 -0400 Received: from david.siemens.de ([192.35.17.14]:23164) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TELXy-0005on-Bj for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:46:38 -0400 Message-ID: <5059DAC4.9020907@siemens.com> Date: Wed, 19 Sep 2012 16:46:28 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Abel Gordon Cc: Nadav Har'El , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit On 2012-09-19 16:43, Abel Gordon wrote: > >> It's imperfect as you need to dedicate a core to pure guest-mode load >> and cannot run userspace on that core (cannot walk through >> userspace-based device models e.g.). > > That's not correct. > For the evaluation, we dedicated a core for each guest to maximize the > performance but this > is not a requirement. You can over-commit cores and share them across > multiple VMs. In this case, > you will be sharing the cycles among all the VMs and the performance per VM > will be degraded. > In addition, if you share a core, an interrupt for a guest may be raised > while the corresponding > VCPU is not running. Depending on the workload, most of the interrupts will > be generated while > the VCPU runs or not. > >> and cannot run userspace on that core (cannot walk through >> userspace-based device models e.g.). > > That's not correct. > We can run any thread (including user-space threads) on the same core we > run the VMs (VCPU threads). > In fact, we did that for the ELI evaluation: we shared a single core to run > the VCPU thread and ALL the host threads (including qemu I/O thread). > >> And it requires that magic bar to >> map the shadow IDT into the guest (hmm, I think Hitachi avoided this). > > Hitachi uses a different technique which seems to have the 2 disadvantages > you previously mentioned while ELI doesn't have these disadvantages. > >> It's invasive as it has to change Linux to maintain those isolated slave >> CPUs. That is, of course, based on code that was published by Hitachi. >> Yours may differ but will still have to solve the same problems. > > ELI does not require isolated/slave CPUs. > OK. Show patches. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:52663) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEKT8-00051K-H1 for qemu-devel@nongnu.org; Wed, 19 Sep 2012 09:37:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEKT2-0006XH-MI for qemu-devel@nongnu.org; Wed, 19 Sep 2012 09:37:34 -0400 Received: from mail-wi0-f175.google.com ([209.85.212.175]:55998) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEKT2-0006WX-Fy for qemu-devel@nongnu.org; Wed, 19 Sep 2012 09:37:28 -0400 Received: by wibhm2 with SMTP id hm2so4078432wib.10 for ; Wed, 19 Sep 2012 06:37:26 -0700 (PDT) Sender: Muli Ben-Yehuda Date: Wed, 19 Sep 2012 16:37:20 +0300 From: Muli Ben-Yehuda Message-ID: <20120919133720.GB22659@snow> References: <50587258.9090303@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50587258.9090303@siemens.com> Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Nadav Amit , Nadav Har'el , GaoYi , "qemu-devel@nongnu.org" , Abel Gordon On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote: > On 2012-09-18 14:50, GaoYi wrote: > > Hi Jan, > > > > I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. > > I wonder whether this mechanism will be incorporated in KVM someday. > > Likely not. Both Intel and AMD will soon ship hardware that > obsoletes this invasive and imperfect software solution, see also > [1]. Hi Jan, I think this decision may be short-sighted. First, it remains to be seen how good the upcoming hardware support for interrupt virtualization will be once it ships. Second, history has been pretty consistent in that it takes several years for hardware to catch up to software, and even when it does, software sometimes still wins. (There are workloads where shadow page tables still outperform EPT/NPT, to give a concrete example). Although ELI works today, I'd be happy to hear what parts of it you find "invasive" and "imperfect" (what software ever is perfect?) Cheers, Muli From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:52992) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TELUt-0007Gc-8e for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:43:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TELUk-0003c6-65 for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:43:27 -0400 Received: from e06smtp17.uk.ibm.com ([195.75.94.113]:45756) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TELUj-0003bX-TC for qemu-devel@nongnu.org; Wed, 19 Sep 2012 10:43:18 -0400 Received: from /spool/local by e06smtp17.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 19 Sep 2012 15:43:15 +0100 Received: from d06av11.portsmouth.uk.ibm.com (d06av11.portsmouth.uk.ibm.com [9.149.37.252]) by b06cxnps4076.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q8JEh6w141943226 for ; Wed, 19 Sep 2012 14:43:06 GMT Received: from d06av11.portsmouth.uk.ibm.com (loopback [127.0.0.1]) by d06av11.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q8JEhCIW025383 for ; Wed, 19 Sep 2012 08:43:12 -0600 In-Reply-To: <5059D26E.2050808@siemens.com> References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> Message-ID: From: Abel Gordon Date: Wed, 19 Sep 2012 17:43:11 +0300 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Nadav Har'El , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit > It's imperfect as you need to dedicate a core to pure guest-mode load > and cannot run userspace on that core (cannot walk through > userspace-based device models e.g.). That's not correct. For the evaluation, we dedicated a core for each guest to maximize the performance but this is not a requirement. You can over-commit cores and share them across multiple VMs. In this case, you will be sharing the cycles among all the VMs and the performance per VM will be degraded. In addition, if you share a core, an interrupt for a guest may be raised while the corresponding VCPU is not running. Depending on the workload, most of the interrupts will be generated while the VCPU runs or not. > and cannot run userspace on that core (cannot walk through > userspace-based device models e.g.). That's not correct. We can run any thread (including user-space threads) on the same core we run the VMs (VCPU threads). In fact, we did that for the ELI evaluation: we shared a single core to run the VCPU thread and ALL the host threads (including qemu I/O thread). > And it requires that magic bar to > map the shadow IDT into the guest (hmm, I think Hitachi avoided this). Hitachi uses a different technique which seems to have the 2 disadvantages you previously mentioned while ELI doesn't have these disadvantages. > It's invasive as it has to change Linux to maintain those isolated slave > CPUs. That is, of course, based on code that was published by Hitachi. > Yours may differ but will still have to solve the same problems. ELI does not require isolated/slave CPUs. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:60481) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEZXL-0000jq-Az for qemu-devel@nongnu.org; Thu, 20 Sep 2012 01:42:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEZXJ-000249-T0 for qemu-devel@nongnu.org; Thu, 20 Sep 2012 01:42:55 -0400 Received: from mail-lb0-f173.google.com ([209.85.217.173]:37290) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEZXJ-00023x-H7 for qemu-devel@nongnu.org; Thu, 20 Sep 2012 01:42:53 -0400 Received: by lbbgm13 with SMTP id gm13so1811986lbb.4 for ; Wed, 19 Sep 2012 22:42:51 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <5059DAC4.9020907@siemens.com> References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> Date: Thu, 20 Sep 2012 13:42:51 +0800 Message-ID: From: GaoYi Content-Type: multipart/alternative; boundary=e0cb4efe358e1df02104ca1b98fa Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Nadav Har'El , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit , Abel Gordon --e0cb4efe358e1df02104ca1b98fa Content-Type: text/plain; charset=ISO-8859-1 The CPU isolation in Hitachi patches is just to improve the real time performance of GUEST. The core of it, direct IRQ delivery, is very similar to that of ELI. For the ELI patches, (1) Since EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and requires modifications on GUEST code, it should not be included in the KVM. Only the ELI delivery part, which plays a vital role in performance improvement, should be considered. (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for some linux kernel as Hitachi patches do), to make this part independent of linux kernel version. Yi 2012/9/19 Jan Kiszka > On 2012-09-19 16:43, Abel Gordon wrote: > > > >> It's imperfect as you need to dedicate a core to pure guest-mode load > >> and cannot run userspace on that core (cannot walk through > >> userspace-based device models e.g.). > > > > That's not correct. > > For the evaluation, we dedicated a core for each guest to maximize the > > performance but this > > is not a requirement. You can over-commit cores and share them across > > multiple VMs. In this case, > > you will be sharing the cycles among all the VMs and the performance per > VM > > will be degraded. > > In addition, if you share a core, an interrupt for a guest may be raised > > while the corresponding > > VCPU is not running. Depending on the workload, most of the interrupts > will > > be generated while > > the VCPU runs or not. > > > >> and cannot run userspace on that core (cannot walk through > >> userspace-based device models e.g.). > > > > That's not correct. > > We can run any thread (including user-space threads) on the same core we > > run the VMs (VCPU threads). > > In fact, we did that for the ELI evaluation: we shared a single core to > run > > the VCPU thread and ALL the host threads (including qemu I/O thread). > > > >> And it requires that magic bar to > >> map the shadow IDT into the guest (hmm, I think Hitachi avoided this). > > > > Hitachi uses a different technique which seems to have the 2 > disadvantages > > you previously mentioned while ELI doesn't have these disadvantages. > > > >> It's invasive as it has to change Linux to maintain those isolated slave > >> CPUs. That is, of course, based on code that was published by Hitachi. > >> Yours may differ but will still have to solve the same problems. > > > > ELI does not require isolated/slave CPUs. > > > > OK. Show patches. > > Jan > > -- > Siemens AG, Corporate Technology, CT RTC ITP SDP-DE > Corporate Competence Center Embedded Linux > --e0cb4efe358e1df02104ca1b98fa Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable =A0=A0 The CPU isolation in Hitachi patches is just to improve the real tim= e performance of GUEST. The core of it, direct IRQ delivery, is very simila= r to that of ELI.
=A0=A0=A0 For the ELI patches,
=A0=A0=A0 (1) Since = EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and req= uires modifications on GUEST code, it should not be included in the KVM. On= ly the ELI delivery part, which plays a vital role in performance improveme= nt, should be considered.
=A0=A0=A0 (2) It should be provided in the kvm-kmod or qemu-kvm ( not just = for some linux kernel as Hitachi patches do), to make this part independent= of linux kernel version.

Yi

2012/= 9/19 Jan Kiszka <jan.kiszka@siemens.com>
On 2= 012-09-19 16:43, Abel Gordon wrote:
>
>> It's imperfect as you need to dedicate a core to pure guest-mo= de load
>> and cannot run userspace on that core (cannot walk through
>> userspace-based device models e.g.).
>
> That's not correct.
> For the evaluation, we dedicated a core for each guest to maximize the=
> performance but this
> is not a requirement. You can over-commit cores and share them across<= br> > multiple VMs. In this case,
> you will be sharing the cycles among all the VMs and the performance p= er VM
> will be degraded.
> In addition, if you share a core, an interrupt for a guest may be rais= ed
> while the corresponding
> VCPU is not running. Depending on the workload, most of the interrupts= will
> be generated while
> the VCPU runs or not.
>
>> and cannot run userspace on that core (cannot walk through
>> userspace-based device models e.g.).
>
> That's not correct.
> We can run any thread (including user-space threads) on the same core = we
> run the VMs (VCPU threads).
> In fact, we did that for the ELI evaluation: we shared a single core t= o run
> the VCPU thread and ALL the host threads (including qemu I/O thread).<= br> >
>> And it requires that magic bar to
>> map the shadow IDT into the guest (hmm, I think Hitachi avoided th= is).
>
> Hitachi uses a different technique which seems to have the 2 disadvant= ages
> you previously mentioned while ELI doesn't have these disadvantage= s.
>
>> It's invasive as it has to change Linux to maintain those isol= ated slave
>> CPUs. That is, of course, based on code that was published by Hita= chi.
>> Yours may differ but will still have to solve the same problems. >
> ELI does not require isolated/slave CPUs.
>

OK. Show patches.

Jan

--
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

--e0cb4efe358e1df02104ca1b98fa-- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:59818) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEaia-0005Q3-Uq for qemu-devel@nongnu.org; Thu, 20 Sep 2012 02:58:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEaiW-0001LK-4M for qemu-devel@nongnu.org; Thu, 20 Sep 2012 02:58:36 -0400 Received: from e06smtp12.uk.ibm.com ([195.75.94.108]:60574) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEaiV-0001L1-RO for qemu-devel@nongnu.org; Thu, 20 Sep 2012 02:58:32 -0400 Received: from /spool/local by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 20 Sep 2012 07:58:30 +0100 Received: from d06av07.portsmouth.uk.ibm.com (d06av07.portsmouth.uk.ibm.com [9.149.37.248]) by b06cxnps4074.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q8K6wKM753215326 for ; Thu, 20 Sep 2012 06:58:20 GMT Received: from d06av07.portsmouth.uk.ibm.com (d06av07.portsmouth.uk.ibm.com [127.0.0.1]) by d06av07.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q8K6QAuc023718 for ; Thu, 20 Sep 2012 02:26:10 -0400 In-Reply-To: References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> Message-ID: From: Abel Gordon Date: Thu, 20 Sep 2012 09:58:24 +0300 MIME-Version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: quoted-printable Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: GaoYi Cc: Nadav Har'El , Jan Kiszka , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit GaoYi wrote on 20/09/2012 08:42:51 AM: > =A0=A0 The CPU isolation in Hitachi patches is just to improve the re= al > time performance of GUEST. The core of it, direct IRQ delivery, is > very similar to that of ELI. > =A0=A0=A0 For the ELI patches, > =A0=A0=A0 (1) Since EOI part of ELI is already supported by the Intel= > Sandy Bridge CPUs and requires modifications on GUEST code, it > should not be included in the KVM. Only the ELI delivery part, which > plays a vital role in performance improvement, should be considered. Giving to the guest direct access to the EOI MSR (if x2APIC is availabl= e) is what we call "ELI completion". Note this mechanism is not so simple,= there are some cases (which are not part of the critical path) where EL= I must trap EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so= we simulated the behavior changing the guest code. In any case, as you can see in the paper, the big part of the improveme= nt comes from "ELI delivery". "ELI completion" improvement will be even smaller with the latest KVM EOI optimizations for the memory based= LAPIC. > =A0=A0=A0 (2) It should be provided in the kvm-kmod or qemu-kvm ( not= just > for some linux kernel as Hitachi patches do), to make this part > independent of linux kernel version. Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we sh= ould also modify VFIO for newer kvm versions. = From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:48916) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEayH-0002YS-M1 for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:14:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEayB-0006Ya-Sg for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:14:49 -0400 Received: from goliath.siemens.de ([192.35.17.28]:31551) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEayB-0006YL-If for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:14:43 -0400 Message-ID: <505AC257.3080000@siemens.com> Date: Thu, 20 Sep 2012 09:14:31 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Abel Gordon Cc: Nadav Har'El , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit On 2012-09-20 08:58, Abel Gordon wrote: > > > GaoYi wrote on 20/09/2012 08:42:51 AM: > >> The CPU isolation in Hitachi patches is just to improve the real >> time performance of GUEST. The core of it, direct IRQ delivery, is >> very similar to that of ELI. >> For the ELI patches, >> (1) Since EOI part of ELI is already supported by the Intel >> Sandy Bridge CPUs and requires modifications on GUEST code, it >> should not be included in the KVM. Only the ELI delivery part, which >> plays a vital role in performance improvement, should be considered. > > Giving to the guest direct access to the EOI MSR (if x2APIC is available) > is what we call "ELI completion". Note this mechanism is not so simple, > there are some cases (which are not part of the critical path) where ELI > must trap > EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we > simulated the behavior changing the guest code. > In any case, as you can see in the paper, the big part of the improvement > comes from "ELI delivery". "ELI completion" improvement will be > even smaller with the latest KVM EOI optimizations for the memory based > LAPIC. > >> (2) It should be provided in the kvm-kmod or qemu-kvm ( not just >> for some linux kernel as Hitachi patches do), to make this part >> independent of linux kernel version. > > Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should > also modify VFIO for newer kvm versions. Again: If you think the feature is non-invasive, send patches against the kernel and QEMU. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:38042) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEb2p-0004TH-N7 for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:19:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEb2j-00089K-EI for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:19:31 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37446) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEb2j-00089G-5d for qemu-devel@nongnu.org; Thu, 20 Sep 2012 03:19:25 -0400 Date: Thu, 20 Sep 2012 10:19:13 +0300 From: Gleb Natapov Message-ID: <20120920071912.GX20907@redhat.com> References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> <505AC257.3080000@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <505AC257.3080000@siemens.com> Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka Cc: Nadav Har'El , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Avi Kivity , Muli Ben-Yehuda , Nadav Amit , Abel Gordon On Thu, Sep 20, 2012 at 09:14:31AM +0200, Jan Kiszka wrote: > On 2012-09-20 08:58, Abel Gordon wrote: > > > > > > GaoYi wrote on 20/09/2012 08:42:51 AM: > > > >> The CPU isolation in Hitachi patches is just to improve the real > >> time performance of GUEST. The core of it, direct IRQ delivery, is > >> very similar to that of ELI. > >> For the ELI patches, > >> (1) Since EOI part of ELI is already supported by the Intel > >> Sandy Bridge CPUs and requires modifications on GUEST code, it > >> should not be included in the KVM. Only the ELI delivery part, which > >> plays a vital role in performance improvement, should be considered. > > > > Giving to the guest direct access to the EOI MSR (if x2APIC is available) > > is what we call "ELI completion". Note this mechanism is not so simple, > > there are some cases (which are not part of the critical path) where ELI > > must trap > > EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we > > simulated the behavior changing the guest code. > > In any case, as you can see in the paper, the big part of the improvement > > comes from "ELI delivery". "ELI completion" improvement will be > > even smaller with the latest KVM EOI optimizations for the memory based > > LAPIC. > > > >> (2) It should be provided in the kvm-kmod or qemu-kvm ( not just > >> for some linux kernel as Hitachi patches do), to make this part > >> independent of linux kernel version. > > > > Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should > > also modify VFIO for newer kvm versions. > > Again: If you think the feature is non-invasive, send patches against > the kernel and QEMU. > And explain why it is better than what modern HW provides. -- Gleb. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:39454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEctz-0008PY-1H for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:18:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEcts-0000Z2-Vk for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:18:30 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59491) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEcts-0000Ys-ND for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:18:24 -0400 Message-ID: <505ADF53.7030306@redhat.com> Date: Thu, 20 Sep 2012 12:18:11 +0300 From: Avi Kivity MIME-Version: 1.0 References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> <505AC257.3080000@siemens.com> <20120920071912.GX20907@redhat.com> In-Reply-To: <20120920071912.GX20907@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gleb Natapov Cc: Nadav Har'El , Jan Kiszka , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Muli Ben-Yehuda , Nadav Amit , Abel Gordon On 09/20/2012 10:19 AM, Gleb Natapov wrote: >> Again: If you think the feature is non-invasive, send patches against >> the kernel and QEMU. >> > And explain why it is better than what modern HW provides. If it's non-invasive (and easily maintainable), it doesn't have to be better. -- error compiling committee.c: too many arguments to function From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:47315) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEcxk-0000zj-9M for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:22:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TEcxZ-0001wo-6A for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:22:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:18550) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TEcxY-0001we-Ud for qemu-devel@nongnu.org; Thu, 20 Sep 2012 05:22:13 -0400 Date: Thu, 20 Sep 2012 12:22:05 +0300 From: Gleb Natapov Message-ID: <20120920092204.GY20907@redhat.com> References: <50587258.9090303@siemens.com> <20120919133720.GB22659@snow> <5059D26E.2050808@siemens.com> <5059DAC4.9020907@siemens.com> <505AC257.3080000@siemens.com> <20120920071912.GX20907@redhat.com> <505ADF53.7030306@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <505ADF53.7030306@redhat.com> Subject: Re: [Qemu-devel] Will the ELI incorporated in theKVM? List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Avi Kivity Cc: Nadav Har'El , Jan Kiszka , GaoYi , "qemu-devel@nongnu.org" , Michael D Day , Muli Ben-Yehuda , Nadav Amit , Abel Gordon On Thu, Sep 20, 2012 at 12:18:11PM +0300, Avi Kivity wrote: > On 09/20/2012 10:19 AM, Gleb Natapov wrote: > >> Again: If you think the feature is non-invasive, send patches against > >> the kernel and QEMU. > >> > > And explain why it is better than what modern HW provides. > > If it's non-invasive (and easily maintainable), it doesn't have to be > better. > > If it were all that it would have been applied already. -- Gleb.