[Qemu-devel] Will the ELI incorporated in theKVM?

[Qemu-devel] Will the ELI incorporated in theKVM?

[GaoYi]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Hi Jan,

   I have followed a previous thread about ELI proposed by Abel Gordon,
http://www.spinics.net/lists/kvm/msg73907.html.
   I wonder whether this mechanism will be incorporated in KVM someday.
   Thanks,

Yi

[-- Attachment #2: Type: text/html, Size: 400 bytes --]

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Jan Kiszka]

On 2012-09-18 14:50, GaoYi wrote:
> Hi Jan,
> 
>    I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html.
>    I wonder whether this mechanism will be incorporated in KVM someday.

Likely not. Both Intel and AMD will soon ship hardware that obsoletes
this invasive and imperfect software solution, see also [1].

Jan

[1] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/97715

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Muli Ben-Yehuda]

On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote:
> On 2012-09-18 14:50, GaoYi wrote:
> > Hi Jan,
> > 
> >    I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html.
> >    I wonder whether this mechanism will be incorporated in KVM someday.
> 
> Likely not. Both Intel and AMD will soon ship hardware that
> obsoletes this invasive and imperfect software solution, see also
> [1].

Hi Jan,

I think this decision may be short-sighted. First, it remains to be
seen how good the upcoming hardware support for interrupt
virtualization will be once it ships. Second, history has been pretty
consistent in that it takes several years for hardware to catch up to
software, and even when it does, software sometimes still wins. (There
are workloads where shadow page tables still outperform EPT/NPT, to
give a concrete example). Although ELI works today, I'd be happy to
hear what parts of it you find "invasive" and "imperfect" (what
software ever is perfect?)

Cheers,
Muli

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Jan Kiszka]

[putting Avi on CC as the final decision maker]

On 2012-09-19 15:37, Muli Ben-Yehuda wrote:
> On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote:
>> On 2012-09-18 14:50, GaoYi wrote:
>>> Hi Jan,
>>>
>>>    I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html.
>>>    I wonder whether this mechanism will be incorporated in KVM someday.
>>
>> Likely not. Both Intel and AMD will soon ship hardware that
>> obsoletes this invasive and imperfect software solution, see also
>> [1].
> 
> Hi Jan,
> 
> I think this decision may be short-sighted. First, it remains to be
> seen how good the upcoming hardware support for interrupt
> virtualization will be once it ships. Second, history has been pretty
> consistent in that it takes several years for hardware to catch up to
> software, and even when it does, software sometimes still wins. (There
> are workloads where shadow page tables still outperform EPT/NPT, to
> give a concrete example). Although ELI works today, I'd be happy to
> hear what parts of it you find "invasive" and "imperfect" (what
> software ever is perfect?)

It's imperfect as you need to dedicate a core to pure guest-mode load
and cannot run userspace on that core (cannot walk through
userspace-based device models e.g.). And it requires that magic bar to
map the shadow IDT into the guest (hmm, I think Hitachi avoided this).

It's invasive as it has to change Linux to maintain those isolated slave
CPUs. That is, of course, based on code that was published by Hitachi.
Yours may differ but will still have to solve the same problems.

Back then in our discussion, it was unclear if and when real hardware
will actually show up. Now the spec is already there, hardware should
follow within months (typically). I'm not aware of limitations regarding
zero-exit virtualization so far. But please share any insights you
collected from studying the specs or recent patches!

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Avi Kivity]

On 09/19/2012 05:10 PM, Jan Kiszka wrote:
>> Although ELI works today, I'd be happy to
>> hear what parts of it you find "invasive" and "imperfect" (what
>> software ever is perfect?)
> 
> It's imperfect as you need to dedicate a core to pure guest-mode load
> and cannot run userspace on that core (cannot walk through
> userspace-based device models e.g.). And it requires that magic bar to
> map the shadow IDT into the guest (hmm, I think Hitachi avoided this).
> 
> It's invasive as it has to change Linux to maintain those isolated slave
> CPUs. That is, of course, based on code that was published by Hitachi.
> Yours may differ but will still have to solve the same problems.
> 
> Back then in our discussion, it was unclear if and when real hardware
> will actually show up. Now the spec is already there, hardware should
> follow within months (typically). I'm not aware of limitations regarding
> zero-exit virtualization so far. But please share any insights you
> collected from studying the specs or recent patches!

This matches my thoughts on the matter exactly.  I would add that ELI
adds a maintenance burden of supporting rarely used code, as ELI can
hardly be called mainstream.  On the other hand, the new hardware
support will be used by every guest running on new hardware, for IPIs,
virtual interrupts, and assigned device interrupts.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Abel Gordon]

> It's imperfect as you need to dedicate a core to pure guest-mode load
> and cannot run userspace on that core (cannot walk through
> userspace-based device models e.g.).

That's not correct.
For the evaluation, we dedicated a core for each guest to maximize the
performance but this
is not a requirement. You can over-commit cores and share them across
multiple VMs. In this case,
you will be sharing the cycles among all the VMs and the performance per VM
will be degraded.
In addition, if you share a core, an interrupt for a guest may be raised
while the corresponding
VCPU is not running. Depending on the workload, most of the interrupts will
be generated while
the VCPU runs or not.

> and cannot run userspace on that core (cannot walk through
> userspace-based device models e.g.).

That's not correct.
We can run any thread (including user-space threads) on the same core we
run the VMs (VCPU threads).
In fact, we did that for the ELI evaluation: we shared a single core to run
the VCPU thread and ALL the host threads (including qemu I/O thread).

> And it requires that magic bar to
> map the shadow IDT into the guest (hmm, I think Hitachi avoided this).

Hitachi uses a different technique which seems to have the 2 disadvantages
you previously mentioned while ELI doesn't have these disadvantages.

> It's invasive as it has to change Linux to maintain those isolated slave
> CPUs. That is, of course, based on code that was published by Hitachi.
> Yours may differ but will still have to solve the same problems.

ELI does not require isolated/slave CPUs.

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Jan Kiszka]

On 2012-09-19 16:43, Abel Gordon wrote:
> 
>> It's imperfect as you need to dedicate a core to pure guest-mode load
>> and cannot run userspace on that core (cannot walk through
>> userspace-based device models e.g.).
> 
> That's not correct.
> For the evaluation, we dedicated a core for each guest to maximize the
> performance but this
> is not a requirement. You can over-commit cores and share them across
> multiple VMs. In this case,
> you will be sharing the cycles among all the VMs and the performance per VM
> will be degraded.
> In addition, if you share a core, an interrupt for a guest may be raised
> while the corresponding
> VCPU is not running. Depending on the workload, most of the interrupts will
> be generated while
> the VCPU runs or not.
> 
>> and cannot run userspace on that core (cannot walk through
>> userspace-based device models e.g.).
> 
> That's not correct.
> We can run any thread (including user-space threads) on the same core we
> run the VMs (VCPU threads).
> In fact, we did that for the ELI evaluation: we shared a single core to run
> the VCPU thread and ALL the host threads (including qemu I/O thread).
> 
>> And it requires that magic bar to
>> map the shadow IDT into the guest (hmm, I think Hitachi avoided this).
> 
> Hitachi uses a different technique which seems to have the 2 disadvantages
> you previously mentioned while ELI doesn't have these disadvantages.
> 
>> It's invasive as it has to change Linux to maintain those isolated slave
>> CPUs. That is, of course, based on code that was published by Hitachi.
>> Yours may differ but will still have to solve the same problems.
> 
> ELI does not require isolated/slave CPUs.
> 

OK. Show patches.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[GaoYi]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

   The CPU isolation in Hitachi patches is just to improve the real time
performance of GUEST. The core of it, direct IRQ delivery, is very similar
to that of ELI.
    For the ELI patches,
    (1) Since EOI part of ELI is already supported by the Intel Sandy
Bridge CPUs and requires modifications on GUEST code, it should not be
included in the KVM. Only the ELI delivery part, which plays a vital role
in performance improvement, should be considered.
    (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for
some linux kernel as Hitachi patches do), to make this part independent of
linux kernel version.

Yi

2012/9/19 Jan Kiszka <jan.kiszka@siemens.com>

> On 2012-09-19 16:43, Abel Gordon wrote:
> >
> >> It's imperfect as you need to dedicate a core to pure guest-mode load
> >> and cannot run userspace on that core (cannot walk through
> >> userspace-based device models e.g.).
> >
> > That's not correct.
> > For the evaluation, we dedicated a core for each guest to maximize the
> > performance but this
> > is not a requirement. You can over-commit cores and share them across
> > multiple VMs. In this case,
> > you will be sharing the cycles among all the VMs and the performance per
> VM
> > will be degraded.
> > In addition, if you share a core, an interrupt for a guest may be raised
> > while the corresponding
> > VCPU is not running. Depending on the workload, most of the interrupts
> will
> > be generated while
> > the VCPU runs or not.
> >
> >> and cannot run userspace on that core (cannot walk through
> >> userspace-based device models e.g.).
> >
> > That's not correct.
> > We can run any thread (including user-space threads) on the same core we
> > run the VMs (VCPU threads).
> > In fact, we did that for the ELI evaluation: we shared a single core to
> run
> > the VCPU thread and ALL the host threads (including qemu I/O thread).
> >
> >> And it requires that magic bar to
> >> map the shadow IDT into the guest (hmm, I think Hitachi avoided this).
> >
> > Hitachi uses a different technique which seems to have the 2
> disadvantages
> > you previously mentioned while ELI doesn't have these disadvantages.
> >
> >> It's invasive as it has to change Linux to maintain those isolated slave
> >> CPUs. That is, of course, based on code that was published by Hitachi.
> >> Yours may differ but will still have to solve the same problems.
> >
> > ELI does not require isolated/slave CPUs.
> >
>
> OK. Show patches.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
> Corporate Competence Center Embedded Linux
>

[-- Attachment #2: Type: text/html, Size: 3245 bytes --]

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Abel Gordon]

GaoYi <gaoyi709@gmail.com> wrote on 20/09/2012 08:42:51 AM:

>    The CPU isolation in Hitachi patches is just to improve the real
> time performance of GUEST. The core of it, direct IRQ delivery, is
> very similar to that of ELI.
>     For the ELI patches,
>     (1) Since EOI part of ELI is already supported by the Intel
> Sandy Bridge CPUs and requires modifications on GUEST code, it
> should not be included in the KVM. Only the ELI delivery part, which
> plays a vital role in performance improvement, should be considered.

Giving to the guest direct access to the EOI MSR (if x2APIC is available)
is what we call "ELI completion". Note this mechanism is not so simple,
there are some cases (which are not part of the critical path) where ELI
must trap
EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we
simulated the behavior changing the guest code.
In any case, as you can see in the paper, the big part of the improvement
comes from "ELI delivery". "ELI completion" improvement will be
even smaller with the latest KVM EOI optimizations for the memory based
LAPIC.

>     (2) It should be provided in the kvm-kmod or qemu-kvm ( not just
> for some linux kernel as Hitachi patches do), to make this part
> independent of linux kernel version.

Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should
also modify VFIO for newer kvm versions.

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Jan Kiszka]

On 2012-09-20 08:58, Abel Gordon wrote:
> 
> 
> GaoYi <gaoyi709@gmail.com> wrote on 20/09/2012 08:42:51 AM:
> 
>>    The CPU isolation in Hitachi patches is just to improve the real
>> time performance of GUEST. The core of it, direct IRQ delivery, is
>> very similar to that of ELI.
>>     For the ELI patches,
>>     (1) Since EOI part of ELI is already supported by the Intel
>> Sandy Bridge CPUs and requires modifications on GUEST code, it
>> should not be included in the KVM. Only the ELI delivery part, which
>> plays a vital role in performance improvement, should be considered.
> 
> Giving to the guest direct access to the EOI MSR (if x2APIC is available)
> is what we call "ELI completion". Note this mechanism is not so simple,
> there are some cases (which are not part of the critical path) where ELI
> must trap
> EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we
> simulated the behavior changing the guest code.
> In any case, as you can see in the paper, the big part of the improvement
> comes from "ELI delivery". "ELI completion" improvement will be
> even smaller with the latest KVM EOI optimizations for the memory based
> LAPIC.
> 
>>     (2) It should be provided in the kvm-kmod or qemu-kvm ( not just
>> for some linux kernel as Hitachi patches do), to make this part
>> independent of linux kernel version.
> 
> Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should
> also modify VFIO for newer kvm versions.

Again: If you think the feature is non-invasive, send patches against
the kernel and QEMU.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Gleb Natapov]

On Thu, Sep 20, 2012 at 09:14:31AM +0200, Jan Kiszka wrote:
> On 2012-09-20 08:58, Abel Gordon wrote:
> > 
> > 
> > GaoYi <gaoyi709@gmail.com> wrote on 20/09/2012 08:42:51 AM:
> > 
> >>    The CPU isolation in Hitachi patches is just to improve the real
> >> time performance of GUEST. The core of it, direct IRQ delivery, is
> >> very similar to that of ELI.
> >>     For the ELI patches,
> >>     (1) Since EOI part of ELI is already supported by the Intel
> >> Sandy Bridge CPUs and requires modifications on GUEST code, it
> >> should not be included in the KVM. Only the ELI delivery part, which
> >> plays a vital role in performance improvement, should be considered.
> > 
> > Giving to the guest direct access to the EOI MSR (if x2APIC is available)
> > is what we call "ELI completion". Note this mechanism is not so simple,
> > there are some cases (which are not part of the critical path) where ELI
> > must trap
> > EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we
> > simulated the behavior changing the guest code.
> > In any case, as you can see in the paper, the big part of the improvement
> > comes from "ELI delivery". "ELI completion" improvement will be
> > even smaller with the latest KVM EOI optimizations for the memory based
> > LAPIC.
> > 
> >>     (2) It should be provided in the kvm-kmod or qemu-kvm ( not just
> >> for some linux kernel as Hitachi patches do), to make this part
> >> independent of linux kernel version.
> > 
> > Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should
> > also modify VFIO for newer kvm versions.
> 
> Again: If you think the feature is non-invasive, send patches against
> the kernel and QEMU.
> 
And explain why it is better than what modern HW provides.

--
			Gleb.

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Avi Kivity]

On 09/20/2012 10:19 AM, Gleb Natapov wrote:
>> Again: If you think the feature is non-invasive, send patches against
>> the kernel and QEMU.
>> 
> And explain why it is better than what modern HW provides.

If it's non-invasive (and easily maintainable), it doesn't have to be
better.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Will the ELI incorporated in theKVM?

[Gleb Natapov]

On Thu, Sep 20, 2012 at 12:18:11PM +0300, Avi Kivity wrote:
> On 09/20/2012 10:19 AM, Gleb Natapov wrote:
> >> Again: If you think the feature is non-invasive, send patches against
> >> the kernel and QEMU.
> >> 
> > And explain why it is better than what modern HW provides.
> 
> If it's non-invasive (and easily maintainable), it doesn't have to be
> better.
> 
> 
If it were all that it would have been applied already.

--
			Gleb.