[Bug 53611] New: nVMX: Add nested EPT

[Bug 53611] New: nVMX: Add nested EPT

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=53611

           Summary: nVMX: Add nested EPT
           Product: Virtualization
           Version: unspecified
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: nyh@math.technion.ac.il
        Regression: No


Created an attachment (id=93101)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=93101)
Nested EPT patches, v2

Nested EPT means emulating EPT for an L1 guest, allowing it to use EPT when
running a nested guest L2. When L1 uses EPT, it allows the L2 guest to set
its own cr3 and take its own page faults without either of L0 or L1 getting
involved. In many workloads this significanlty improves L2's performance over
the previous two alternatives (shadow page tables over ept, and shadow page
tables over shadow page tables). As an example, I measured a single-threaded
"make", which has a lot of context switches and page faults, on the three
options:

 shadow over shadow: 105 seconds
 shadow over EPT: 87 seconds      (this is the default currently)
 EPT over EPT: 29 seconds

 single-level virtualization (with EPT): 25 seconds

So clearly nested EPT would be a big win for such workloads.

I attach a patch set which I worked on and allowed me to measure the above
results. This is the same patch set I sent to KVM mailing list on August 1st,
2012, titled "nEPT v2: Nested EPT support for Nested VMX".

This patch set still needs some work: it is known to only work in some setups
but not others, and the file "announce" in the attached tar lists 5 things
which definitely need to be done. There were a few additional comments in the
mailing list - see
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/95395

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 53611] nVMX: Add nested EPT

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=53611


Nadav Har'El <nyh@math.technion.ac.il> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |53601




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

On 2013-02-11 13:49, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=53611
> 
>            Summary: nVMX: Add nested EPT
>            Product: Virtualization
>            Version: unspecified
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: kvm
>         AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
>         ReportedBy: nyh@math.technion.ac.il
>         Regression: No
> 
> 
> Created an attachment (id=93101)
>  --> (https://bugzilla.kernel.org/attachment.cgi?id=93101)
> Nested EPT patches, v2
> 
> Nested EPT means emulating EPT for an L1 guest, allowing it to use EPT when
> running a nested guest L2. When L1 uses EPT, it allows the L2 guest to set
> its own cr3 and take its own page faults without either of L0 or L1 getting
> involved. In many workloads this significanlty improves L2's performance over
> the previous two alternatives (shadow page tables over ept, and shadow page
> tables over shadow page tables). As an example, I measured a single-threaded
> "make", which has a lot of context switches and page faults, on the three
> options:
> 
>  shadow over shadow: 105 seconds
>  shadow over EPT: 87 seconds      (this is the default currently)
>  EPT over EPT: 29 seconds
> 
>  single-level virtualization (with EPT): 25 seconds
> 
> So clearly nested EPT would be a big win for such workloads.
> 
> I attach a patch set which I worked on and allowed me to measure the above
> results. This is the same patch set I sent to KVM mailing list on August 1st,
> 2012, titled "nEPT v2: Nested EPT support for Nested VMX".
> 
> This patch set still needs some work: it is known to only work in some setups
> but not others, and the file "announce" in the attached tar lists 5 things
> which definitely need to be done. There were a few additional comments in the
> mailing list - see
> http://comments.gmane.org/gmane.comp.emulators.kvm.devel/95395
> 

I suppose they do not apply anymore as well. Do you have a recent tree
around somewhere or plan to resume work on it?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nadav Har'El]

Hi,

On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
> On 2013-02-11 13:49, bugzilla-daemon@bugzilla.kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=53611
> >            Summary: nVMX: Add nested EPT

Yikes, I didn't realize that these bugzilla edits all get spammed to the
entire mailing list :( Sorry about those...

> I suppose they do not apply anymore as well. Do you have a recent tree
> around somewhere or plan to resume work on it?

Unfortunately, no - I did not have time to work on these patches since
August.

The reason I'm now stuffing these things into the bug tracker is that
at the end of this month I am leaving IBM to a new job, so I'm pretty
sure I won't have time myself to continue any work on nested VMX, and
would like for the missing nested-VMX features to be documented in case
someone else comes along and wants to improve it. So unfortunately, you
should expect more of this bugzilla spam on the mailing list...

Nadav.

-- 
Nadav Har'El                        |         Monday, Feb 11 2013, 1 Adar 5773
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |The message above is just this
http://nadav.harel.org.il           |signature's way of propagating itself.

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

On 2013-02-11 14:27, Nadav Har'El wrote:
> Hi,
> 
> On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>> On 2013-02-11 13:49, bugzilla-daemon@bugzilla.kernel.org wrote:
>>> https://bugzilla.kernel.org/show_bug.cgi?id=53611
>>>            Summary: nVMX: Add nested EPT
> 
> Yikes, I didn't realize that these bugzilla edits all get spammed to the
> entire mailing list :( Sorry about those...
> 
>> I suppose they do not apply anymore as well. Do you have a recent tree
>> around somewhere or plan to resume work on it?
> 
> Unfortunately, no - I did not have time to work on these patches since
> August.
> 
> The reason I'm now stuffing these things into the bug tracker is that
> at the end of this month I am leaving IBM to a new job, so I'm pretty
> sure I won't have time myself to continue any work on nested VMX, and
> would like for the missing nested-VMX features to be documented in case
> someone else comes along and wants to improve it. So unfortunately, you
> should expect more of this bugzilla spam on the mailing list...

A pity that you cannot finish this great work. But documenting the open
issues is definitely helpful and welcome.

Best wishes,
Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Mon, Feb 11, 2013 at 5:27 AM, Nadav Har'El <nyh@math.technion.ac.il> wrote:
> Hi,
>
> On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>> On 2013-02-11 13:49, bugzilla-daemon@bugzilla.kernel.org wrote:
>> > https://bugzilla.kernel.org/show_bug.cgi?id=53611
>> >            Summary: nVMX: Add nested EPT
>
> Yikes, I didn't realize that these bugzilla edits all get spammed to the
> entire mailing list :( Sorry about those...
>
>> I suppose they do not apply anymore as well. Do you have a recent tree
>> around somewhere or plan to resume work on it?
>
> Unfortunately, no - I did not have time to work on these patches since
> August.
>
> The reason I'm now stuffing these things into the bug tracker is that
> at the end of this month I am leaving IBM to a new job, so I'm pretty
> sure I won't have time myself to continue any work on nested VMX, and
> would like for the missing nested-VMX features to be documented in case
> someone else comes along and wants to improve it. So unfortunately, you
> should expect more of this bugzilla spam on the mailing list...
>

I looked at your (old) patches, and they seem to be very useful
although some of them require rebasing or rewriting. We are interested
in completing the nested-VMX features.

-- 
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

On 2013-02-12 20:13, Nakajima, Jun wrote:
> On Mon, Feb 11, 2013 at 5:27 AM, Nadav Har'El <nyh@math.technion.ac.il> wrote:
>> Hi,
>>
>> On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>>> On 2013-02-11 13:49, bugzilla-daemon@bugzilla.kernel.org wrote:
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=53611
>>>>            Summary: nVMX: Add nested EPT
>>
>> Yikes, I didn't realize that these bugzilla edits all get spammed to the
>> entire mailing list :( Sorry about those...
>>
>>> I suppose they do not apply anymore as well. Do you have a recent tree
>>> around somewhere or plan to resume work on it?
>>
>> Unfortunately, no - I did not have time to work on these patches since
>> August.
>>
>> The reason I'm now stuffing these things into the bug tracker is that
>> at the end of this month I am leaving IBM to a new job, so I'm pretty
>> sure I won't have time myself to continue any work on nested VMX, and
>> would like for the missing nested-VMX features to be documented in case
>> someone else comes along and wants to improve it. So unfortunately, you
>> should expect more of this bugzilla spam on the mailing list...
>>
> 
> I looked at your (old) patches, and they seem to be very useful
> although some of them require rebasing or rewriting. We are interested
> in completing the nested-VMX features.

That's great news. Can you estimate when you will be able to work on it?

I will have a use case for nEPT soon - testing purposes. But working
into the KVM MMU and doing the port myself may unfortunately consume too
much time here.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Tue, Feb 12, 2013 at 11:43 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
>
> On 2013-02-12 20:13, Nakajima, Jun wrote:
> > I looked at your (old) patches, and they seem to be very useful
> > although some of them require rebasing or rewriting. We are interested
> > in completing the nested-VMX features.
>
> That's great news. Can you estimate when you will be able to work on it?
>

We have started looking at the pataches first. But I couldn't
reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit)  (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Without the patches, the L2 guest works. With it, it hangs at boot
time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration
(e.g kernel/commit id used, L1/L2 guests) was documented as well.

> I will have a use case for nEPT soon - testing purposes. But working
> into the KVM MMU and doing the port myself may unfortunately consume too
> much time here.
>
> Jan
>

--
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nadav Har'El]

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
> We have started looking at the pataches first. But I couldn't
> reproduce the results by simply applying the original patches to v3.6:
> - L2 Ubuntu 12.04 (64-bit)  (smp 2)
> - L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
> - L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
> ones in nept-v2.tgz).
> https://bugzilla.kernel.org/attachment.cgi?id=93101
> 
> Without the patches, the L2 guest works. With it, it hangs at boot
> time (just black screen):
> - EPT was detected by L1 KVM.
> - UP L2 didn't help.
> - Looks like it's looping at EPT_walk_add_generic at the same address in L0.
> 
> Will take a closer look. It would be helpful if the test configuration
> (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit
ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33,
while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

I've heard another report of my patch not working with newer L1/L2 -
the report said that L2 failed to boot (like you reported), and also
that L1 became unstable (running anything in it gave a memory fault).
So it is very likely that this code still has bugs - but since I already
know of errors and holes that need to be plugged (see the announcement file
together with the patches), it's not very surprising :( These patches
definitely need some lovin', but it's easier than starting from scratch.

Nadav.

-- 
Nadav Har'El                        |       Tuesday, Feb 26 2013, 16 Adar 5773
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |I think therefore I am. My computer
http://nadav.harel.org.il           |thinks for me, therefore I am not.

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

On 2013-02-26 15:11, Nadav Har'El wrote:
> On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>> We have started looking at the pataches first. But I couldn't
>> reproduce the results by simply applying the original patches to v3.6:
>> - L2 Ubuntu 12.04 (64-bit)  (smp 2)
>> - L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
>> - L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
>> ones in nept-v2.tgz).
>> https://bugzilla.kernel.org/attachment.cgi?id=93101
>>
>> Without the patches, the L2 guest works. With it, it hangs at boot
>> time (just black screen):
>> - EPT was detected by L1 KVM.
>> - UP L2 didn't help.
>> - Looks like it's looping at EPT_walk_add_generic at the same address in L0.
>>
>> Will take a closer look. It would be helpful if the test configuration
>> (e.g kernel/commit id used, L1/L2 guests) was documented as well.
> 
> I sent the patches in August 1st, and they applied to commit
> ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.
> 
> In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33,
> while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.
> 
> I've heard another report of my patch not working with newer L1/L2 -
> the report said that L2 failed to boot (like you reported), and also
> that L1 became unstable (running anything in it gave a memory fault).
> So it is very likely that this code still has bugs - but since I already
> know of errors and holes that need to be plugged (see the announcement file
> together with the patches), it's not very surprising :( These patches
> definitely need some lovin', but it's easier than starting from scratch.

FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for
3.6) for a while. They work OK for my use case (static mapping) but
apparently lock up L2 when starting KVM on KVM, just as reported. I
didn't look into any details there, still busy with fixing other issues
like CR0/CR4 handling (which I came across while adding unrestricted
guest support on top of EPT).

Given that I'm porting now patches between that branch and "next" back
and forth (I depend on EPT), it would be really great if someone
familiar with the KVM MMU (or enough time) could port the series to the
current git head. That would not solve remaining bugs but could trigger
more development, maybe also help me jumping into this.

Thanks,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Bug 53611] New: nVMX: Add nested EPT

[Gleb Natapov]

On Tue, Feb 26, 2013 at 08:43:13PM +0100, Jan Kiszka wrote:
> On 2013-02-26 15:11, Nadav Har'El wrote:
> > On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
> >> We have started looking at the pataches first. But I couldn't
> >> reproduce the results by simply applying the original patches to v3.6:
> >> - L2 Ubuntu 12.04 (64-bit)  (smp 2)
> >> - L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
> >> - L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
> >> ones in nept-v2.tgz).
> >> https://bugzilla.kernel.org/attachment.cgi?id=93101
> >>
> >> Without the patches, the L2 guest works. With it, it hangs at boot
> >> time (just black screen):
> >> - EPT was detected by L1 KVM.
> >> - UP L2 didn't help.
> >> - Looks like it's looping at EPT_walk_add_generic at the same address in L0.
> >>
> >> Will take a closer look. It would be helpful if the test configuration
> >> (e.g kernel/commit id used, L1/L2 guests) was documented as well.
> > 
> > I sent the patches in August 1st, and they applied to commit
> > ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.
> > 
> > In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33,
> > while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.
> > 
> > I've heard another report of my patch not working with newer L1/L2 -
> > the report said that L2 failed to boot (like you reported), and also
> > that L1 became unstable (running anything in it gave a memory fault).
> > So it is very likely that this code still has bugs - but since I already
> > know of errors and holes that need to be plugged (see the announcement file
> > together with the patches), it's not very surprising :( These patches
> > definitely need some lovin', but it's easier than starting from scratch.
> 
> FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for
> 3.6) for a while. They work OK for my use case (static mapping) but
> apparently lock up L2 when starting KVM on KVM, just as reported. I
> didn't look into any details there, still busy with fixing other issues
> like CR0/CR4 handling (which I came across while adding unrestricted
> guest support on top of EPT).
> 
> Given that I'm porting now patches between that branch and "next" back
> and forth (I depend on EPT), it would be really great if someone
> familiar with the KVM MMU (or enough time) could port the series to the
> current git head. That would not solve remaining bugs but could trigger
> more development, maybe also help me jumping into this.
> 
I'd like to do that. See if I'll have time...

--
			Gleb.

[Bug 53611] nVMX: Add nested EPT

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=53611





--- Comment #1 from Nadav Har'El <nyh@math.technion.ac.il>  2013-02-27 08:14:13 ---
In addition to the known issues list in the "announce" file attached above, I
thought of several more issues that should be considered:

1. When switching back and forth between L1 and L2 it will be a waste to throw
away the EPT table already built. So I hope (need to check...) that the EPT
table is cached. But what is the cache key - the cr3? But cr3 has a different
meaning in L2 and L1, so it might not be correct to use that as the key.

2. When L0 swaps out pages, it needs to remove these entries in all EPT tables,
including the cached EPT02 even if not currently used. Does this happen
correctly?

3. If L1 uses EPT ("nested EPT") and gives us a malformed EPT12 table, we may
need to inject an EPT_MISCONFIGURATION exit when building the merged EPT02
entry. Typically, we do this building (see "fetch" in paging_tmpl.h) when
handling an EPT violation exit from L2, so if we encounter this problem
instead of reentering L2 immediately, we should exit to L1 with an EPT
misconfigration. I'm not sure exactly how to notice this problem. Perhaps the
pagetable walking code, which in our case walks EPT12 already notices a problem
and does something (#GP perhaps?) and we need to have it do the EPT misconfig
instead. But it is possible we need to add additional tests that are not done
for normal page tables - in particularly regarding reserved bits, and
especially bit 5 (in EPT it is reserved, in normal page tables it is the
accessed bit). This issue is low priority, as it only deals with the error
path; A well-written L1 will not caused EPT configurations anyway.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Tue, Feb 26, 2013 at 11:43 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2013-02-26 15:11, Nadav Har'El wrote:
>> On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>>> We have started looking at the pataches first. But I couldn't
>>> reproduce the results by simply applying the original patches to v3.6:
>>> - L2 Ubuntu 12.04 (64-bit)  (smp 2)
>>> - L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
>>> - L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
>>> ones in nept-v2.tgz).
>>> https://bugzilla.kernel.org/attachment.cgi?id=93101
>>>
>>> Without the patches, the L2 guest works. With it, it hangs at boot
>>> time (just black screen):
>>> - EPT was detected by L1 KVM.
>>> - UP L2 didn't help.
>>> - Looks like it's looping at EPT_walk_add_generic at the same address in L0.
>>>
>>> Will take a closer look. It would be helpful if the test configuration
>>> (e.g kernel/commit id used, L1/L2 guests) was documented as well.
>>
>> I sent the patches in August 1st, and they applied to commit
>> ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.
>>
>> In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33,
>> while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.
>>
>> I've heard another report of my patch not working with newer L1/L2 -
>> the report said that L2 failed to boot (like you reported), and also
>> that L1 became unstable (running anything in it gave a memory fault).
>> So it is very likely that this code still has bugs - but since I already
>> know of errors and holes that need to be plugged (see the announcement file
>> together with the patches), it's not very surprising :( These patches
>> definitely need some lovin', but it's easier than starting from scratch.
>
> FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for
> 3.6) for a while. They work OK for my use case (static mapping) but
> apparently lock up L2 when starting KVM on KVM, just as reported. I
> didn't look into any details there, still busy with fixing other issues
> like CR0/CR4 handling (which I came across while adding unrestricted
> guest support on top of EPT).

I have some updates on this. We rebased the patched to the latest KVM
(L0). It turned out that the version of L1 KVM/Linux matters. At that
time, actually I used v3.7 kernel for L1, and the L2 didn't work as I
described above. If I use v3.5 or older for L1, L2 works with the EPT
patches. So, I guess some changes made to v3.6 might have exposed a
bug with the nested EPT patches or somewhere. We are looking at the
changes to root-cause it.

>
> Given that I'm porting now patches between that branch and "next" back
> and forth (I depend on EPT), it would be really great if someone
> familiar with the KVM MMU (or enough time) could port the series to the
> current git head. That would not solve remaining bugs but could trigger
> more development, maybe also help me jumping into this.
>
> Thanks,
> Jan
>


-- 
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3029 bytes --]

On 2013-03-05 05:45, Nakajima, Jun wrote:
> On Tue, Feb 26, 2013 at 11:43 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2013-02-26 15:11, Nadav Har'El wrote:
>>> On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":
>>>> We have started looking at the pataches first. But I couldn't
>>>> reproduce the results by simply applying the original patches to v3.6:
>>>> - L2 Ubuntu 12.04 (64-bit)  (smp 2)
>>>> - L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
>>>> - L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the
>>>> ones in nept-v2.tgz).
>>>> https://bugzilla.kernel.org/attachment.cgi?id=93101
>>>>
>>>> Without the patches, the L2 guest works. With it, it hangs at boot
>>>> time (just black screen):
>>>> - EPT was detected by L1 KVM.
>>>> - UP L2 didn't help.
>>>> - Looks like it's looping at EPT_walk_add_generic at the same address in L0.
>>>>
>>>> Will take a closer look. It would be helpful if the test configuration
>>>> (e.g kernel/commit id used, L1/L2 guests) was documented as well.
>>>
>>> I sent the patches in August 1st, and they applied to commit
>>> ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.
>>>
>>> In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33,
>>> while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.
>>>
>>> I've heard another report of my patch not working with newer L1/L2 -
>>> the report said that L2 failed to boot (like you reported), and also
>>> that L1 became unstable (running anything in it gave a memory fault).
>>> So it is very likely that this code still has bugs - but since I already
>>> know of errors and holes that need to be plugged (see the announcement file
>>> together with the patches), it's not very surprising :( These patches
>>> definitely need some lovin', but it's easier than starting from scratch.
>>
>> FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for
>> 3.6) for a while. They work OK for my use case (static mapping) but
>> apparently lock up L2 when starting KVM on KVM, just as reported. I
>> didn't look into any details there, still busy with fixing other issues
>> like CR0/CR4 handling (which I came across while adding unrestricted
>> guest support on top of EPT).
> 
> I have some updates on this. We rebased the patched to the latest KVM
> (L0). It turned out that the version of L1 KVM/Linux matters. At that
> time, actually I used v3.7 kernel for L1, and the L2 didn't work as I
> described above. If I use v3.5 or older for L1, L2 works with the EPT
> patches. So, I guess some changes made to v3.6 might have exposed a
> bug with the nested EPT patches or somewhere. We are looking at the
> changes to root-cause it.

Great to hear! Would you mind to share your work early, even when it's
not yet stable?

At least regarding lockups or misbehaviors of L1 and L2, some of the
patches I posted recently may help. Did you try to merge them as well?

Thanks,
Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Mon, Mar 4, 2013 at 8:45 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> I have some updates on this. We rebased the patched to the latest KVM
> (L0). It turned out that the version of L1 KVM/Linux matters. At that
> time, actually I used v3.7 kernel for L1, and the L2 didn't work as I
> described above. If I use v3.5 or older for L1, L2 works with the EPT
> patches. So, I guess some changes made to v3.6 might have exposed a
> bug with the nested EPT patches or somewhere. We are looking at the
> changes to root-cause it.
>

Finally I've had more time to work on this, and I think I've fixed
this. The problem was that the exit qualification for EPT violation
(to L1) was not accurate (enough). And I needed to save the exit
qualification upon EPT violation somewhere. Today, that information is
converted to error_code (see below), and we lose the information.  We
need to use  at least the lower 3 bits when injecting EPT violation to
the L1 VMM. I tried to use the upper bytes of error_code to pass  part
of the exit qualification, but it didn't work well. Any suggestion for
the place to store the value? kvm_vcpu?

       ...
        /* It is a write fault? */
        error_code = exit_qualification & (1U << 1);
        /* ept page table is present? */
        error_code |= (exit_qualification >> 3) & 0x1;

        return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);

--
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

On 2013-03-22 07:23, Nakajima, Jun wrote:
> On Mon, Mar 4, 2013 at 8:45 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>> I have some updates on this. We rebased the patched to the latest KVM
>> (L0). It turned out that the version of L1 KVM/Linux matters. At that
>> time, actually I used v3.7 kernel for L1, and the L2 didn't work as I
>> described above. If I use v3.5 or older for L1, L2 works with the EPT
>> patches. So, I guess some changes made to v3.6 might have exposed a
>> bug with the nested EPT patches or somewhere. We are looking at the
>> changes to root-cause it.
>>
> 
> Finally I've had more time to work on this, and I think I've fixed
> this. The problem was that the exit qualification for EPT violation
> (to L1) was not accurate (enough). And I needed to save the exit
> qualification upon EPT violation somewhere. Today, that information is
> converted to error_code (see below), and we lose the information.  We
> need to use  at least the lower 3 bits when injecting EPT violation to
> the L1 VMM. I tried to use the upper bytes of error_code to pass  part
> of the exit qualification, but it didn't work well. Any suggestion for
> the place to store the value? kvm_vcpu?
> 
>        ...
>         /* It is a write fault? */
>         error_code = exit_qualification & (1U << 1);
>         /* ept page table is present? */
>         error_code |= (exit_qualification >> 3) & 0x1;
> 
>         return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);

I don't have a full picture (already asked you to post / git-push your
intermediate state), but nested related states typically go to
nested_vmx, thus vcpu_vmx.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

On 2013-03-22 17:45, Jan Kiszka wrote:
> On 2013-03-22 07:23, Nakajima, Jun wrote:
>> On Mon, Mar 4, 2013 at 8:45 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>>> I have some updates on this. We rebased the patched to the latest KVM
>>> (L0). It turned out that the version of L1 KVM/Linux matters. At that
>>> time, actually I used v3.7 kernel for L1, and the L2 didn't work as I
>>> described above. If I use v3.5 or older for L1, L2 works with the EPT
>>> patches. So, I guess some changes made to v3.6 might have exposed a
>>> bug with the nested EPT patches or somewhere. We are looking at the
>>> changes to root-cause it.
>>>
>>
>> Finally I've had more time to work on this, and I think I've fixed
>> this. The problem was that the exit qualification for EPT violation
>> (to L1) was not accurate (enough). And I needed to save the exit
>> qualification upon EPT violation somewhere. Today, that information is
>> converted to error_code (see below), and we lose the information.  We
>> need to use  at least the lower 3 bits when injecting EPT violation to
>> the L1 VMM. I tried to use the upper bytes of error_code to pass  part
>> of the exit qualification, but it didn't work well. Any suggestion for
>> the place to store the value? kvm_vcpu?
>>
>>        ...
>>         /* It is a write fault? */
>>         error_code = exit_qualification & (1U << 1);
>>         /* ept page table is present? */
>>         error_code |= (exit_qualification >> 3) & 0x1;
>>
>>         return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);
> 
> I don't have a full picture (already asked you to post / git-push your
> intermediate state), but nested related states typically go to
> nested_vmx, thus vcpu_vmx.

Ping regarding publication. I'm about to redo your porting work as we
are making no progress.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Wed, Apr 24, 2013 at 12:25 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>
>> I don't have a full picture (already asked you to post / git-push your
>> intermediate state), but nested related states typically go to
>> nested_vmx, thus vcpu_vmx.
>
> Ping regarding publication. I'm about to redo your porting work as we
> are making no progress.
>

Sorry about the slow progress. We've been distracted by some priority
things. The patches are ready (i.e. working), but we are cleaning them
up. I'll send what we have today.

--
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

On 2013-04-24 17:55, Nakajima, Jun wrote:
> On Wed, Apr 24, 2013 at 12:25 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>>
>>> I don't have a full picture (already asked you to post / git-push your
>>> intermediate state), but nested related states typically go to
>>> nested_vmx, thus vcpu_vmx.
>>
>> Ping regarding publication. I'm about to redo your porting work as we
>> are making no progress.
>>
> 
> Sorry about the slow progress. We've been distracted by some priority
> things. The patches are ready (i.e. working), but we are cleaning them
> up. I'll send what we have today.

Great news, thanks a lot!

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Wed, Apr 24, 2013 at 8:55 AM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> Sorry about the slow progress. We've been distracted by some priority
> things. The patches are ready (i.e. working), but we are cleaning them
> up. I'll send what we have today.

So, I have sent them, and frankly we are still cleaning up.  Please
bear with us.
We are also sending one more patchset to deal with EPT
misconfiguration, but Linux should run in L2 on top of L1 KVM.

--
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Gleb Natapov]

On Thu, Apr 25, 2013 at 01:00:42AM -0700, Nakajima, Jun wrote:
> On Wed, Apr 24, 2013 at 8:55 AM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> > Sorry about the slow progress. We've been distracted by some priority
> > things. The patches are ready (i.e. working), but we are cleaning them
> > up. I'll send what we have today.
> 
> So, I have sent them, and frankly we are still cleaning up.  Please
> bear with us.
> We are also sending one more patchset to deal with EPT
> misconfiguration, but Linux should run in L2 on top of L1 KVM.
> 
The patches are mangled and unreadable. Please resend using "git
send-email".

--
			Gleb.

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On 2013-04-25 10:00, Nakajima, Jun wrote:
> On Wed, Apr 24, 2013 at 8:55 AM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>> Sorry about the slow progress. We've been distracted by some priority
>> things. The patches are ready (i.e. working), but we are cleaning them
>> up. I'll send what we have today.
> 
> So, I have sent them, and frankly we are still cleaning up.  Please
> bear with us.
> We are also sending one more patchset to deal with EPT
> misconfiguration, but Linux should run in L2 on top of L1 KVM.

That's great but - as Gleb already said - unfortunately not yet usable.
I'd like to rebase my fixes and enhancements (unrestricted guest mode
specifically) on top these days, and also run some tests with a non-KVM
guest. So, if git send-email is not yet working there, I would also be
happy about a public git repository.

Thanks,
Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Bug 53611] New: nVMX: Add nested EPT

[Nakajima, Jun]

On Thu, Apr 25, 2013 at 11:26 PM, Jan Kiszka <jan.kiszka@web.de> wrote:

> That's great but - as Gleb already said - unfortunately not yet usable.
> I'd like to rebase my fixes and enhancements (unrestricted guest mode
> specifically) on top these days, and also run some tests with a non-KVM
> guest. So, if git send-email is not yet working there, I would also be
> happy about a public git repository.
>

I re-submitted the patches last night using git send-email this time.
We had some email problems at that time, and I needed to use a
workaround (imap-send) at that time (and it didn't work well).

-- 
Jun
Intel Open Source Technology Center

Re: [Bug 53611] New: nVMX: Add nested EPT

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On 2013-04-26 18:07, Nakajima, Jun wrote:
> On Thu, Apr 25, 2013 at 11:26 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> 
>> That's great but - as Gleb already said - unfortunately not yet usable.
>> I'd like to rebase my fixes and enhancements (unrestricted guest mode
>> specifically) on top these days, and also run some tests with a non-KVM
>> guest. So, if git send-email is not yet working there, I would also be
>> happy about a public git repository.
>>
> 
> I re-submitted the patches last night using git send-email this time.
> We had some email problems at that time, and I needed to use a
> workaround (imap-send) at that time (and it didn't work well).

I've picked them up (except for Xinhao's follow-up patch) and rebased
them over next + my pending patches:

    git://git.kiszka.org/linux-kvm.git queues/nept

Some patches required a bit massaging to apply, and the last one had a
trivial style issue. Feel free to integrate the changes. I didn't look
into functional details yet.

Instead, I've rebased my unrestricted guest mode patch plus a bunch of
fixes around nEPT and that feature. See the branch above. I'm currently
testing them, and it looks very good so far. Unrestricted guest mode
speeds up L2's BIOS and boot loader phase noticeably.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

[Bug 53611] nVMX: Add nested EPT

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Bandan Das <bsd@makefile.in> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |94971

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 53611] nVMX: Add nested EPT

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Paolo Bonzini <bonzini@gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |bonzini@gnu.org
     Kernel Version|                            |3.19
         Resolution|---                         |CODE_FIX

--- Comment #2 from Paolo Bonzini <bonzini@gnu.org> ---
Fixed by commit afa61f752ba6 (Advertise the support of EPT to the L1 guest,
through the appropriate MSR., 2013-08-07)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.