From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoffer Dall Subject: Re: [RFC PATCH v2 00/38] Nested Virtualization on KVM/ARM Date: Wed, 19 Jul 2017 10:49:38 +0200 Message-ID: <20170719084938.GA11435@cbox> References: <1500397144-16232-1-git-send-email-jintack.lim@linaro.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 699BC40D68 for ; Wed, 19 Jul 2017 04:48:52 -0400 (EDT) Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PuHbGQdwoRqY for ; Wed, 19 Jul 2017 04:48:51 -0400 (EDT) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 2FF2740C5A for ; Wed, 19 Jul 2017 04:48:50 -0400 (EDT) Received: by mail-wm0-f45.google.com with SMTP id t70so34995344wmt.1 for ; Wed, 19 Jul 2017 01:49:40 -0700 (PDT) Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: Jintack Lim Cc: KVM General , david.daney@cavium.com, Catalin Marinas , Will Deacon , kvmarm@lists.cs.columbia.edu, stefan@hello-penguin.com, corbet@lwn.net, daniel.lezcano@linaro.org, linux@armlinux.org.uk, arm-mail-list , andy.gross@linaro.org, Marc Zyngier , Bandan Das , cov@codeaurora.org, wcohen@redhat.com, mchehab@kernel.org, ard.biesheuvel@linaro.org, linux-doc@vger.kernel.org, lkml - Kernel Mailing List , Paolo Bonzini , akpm@linux-foundation.org List-Id: kvmarm@lists.cs.columbia.edu SGkgSmludGFjaywKCk9uIFR1ZSwgSnVsIDE4LCAyMDE3IGF0IDEwOjIzOjA1UE0gLTA0MDAsIEpp bnRhY2sgTGltIHdyb3RlOgo+IE9uIFR1ZSwgSnVsIDE4LCAyMDE3IGF0IDEyOjU4IFBNLCBKaW50 YWNrIExpbSA8amludGFjay5saW1AbGluYXJvLm9yZz4gd3JvdGU6Cj4gPiBOZXN0ZWQgdmlydHVh bGl6YXRpb24gaXMgdGhlIGFiaWxpdHkgdG8gcnVuIGEgdmlydHVhbCBtYWNoaW5lIGluc2lkZSBh bm90aGVyCj4gPiB2aXJ0dWFsIG1hY2hpbmUuIEluIG90aGVyIHdvcmRzLCBpdOKAmXMgYWJvdXQg cnVubmluZyBhIGh5cGVydmlzb3IgKHRoZSBndWVzdAo+ID4gaHlwZXJ2aXNvcikgb24gdG9wIG9m IGFub3RoZXIgaHlwZXJ2aXNvciAodGhlIGhvc3QgaHlwZXJ2aXNvcikuCj4gPgo+ID4gU3VwcG9y dGluZyBuZXN0ZWQgdmlydHVhbGl6YXRpb24gb24gQVJNIG1lYW5zIHRoYXQgdGhlIGh5cGVydmlz b3IgcHJvdmlkZXMgbm90Cj4gPiBvbmx5IEVMMC9FTDEgZXhlY3V0aW9uIGVudmlyb25tZW50IHRv IFZNcyBhcyBpdCB1c3VhbGx5IGRvZXMgYnV0IGFsc28gdGhlCj4gPiB2aXJ0dWFsaXphdGlvbiBl eHRlbnNpb25zIGluY2x1ZGluZyBFTDIgZXhlY3V0aW9uIGVudmlyb25tZW50LiBPbmNlIHRoZSBo b3N0Cj4gPiBoeXBlcnZpc29yIHByb3ZpZGVzIHRob3NlIGV4ZWN1dGlvbiBlbnZpcm9ubWVudHMg dG8gdGhlIFZNcywgdGhlbiB0aGUgZ3Vlc3QKPiA+IGh5cGVydmlzb3IgY2FuIHJ1biBpdHMgb3du IFZNcyAobmVzdGVkIFZNcykgbmF0dXJhbGx5Lgo+ID4KPiA+IFRoaXMgc2VyaWVzIHN1cHBvcnRz IG5lc3RlZCB2aXJ0dWFsaXphdGlvbiBvbiBhcm02NC4gQVJNIHJlY2VudGx5IGFubm91bmNlZCBh bgo+ID4gZXh0ZW5zaW9uIChBUk12OC4zKSB3aGljaCBoYXMgc3VwcG9ydCBmb3IgbmVzdGVkIHZp cnR1YWxpemF0aW9uWzFdLiBUaGlzIHBhdGNoCj4gPiBzZXQgaXMgYmFzZWQgb24gdGhlIEFSTXY4 LjMgc3BlY2lmaWNhdGlvbiBhbmQgdGVzdGVkIG9uIHRoZSBGYXN0TW9kZWwgd2l0aAo+ID4gQVJN djguMyBleHRlbnNpb24uCj4gPgo+ID4gVGhlIHdob2xlIHBhdGNoIHNldCB0byBzdXBwb3J0IG5l c3RlZCB2aXJ0dWFsaXphdGlvbiBpcyBodWdlIG92ZXIgNzAKPiA+IHBhdGNoZXMsIHNvIEkgY2F0 ZWdvcml6ZWQgdGhlbSBpbnRvIGZvdXIgcGFydHM6IENQVSwgbWVtb3J5LCBWR0lDLCBhbmQgdGlt ZXIKPiA+IHZpcnR1YWxpemF0aW9uLiBUaGlzIHBhdGNoIHNlcmllcyBpcyB0aGUgZmlyc3QgcGFy dC4KPiA+Cj4gPiBDUFUgdmlydHVhbGl6YXRpb24gcGF0Y2ggc2VyaWVzIHByb3ZpZGVzIGJhc2lj IG5lc3RlZCB2aXJ0dWFsaXphdGlvbiBmcmFtZXdvcmsKPiA+IGFuZCBpbnN0cnVjdGlvbiBlbXVs YXRpb25zIGluY2x1ZGluZyB2OC4xIFZIRSBmZWF0dXJlIGFuZCB2OC4zIG5lc3RlZAo+ID4gdmly dHVhbGl6YXRpb24gZmVhdHVyZSBmb3IgVk1zLgo+ID4KPiA+IFRoaXMgcGF0Y2ggc2VyaWVzIGFn YWluIGNhbiBiZSBkaXZpZGVkIGludG8gZm91ciBwYXJ0cy4gUGF0Y2ggMSB0byA1IGludHJvZHVj ZXMKPiA+IG5lc3RlZCB2aXJ0dWFsaXphdGlvbiBieSBkaXNjb3ZlcmluZyBoYXJkd2FyZSBmZWF0 dXJlLCBhZGRpbmcgYSBrZXJuZWwKPiA+IHBhcmFtZXRlciBhbmQgYWxsb3dpbmcgdGhlIHVzZXJz cGFjZSB0byBzZXQgdGhlIGluaXRpYWwgQ1BVIG1vZGUgdG8gRUwyLgo+ID4KPiA+IFBhdGNoIDYg dG8gMjUgYXJlIHRvIHN1cHBvcnQgdGhlIEVMMiBleGVjdXRpb24gZW52aXJvbm1lbnQsIHRoZSB2 aXJ0dWFsIEVMMiwgdG8KPiA+IGEgVk0gb24gdjguMCBhcmNoaXRlY3R1cmUuIFdlIGRlLXByaXZp bGVnZSB0aGUgZ3Vlc3QgaHlwZXJ2aXNvciBhbmQgZW11bGF0ZSB0aGUKPiA+IHZpcnR1YWwgRUwy IG1vZGUgaW4gRUwxIHVzaW5nIHRoZSBoYXJkd2FyZSBmZWF0dXJlcyBwcm92aWRlZCBieSBBUk12 OC4zOyBUaGUKPiA+IGhvc3QgaHlwZXJ2aXNvciBtYW5hZ2VzIHZpcnR1YWwgRUwyIHJlZ2lzdGVy IHN0YXRlIGZvciB0aGUgZ3Vlc3QgaHlwZXJ2aXNvcgo+ID4gYW5kIHNoYWRvdyBFTDEgcmVnaXN0 ZXIgc3RhdGUgdGhhdCByZWZsZWN0cyB0aGUgdmlydHVhbCBFTDIgcmVnaXN0ZXIgc3RhdGUgdG8K PiA+IHJ1biB0aGUgZ3Vlc3QgaHlwZXJ2aXNvciBpbiBFTDEuCj4gPgo+ID4gUGF0Y2ggMjYgdG8g MzMgYWRkIHN1cHBvcnQgZm9yIHRoZSB2aXJ0dWFsIEVMMiB3aXRoIFZpcnR1YWxpemF0aW9uIEhv c3QKPiA+IEV4dGVuc2lvbnMuIFRoZXNlIHBhdGNoZXMgZW11bGF0ZSBuZXdseSBkZWZpbmVkIHJl Z2lzdGVycyBhbmQgYml0cyBpbiB2OC4xIGFuZAo+ID4gYWxsb3cgdGhlIHZpcnR1YWwgRUwyIHRv IGFjY2VzcyBFTDIgcmVnaXN0ZXIgc3RhdGVzIHZpYSBFTDEgcmVnaXN0ZXIgYWNjZXNzZXMKPiA+ IGFzIGluIHRoZSByZWFsIEVMMi4KPiA+Cj4gPiBQYXRjaCAzNCB0byAzOCBhcmUgdG8gc3VwcG9y dCBmb3IgdGhlIHZpcnR1YWwgRUwyIHdpdGggbmVzdGVkIHZpcnR1YWxpemF0aW9uLgo+ID4gVGhl c2UgZW5hYmxlIHJlY3Vyc2l2ZSBuZXN0ZWQgdmlydHVhbGl6YXRpb24uCj4gPgo+ID4gVGhpcyBw YXRjaCBzZXQgaXMgdGVzdGVkIG9uIHRoZSBGYXN0TW9kZWwgd2l0aCB0aGUgdjguMyBleHRlbnNp b24gZm9yIGFybTY0IGFuZAo+ID4gYSBjdWJpZXRydWNrIGZvciBhcm0zMi4gT24gdGhlIEZhc3RN b2RlbCwgdGhlIGhvc3QgYW5kIHRoZSBndWVzdCBrZXJuZWxzIGFyZQo+ID4gY29tcGlsZWQgd2l0 aCBhbmQgd2l0aG91dCBWSEUsIHNvIHRoZXJlIGFyZSBmb3VyIGNvbWJpbmF0aW9ucy4gSSB3YXMg YWJsZSB0bwo+ID4gYm9vdCBTTVAgTGludXggaW4gdGhlIG5lc3RlZCBWTSBvbiBhbGwgZm91ciBj b25maWd1cmF0aW9ucyBhbmQgYWJsZSB0byBydW4KPiA+IGhhY2tiZW5jaC4gSSBhbHNvIGNoZWNr ZWQgdGhhdCByZWd1bGFyIFZNcyBjb3VsZCBib290IHdoZW4gdGhlIG5lc3RlZAo+ID4gdmlydHVh bGl6YXRpb24ga2VybmVsIHBhcmFtZXRlciB3YXMgbm90IHNldC4gT24gdGhlIGN1YmlldHJ1Y2ss IEkgYWxzbyB2ZXJpZmllZAo+ID4gdGhhdCByZWd1bGFyIFZNcyBjb3VsZCBib290IGFzIHdlbGwu Cj4gPgo+ID4gSSdsbCBzaGFyZSBteSBleHBlcmltZW50IHNldHVwIHNob3J0bHkuCj4gCj4gSSBz dW1tYXJpemVkIG15IGV4cGVyaW1lbnQgc2V0dXAgaGVyZS4KPiAKPiBodHRwczovL2dpdGh1Yi5j b20vY29sdW1iaWEvbmVzdGluZy1wdWIvd2lraS9OZXN0ZWQtdmlydHVhbGl6YXRpb24tb24tQVJN LXNldHVwCj4gCgpUaGFua3MgZm9yIHNoYXJpbmcgdGhpcy4KCj4gPgo+ID4gRXZlbiB0aG91Z2gg dGhpcyB3b3JrIGhhcyBzb21lIGxpbWl0YXRpb25zIGFuZCBUT0RPcywgSSdkIGFwcHJlY2lhdGUg ZWFybHkKPiA+IGZlZWRiYWNrIG9uIHRoaXMgUkZDLiBTcGVjaWZpY2FsbHksIEknbSBpbnRlcmVz dGVkIGluOgo+ID4KPiA+IC0gT3ZlcmFsbCBkZXNpZ24gdG8gbWFuYWdlIHZjcHUgY29udGV4dCBm b3IgdGhlIHZpcnR1YWwgRUwyCj4gPiAtIFZlcmlmeWluZyBjb3JyZWN0IEVMMiByZWdpc3RlciBj b25maWd1cmF0aW9ucyBzdWNoIGFzIEhDUl9FTDIsIENQVFJfRUwyCj4gPiAgIChQYXRjaCAzMCBh bmQgMzIpCj4gPiAtIFBhdGNoIG9yZ2FuaXphdGlvbiBhbmQgY29kaW5nIHN0eWxlCj4gCj4gSSBh bHNvIHdvbmRlciBpZiB0aGUgaGFyZHdhcmUgYW5kL29yIEtWTSBkbyBub3Qgc3VwcG9ydCBuZXN0 ZWQKPiB2aXJ0dWFsaXphdGlvbiBidXQgdGhlIHVzZXJzcGFjZSB1c2VzIG5lc3RlZCB2aXJ0dWFs aXphdGlvbiBvcHRpb24sCj4gd2hpY2ggb25lIGlzIGJldHRlcjogZ2l2aW5nIGFuIGVycm9yIG9y IGxhdW5jaGluZyBhIHJlZ3VsYXIgVk0KPiBzaWxlbnRseS4KPiAKCkkgdGhpbmsgS1ZNIHNob3Vs ZCBjb21wbGFpbiB0byB1c2Vyc3BhY2UgaWYgdXNlcnNwYWNlIHRyaWVzIHRvIHNldCBhCmZlYXR1 cmUgaXQgZG9lcyBub3Qgc3VwcG9ydCwgYW5kIEkgdGhpbmsgdXNlcnNwYWNlIHNob3VsZCBnaXZl IGFzCm1lYW5pbmdmdWwgYW4gZXJyb3IgbWVzc2FnZSBhcyBwb3NzaWJsZSB0byB0aGUgdXNlciB3 aGVuIHRoYXQgaGFwcGVucy4KClRoYW5rcywKLUNocmlzdG9mZmVyCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmt2bWFybSBtYWlsaW5nIGxpc3QKa3ZtYXJt QGxpc3RzLmNzLmNvbHVtYmlhLmVkdQpodHRwczovL2xpc3RzLmNzLmNvbHVtYmlhLmVkdS9tYWls bWFuL2xpc3RpbmZvL2t2bWFybQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: cdall@linaro.org (Christoffer Dall) Date: Wed, 19 Jul 2017 10:49:38 +0200 Subject: [RFC PATCH v2 00/38] Nested Virtualization on KVM/ARM In-Reply-To: References: <1500397144-16232-1-git-send-email-jintack.lim@linaro.org> Message-ID: <20170719084938.GA11435@cbox> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi Jintack, On Tue, Jul 18, 2017 at 10:23:05PM -0400, Jintack Lim wrote: > On Tue, Jul 18, 2017 at 12:58 PM, Jintack Lim wrote: > > Nested virtualization is the ability to run a virtual machine inside another > > virtual machine. In other words, it?s about running a hypervisor (the guest > > hypervisor) on top of another hypervisor (the host hypervisor). > > > > Supporting nested virtualization on ARM means that the hypervisor provides not > > only EL0/EL1 execution environment to VMs as it usually does but also the > > virtualization extensions including EL2 execution environment. Once the host > > hypervisor provides those execution environments to the VMs, then the guest > > hypervisor can run its own VMs (nested VMs) naturally. > > > > This series supports nested virtualization on arm64. ARM recently announced an > > extension (ARMv8.3) which has support for nested virtualization[1]. This patch > > set is based on the ARMv8.3 specification and tested on the FastModel with > > ARMv8.3 extension. > > > > The whole patch set to support nested virtualization is huge over 70 > > patches, so I categorized them into four parts: CPU, memory, VGIC, and timer > > virtualization. This patch series is the first part. > > > > CPU virtualization patch series provides basic nested virtualization framework > > and instruction emulations including v8.1 VHE feature and v8.3 nested > > virtualization feature for VMs. > > > > This patch series again can be divided into four parts. Patch 1 to 5 introduces > > nested virtualization by discovering hardware feature, adding a kernel > > parameter and allowing the userspace to set the initial CPU mode to EL2. > > > > Patch 6 to 25 are to support the EL2 execution environment, the virtual EL2, to > > a VM on v8.0 architecture. We de-privilege the guest hypervisor and emulate the > > virtual EL2 mode in EL1 using the hardware features provided by ARMv8.3; The > > host hypervisor manages virtual EL2 register state for the guest hypervisor > > and shadow EL1 register state that reflects the virtual EL2 register state to > > run the guest hypervisor in EL1. > > > > Patch 26 to 33 add support for the virtual EL2 with Virtualization Host > > Extensions. These patches emulate newly defined registers and bits in v8.1 and > > allow the virtual EL2 to access EL2 register states via EL1 register accesses > > as in the real EL2. > > > > Patch 34 to 38 are to support for the virtual EL2 with nested virtualization. > > These enable recursive nested virtualization. > > > > This patch set is tested on the FastModel with the v8.3 extension for arm64 and > > a cubietruck for arm32. On the FastModel, the host and the guest kernels are > > compiled with and without VHE, so there are four combinations. I was able to > > boot SMP Linux in the nested VM on all four configurations and able to run > > hackbench. I also checked that regular VMs could boot when the nested > > virtualization kernel parameter was not set. On the cubietruck, I also verified > > that regular VMs could boot as well. > > > > I'll share my experiment setup shortly. > > I summarized my experiment setup here. > > https://github.com/columbia/nesting-pub/wiki/Nested-virtualization-on-ARM-setup > Thanks for sharing this. > > > > Even though this work has some limitations and TODOs, I'd appreciate early > > feedback on this RFC. Specifically, I'm interested in: > > > > - Overall design to manage vcpu context for the virtual EL2 > > - Verifying correct EL2 register configurations such as HCR_EL2, CPTR_EL2 > > (Patch 30 and 32) > > - Patch organization and coding style > > I also wonder if the hardware and/or KVM do not support nested > virtualization but the userspace uses nested virtualization option, > which one is better: giving an error or launching a regular VM > silently. > I think KVM should complain to userspace if userspace tries to set a feature it does not support, and I think userspace should give as meaningful an error message as possible to the user when that happens. Thanks, -Christoffer From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753200AbdGSItu (ORCPT ); Wed, 19 Jul 2017 04:49:50 -0400 Received: from mail-wm0-f44.google.com ([74.125.82.44]:35803 "EHLO mail-wm0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751754AbdGSItl (ORCPT ); Wed, 19 Jul 2017 04:49:41 -0400 Date: Wed, 19 Jul 2017 10:49:38 +0200 From: Christoffer Dall To: Jintack Lim Cc: kvmarm@lists.cs.columbia.edu, Christoffer Dall , Marc Zyngier , corbet@lwn.net, Paolo Bonzini , Radim =?utf-8?B?S3LEjW3DocWZ?= , linux@armlinux.org.uk, Catalin Marinas , Will Deacon , akpm@linux-foundation.org, mchehab@kernel.org, cov@codeaurora.org, daniel.lezcano@linaro.org, david.daney@cavium.com, mark.rutland@arm.com, Suzuki K Poulose , stefan@hello-penguin.com, andy.gross@linaro.org, wcohen@redhat.com, ard.biesheuvel@linaro.org, shankerd@codeaurora.org, vladimir.murzin@arm.com, james.morse@arm.com, linux-doc@vger.kernel.org, lkml - Kernel Mailing List , KVM General , arm-mail-list , Bandan Das Subject: Re: [RFC PATCH v2 00/38] Nested Virtualization on KVM/ARM Message-ID: <20170719084938.GA11435@cbox> References: <1500397144-16232-1-git-send-email-jintack.lim@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Jintack, On Tue, Jul 18, 2017 at 10:23:05PM -0400, Jintack Lim wrote: > On Tue, Jul 18, 2017 at 12:58 PM, Jintack Lim wrote: > > Nested virtualization is the ability to run a virtual machine inside another > > virtual machine. In other words, it’s about running a hypervisor (the guest > > hypervisor) on top of another hypervisor (the host hypervisor). > > > > Supporting nested virtualization on ARM means that the hypervisor provides not > > only EL0/EL1 execution environment to VMs as it usually does but also the > > virtualization extensions including EL2 execution environment. Once the host > > hypervisor provides those execution environments to the VMs, then the guest > > hypervisor can run its own VMs (nested VMs) naturally. > > > > This series supports nested virtualization on arm64. ARM recently announced an > > extension (ARMv8.3) which has support for nested virtualization[1]. This patch > > set is based on the ARMv8.3 specification and tested on the FastModel with > > ARMv8.3 extension. > > > > The whole patch set to support nested virtualization is huge over 70 > > patches, so I categorized them into four parts: CPU, memory, VGIC, and timer > > virtualization. This patch series is the first part. > > > > CPU virtualization patch series provides basic nested virtualization framework > > and instruction emulations including v8.1 VHE feature and v8.3 nested > > virtualization feature for VMs. > > > > This patch series again can be divided into four parts. Patch 1 to 5 introduces > > nested virtualization by discovering hardware feature, adding a kernel > > parameter and allowing the userspace to set the initial CPU mode to EL2. > > > > Patch 6 to 25 are to support the EL2 execution environment, the virtual EL2, to > > a VM on v8.0 architecture. We de-privilege the guest hypervisor and emulate the > > virtual EL2 mode in EL1 using the hardware features provided by ARMv8.3; The > > host hypervisor manages virtual EL2 register state for the guest hypervisor > > and shadow EL1 register state that reflects the virtual EL2 register state to > > run the guest hypervisor in EL1. > > > > Patch 26 to 33 add support for the virtual EL2 with Virtualization Host > > Extensions. These patches emulate newly defined registers and bits in v8.1 and > > allow the virtual EL2 to access EL2 register states via EL1 register accesses > > as in the real EL2. > > > > Patch 34 to 38 are to support for the virtual EL2 with nested virtualization. > > These enable recursive nested virtualization. > > > > This patch set is tested on the FastModel with the v8.3 extension for arm64 and > > a cubietruck for arm32. On the FastModel, the host and the guest kernels are > > compiled with and without VHE, so there are four combinations. I was able to > > boot SMP Linux in the nested VM on all four configurations and able to run > > hackbench. I also checked that regular VMs could boot when the nested > > virtualization kernel parameter was not set. On the cubietruck, I also verified > > that regular VMs could boot as well. > > > > I'll share my experiment setup shortly. > > I summarized my experiment setup here. > > https://github.com/columbia/nesting-pub/wiki/Nested-virtualization-on-ARM-setup > Thanks for sharing this. > > > > Even though this work has some limitations and TODOs, I'd appreciate early > > feedback on this RFC. Specifically, I'm interested in: > > > > - Overall design to manage vcpu context for the virtual EL2 > > - Verifying correct EL2 register configurations such as HCR_EL2, CPTR_EL2 > > (Patch 30 and 32) > > - Patch organization and coding style > > I also wonder if the hardware and/or KVM do not support nested > virtualization but the userspace uses nested virtualization option, > which one is better: giving an error or launching a regular VM > silently. > I think KVM should complain to userspace if userspace tries to set a feature it does not support, and I think userspace should give as meaningful an error message as possible to the user when that happens. Thanks, -Christoffer