From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoffer Dall Subject: Re: [RFC 00/55] Nested Virtualization on KVM/ARM Date: Wed, 22 Feb 2017 19:23:31 +0100 Message-ID: <20170222182331.GA27653@cbox> References: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id AEC6D40CD5 for ; Wed, 22 Feb 2017 13:22:48 -0500 (EST) Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R+Wy8Xgu6jLR for ; Wed, 22 Feb 2017 13:22:46 -0500 (EST) Received: from mail-wr0-f180.google.com (mail-wr0-f180.google.com [209.85.128.180]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 94BE840C38 for ; Wed, 22 Feb 2017 13:22:45 -0500 (EST) Received: by mail-wr0-f180.google.com with SMTP id 89so7914085wrr.3 for ; Wed, 22 Feb 2017 10:23:44 -0800 (PST) Content-Disposition: inline In-Reply-To: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu To: Jintack Lim Cc: kvm@vger.kernel.org, catalin.marinas@arm.com, will.deacon@arm.com, kvmarm@lists.cs.columbia.edu, shihwei@cs.columbia.edu, lorenzo.pieralisi@arm.com, linux@armlinux.org.uk, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, andre.przywara@arm.com, kevin.brodsky@arm.com, wcohen@redhat.com, anna-maria@linutronix.de, geoff@infradead.org, linux-kernel@vger.kernel.org, pbonzini@redhat.com List-Id: kvmarm@lists.cs.columbia.edu SGkgSmludGFjaywKCgpPbiBNb24sIEphbiAwOSwgMjAxNyBhdCAwMToyMzo1NkFNIC0wNTAwLCBK aW50YWNrIExpbSB3cm90ZToKPiBOZXN0ZWQgdmlydHVhbGl6YXRpb24gaXMgdGhlIGFiaWxpdHkg dG8gcnVuIGEgdmlydHVhbCBtYWNoaW5lIGluc2lkZSBhbm90aGVyCj4gdmlydHVhbCBtYWNoaW5l LiBJbiBvdGhlciB3b3JkcywgaXTigJlzIGFib3V0IHJ1bm5pbmcgYSBoeXBlcnZpc29yICh0aGUg Z3Vlc3QKPiBoeXBlcnZpc29yKSBvbiB0b3Agb2YgYW5vdGhlciBoeXBlcnZpc29yICh0aGUgaG9z dCBoeXBlcnZpc29yKS4KPiAKPiBUaGlzIHNlcmllcyBzdXBwb3J0cyBuZXN0ZWQgdmlydHVhbGl6 YXRpb24gb24gYXJtNjQuIEFSTSByZWNlbnRseSBhbm5vdW5jZWQgYW4KPiBleHRlbnNpb24gKEFS TXY4LjMpIHdoaWNoIGhhcyBzdXBwb3J0IGZvciBuZXN0ZWQgdmlydHVhbGl6YXRpb25bMV0uIFRo aXMgc2VyaWVzCj4gaXMgYmFzZWQgb24gdGhlIEFSTXY4LjMgc3BlY2lmaWNhdGlvbi4KPiAKPiBT dXBwb3J0aW5nIG5lc3RlZCB2aXJ0dWFsaXphdGlvbiBtZWFucyB0aGF0IHRoZSBoeXBlcnZpc29y IHByb3ZpZGVzIG5vdCBvbmx5Cj4gRUwwL0VMMSBleGVjdXRpb24gZW52aXJvbm1lbnQgd2l0aCBW TXMgYXMgaXQgdXN1YWxseSBkb2VzLCBidXQgYWxzbyB0aGUKPiB2aXJ0dWFsaXphdGlvbiBleHRl bnNpb25zIGluY2x1ZGluZyBFTDIgZXhlY3V0aW9uIGVudmlyb25tZW50IHdpdGggdGhlIFZNcy4K PiBPbmNlIHRoZSBob3N0IGh5cGVydmlzb3IgcHJvdmlkZXMgdGhvc2UgZXhlY3V0aW9uIGVudmly b25tZW50IHdpdGggdGhlIFZNcywKPiB0aGVuIHRoZSBndWVzdCBoeXBlcnZpc29yIGNhbiBydW4g aXRzIG93biBWTXMgKG5lc3RlZCBWTXMpIG5hdHVyYWxseS4KPiAKPiBUbyBzdXBwb3J0IG5lc3Rl ZCB2aXJ0dWFsaXphdGlvbiBvbiBBUk0gdGhlIGh5cGVydmlzb3IgbXVzdCBlbXVsYXRlIGEgdmly dHVhbAo+IGV4ZWN1dGlvbiBlbnZpcm9ubWVudCBjb25zaXN0aW5nIG9mIEVMMiwgRUwxLCBhbmQg RUwwLCBhcyB0aGUgZ3Vlc3QgaHlwZXJ2aXNvcgo+IHdpbGwgcnVuIGluIGEgdmlydHVhbCBFTDIg bW9kZS4gIE5vcm1hbGx5IEtWTS9BUk0gb25seSBlbXVsYXRlZCBhIFZNIHN1cHBvcnRpbmcKPiBF TDEvMCBydW5uaW5nIGluIHRoZWlyIHJlc3BlY3RpdmUgbmF0aXZlIENQVSBtb2RlcywgYnV0IHdp dGggbmVzdGVkCj4gdmlydHVhbGl6YXRpb24gd2UgZGVwcml2aWxlZ2UgdGhlIGd1ZXN0IGh5cGVy dmlzb3IgYW5kIGVtdWxhdGUgYSB2aXJ0dWFsIEVMMgo+IGV4ZWN1dGlvbiBtb2RlIGluIEVMMSB1 c2luZyB0aGUgaGFyZHdhcmUgZmVhdHVyZXMgcHJvdmlkZWQgYnkgQVJNdjguMyB0byB0cmFwCj4g RUwyIG9wZXJhdGlvbnMgdG8gRUwxLiBUbyBkbyB0aGF0IHRoZSBob3N0IGh5cGVydmlzb3IgbmVl ZHMgdG8gbWFuYWdlIEVMMgo+IHJlZ2lzdGVyIHN0YXRlIGZvciB0aGUgZ3Vlc3QgaHlwZXJ2aXNv ciwgYW5kIHNoYWRvdyBFTDEgcmVnaXN0ZXIgc3RhdGUgdGhhdAo+IHJlZmxlY3RzIHRoZSBFTDIg cmVnaXN0ZXIgc3RhdGUgdG8gcnVuIHRoZSBndWVzdCBoeXBlcnZpc29yIGluIEVMMS4gU2VlIHBh dGNoIDYKPiB0aHJvdWdoIDEwIGZvciB0aGlzLgo+IAo+IEZvciBtZW1vcnkgdmlydHVhbGl6YXRp b24sIHRoZSBiaWdnZXN0IGlzc3VlIGlzIHRoYXQgd2Ugbm93IGhhdmUgbW9yZSB0aGFuIHR3bwo+ IHN0YWdlcyBvZiB0cmFuc2xhdGlvbiB3aGVuIHJ1bm5pbmcgbmVzdGVkIFZNcy4gV2UgY2hvb3Nl IHRvIG1lcmdlIHR3byBzdGFnZS0yCj4gcGFnZSB0YWJsZXMgKG9uZSBmcm9tIHRoZSBndWVzdCBo eXBlcnZpc29yIGFuZCB0aGUgb3RoZXIgZnJvbSB0aGUgaG9zdAo+IGh5cGVydmlzb3IpIGFuZCBj cmVhdGUgc2hhZG93IHN0YWdlLTIgcGFnZSB0YWJsZXMsIHdoaWNoIGhhdmUgbWFwcGluZ3MgZnJv bSB0aGUKPiBuZXN0ZWQgVk3igJlzIHBoeXNpY2FsIGFkZHJlc3NlcyB0byB0aGUgbWFjaGluZSBw aHlzaWNhbCBhZGRyZXNzZXMuIFN0YWdlLTEKPiB0cmFuc2xhdGlvbiBpcyBkb25lIGJ5IHRoZSBo YXJkd2FyZSBhcyBpcyBkb25lIGZvciB0aGUgbm9ybWFsIFZNcy4KPiAKPiBUbyBwcm92aWRlIFZH SUMgc3VwcG9ydCB0byB0aGUgZ3Vlc3QgaHlwZXJ2aXNvciwgd2UgZW11bGF0ZSB0aGUgR0lDCj4g dmlydHVhbGl6YXRpb24gZXh0ZW5zaW9ucyB1c2luZyB0cmFwLWFuZC1lbXVsYXRlIHRvIGEgdmly dHVhbCBHSUMgSHlwZXJ2aXNvcgo+IENvbnRyb2wgSW50ZXJmYWNlLiAgRnVydGhlcm1vcmUsIHdl IGNhbiBzdGlsbCB1c2UgdGhlIEdJQyBWRSBoYXJkd2FyZSBmZWF0dXJlcwo+IHRvIGRlbGl2ZXIg dmlydHVhbCBpbnRlcnJ1cHRzIHRvIHRoZSBuZXN0ZWQgVk0sIGJ5IGRpcmVjdGx5IG1hcHBpbmcg dGhlIEdJQwo+IFZDUFUgaW50ZXJmYWNlIHRvIHRoZSBuZXN0ZWQgVk0gYW5kIHN3aXRjaGluZyB0 aGUgY29udGVudCBvZiB0aGUgR0lDIEh5cGVydmlzb3IKPiBDb250cm9sIGludGVyZmFjZSB3aGVu IGFsdGVybmF0aW5nIGJldHdlZW4gYSBuZXN0ZWQgVk0gYW5kIGEgbm9ybWFsIFZNLiAgU2VlCj4g cGF0Y2hlcyAyNSB0aHJvdWdoIDMyLCBhbmQgNTAgdGhyb3VnaCA1MiBmb3IgbW9yZSBpbmZvcm1h dGlvbi4KPiAKPiBGb3IgdGltZXIgdmlydHVhbGl6YXRpb24sIHRoZSBndWVzdCBoeXBlcnZpc29y IGV4cGVjdHMgdG8gaGF2ZSBhY2Nlc3MgdG8gdGhlCj4gRUwyIHBoeXNpY2FsIHRpbWVyLCB0aGUg RUwxIHBoeXNpY2FsIHRpbWVyIGFuZCB0aGUgdmlydHVhbCB0aW1lci4gU28sIHRoZSBob3N0Cj4g aHlwZXJ2aXNvciBuZWVkcyB0byBwcm92aWRlIGFsbCBvZiB0aGVtLiBUaGUgdmlydHVhbCB0aW1l ciBpcyBhbHdheXMgYXZhaWxhYmxlCj4gdG8gVk1zLiBUaGUgcGh5c2ljYWwgdGltZXIgaXMgYXZh aWxhYmxlIHRvIFZNcyB2aWEgbXkgcHJldmlvdXMgcGF0Y2ggc2VyaWVzWzNdLgo+IFRoZSBFTDIg cGh5c2ljYWwgdGltZXIgaXMgbm90IHN1cHBvcnRlZCB5ZXQgaW4gdGhpcyBSRkMuIFdlIHBsYW4g dG8gc3VwcG9ydAo+IHRoaXMgYXMgaXQgaXMgcmVxdWlyZWQgdG8gcnVuIG90aGVyIGd1ZXN0IGh5 cGVydmlzb3JzIHN1Y2ggYXMgWGVuLgo+IAo+IEV2ZW4gdGhvdWdoIHRoaXMgd29yayBpcyBub3Qg Y29tcGxldGUgKHNlZSBsaW1pdGF0aW9ucyBiZWxvdyksIEknZCBhcHByZWNpYXRlCj4gZWFybHkg ZmVlZGJhY2sgb24gdGhpcyBSRkMuIFNwZWNpZmljYWxseSwgSSdtIGludGVyZXN0ZWQgaW46Cj4g LSBJcyBpdCBiZXR0ZXIgdG8gaGF2ZSBhIGtlcm5lbCBjb25maWcgb3IgdG8gbWFrZSBpdCBjb25m aWd1cmFibGUgYXQgcnVudGltZT8KPiAtIEkgd29uZGVyIGlmIHRoZSBkYXRhIHN0cnVjdHVyZSBm b3IgbWVtb3J5IG1hbmFnZW1lbnQgbWFrZXMgc2Vuc2UuCj4gLSBXaGF0IGFyY2hpdGVjdHVyZSB2 ZXJzaW9uIGRvIHdlIHN1cHBvcnQgZm9yIHRoZSBndWVzdCBoeXBlcnZpc29yLCBhbmQgaG93Pwo+ ICAgRm9yIGV4YW1wbGUsIGRvIHdlIGFsd2F5cyBzdXBwb3J0IGFsbCBhcmNoaXRlY3R1cmUgdmVy c2lvbnMgb3IgdGhlIHNhbWUKPiAgIGFyY2hpdGVjdHVyZSBhcyB0aGUgdW5kZXJseWluZyBoYXJk d2FyZSBwbGF0Zm9ybT8gT3IgaXMgaXQgYmV0dGVyCj4gICB0byBtYWtlIGl0IGNvbmZpZ3VyYWJs ZSBmcm9tIHRoZSB1c2Vyc3BhY2U/Cj4gLSBJbml0aWFsIGNvbW1lbnRzIG9uIHRoZSBvdmVyYWxs IGRlc2lnbj8KPiAKPiBUaGlzIHBhdGNoIHNlcmllcyBpcyBiYXNlZCBvbiBrdm0tYXJtLWZvci00 LjktcmM3IHdpdGggdGhlIHBhdGNoIHNlcmllcyB0byBwcm92aWRlCj4gVk1zIHdpdGggdGhlIEVM MSBwaHlzaWNhbCB0aW1lclsyXS4KPiAKPiBHaXQ6IGh0dHBzOi8vZ2l0aHViLmNvbS9jb2x1bWJp YS9uZXN0aW5nLXB1Yi90cmVlL3JmYy12MQo+IAo+IFRlc3Rpbmc6Cj4gV2UgaGF2ZSB0ZXN0ZWQg dGhpcyBvbiBBUk12OC4wIChBcHBsaWVkIE1pY3JvIFgtR2VuZSlbM10gc2luY2UgQVJNdjguMyBo YXJkd2FyZQo+IGlzIG5vdCBhdmFpbGFibGUgeWV0LiBXZSBoYXZlIHBhcmF2aXJ0dWFsaXplZCB0 aGUgZ3Vlc3QgaHlwZXJ2aXNvciB0byB0cmFwIHRvCj4gRUwyIGFzIHNwZWNpZmllZCBpbiBBUk12 OC4zIHNwZWNpZmljYXRpb24gdXNpbmcgaHZjIGluc3RydWN0aW9uLiBXZSBwbGFuIHRvCj4gdGVz dCB0aGlzIG9uIEFSTXY4LjMgbW9kZWwsIGFuZCB3aWxsIHBvc3QgdGhlIHJlc3VsdCBhbmQgdjIg aWYgbmVjZXNzYXJ5Lgo+IAo+IExpbWl0YXRpb25zOgo+IC0gVGhpcyBwYXRjaCBzZXJpZXMgb25s eSBzdXBwb3J0cyBhcm02NCwgbm90IGFybS4gQWxsIHRoZSBwYXRjaGVzIGNvbXBpbGUgb24KPiAg IGFybSwgYnV0IEkgaGF2ZW4ndCB0cnkgdG8gYm9vdCBub3JtYWwgVk1zIG9uIGl0Lgo+IC0gVGhl IGd1ZXN0IGh5cGVydmlzb3Igd2l0aCBWSEUgKEFSTXY4LjEpIGlzIG5vdCBzdXBwb3J0ZWQgaW4g dGhpcyBSRkMuIEkgaGF2ZQo+ICAgcGF0Y2hlcyBmb3IgdGhhdCwgYnV0IHRoZXkgbmVlZCB0byBi ZSBjbGVhbmVkIHVwLgo+IC0gUmVjdXJzaXZlIG5lc3RpbmcgKGkuZS4gZW11bGF0aW5nIEFSTXY4 LjMgaW4gdGhlIFZNKSBpcyBub3QgdGVzdGVkIHlldC4KPiAtIE90aGVyIGh5cGVydmlzb3JzIChz dWNoIGFzIFhlbikgb24gS1ZNIGFyZSBub3QgdGVzdGVkLgo+IAo+IFRPRE86Cj4gLSBUZXN0IHRv IGJvb3Qgbm9ybWFsIFZNcyBvbiBhcm0gYXJjaGl0ZWN0dXJlCj4gLSBUZXN0IHRoaXMgb24gQVJN djguMyBtb2RlbAo+IC0gU3VwcG9ydCB0aGUgZ3Vlc3QgaHlwZXJ2aXNvciB3aXRoIFZIRQo+IC0g UHJvdmlkZSB0aGUgZ3Vlc3QgaHlwZXJ2aXNvciB3aXRoIHRoZSBFTDIgcGh5c2ljYWwgdGltZXIK PiAtIFJ1biBvdGhlciBoeXBlcnZpc29ycyBzdWNoIGFzIFhlbiBvbiBLVk0KPiAKCkkgaGF2ZSBh IGNvdXBsZSBvZiBvdmVyYWxsIHF1ZXN0aW9ucyBhbmQgY29tbWVudHMgb24gdGhpcyBzZXJpZXM6 CgpGaXJzdCwgSSB0aGluayB3ZSBzaG91bGQgbWFrZSBzdXJlIHRoYXQgdGhlIHNlcmllcyBhY3R1 YWxseSB3b3JrcyB3aXRoCnY4LjMgb24gdGhlIG1vZGVsIHVzaW5nIGJvdGggVkhFIGFuZCBub24t VkhFIGZvciB0aGUgaG9zdCBoeXBlcnZpc29yLgoKU2Vjb25kLCB0aGlzIHBhdGNoIHNldCBpcyBw cmV0dHkgbGFyZ2Ugb3ZlcmFsbCBhbmQgaXQgd291bGQgYmUgZ3JlYXQgaWYKd2UgY291bGQgc3Bs aXQgaXQgdXAgaW50byBzb21lIHNsaWdodGx5IG1vcmUgbWFuYWdlYWJsZSBiaXRzLiAgSSdtIG5v dApleGFjdGx5IGhvdyB0byBkbyB0aGF0LCBidXQgcGVyaGFwcyB3ZSBjYW4gcmV3b3JrIGl0IHNv IHRoYXQgd2UgYWRkIGJpdHMKb2YgZnJhbWV3b3JrIChDUFUsIG1lbW9yeSwgaW50ZXJydXB0LCB0 aW1lcnMpIGFzIGluZGl2aWR1YWwgc2VyaWVzLCBhbmQKZmluYWxseSB3ZSBwbHVnIGFsbCB0aGUg bG9naWMgdG9nZXRoZXIgd2l0aCB0aGUgY3VycmVudCBmbG93LiAgV2hhdCBkbwp5b3UgdGhpbms/ CgpUaGlyZCwgd2Ugc2hvdWxkIGZvbGxvdyB0aGUgZmVlZGJhY2sgZnJvbSBEYXZpZCBhYm91dCBu b3QgdXNpbmcgYSBrZXJuZWwKY29uZmlnIG9wdGlvbi4gIEknbSBhZnJhaWQgdGhhdCBzb21lIGNv ZGUgd2lsbCBiaXRyb3QgdG9vIGZhc3QgaWYgZ3VpZGVkCmJ5IGEga2VybmVsIGNvbmZpZyBvcHRp b24sIHNvIGEgcnVudGltZSBwYXJhbWV0ZXIgYW5kIHVzaW5nIHN0YXRpYyBrZXlzCndoZXJlIHJl bGV2YW50IHNlZW1zIGxpa2UgYSBiZXR0ZXIgYXBwcm9hY2ggdG8gbWUuICBCdXQgc2luY2UgS1ZN L0FSTSBpcwpub3QgbG9hZGVkIGFzIGEgbW9kdWxlLCB0aGlzIHdvdWxkIGhhdmUgdG8gYmUgYSBr ZXJuZWwgY21kbGluZQpwYXJhbWV0ZXIuICBXaGF0IGRvIHBlb3BsZSB0aGluaz8KCkZvdXJ0aCwg dGhlcmUgYXJlIHNvbWUgcGxhY2VzIHdoZXJlIHdlIGhhdmUgaGFyZC1jb2RlZCBpbmZvcm1hdGlv biAobGlrZQp0aGUgbG9jYXRpb24gb2YgdGhlIEdJQ0gvR0lDViBpbnRlcmZhY2VzKSB3aGljaCBo YXZlIHRvIGJlIGZpeGVkIGJ5CmFkZGluZyB0aGUgcmVxdWlyZWQgdXNlcnNwYWNlIGludGVyZmFj ZXMuCgpGaWZ0aCwgdGhlIG9yZGVyaW5nIG9mIHRoZSBwYXRjaGVzIG5lZWRzIGEgYml0IG9mIGxv dmUuIEkgdGhpbmsgaXQncwppbXBvcnRhbnQgdGhhdCB3ZSBidWlsZCB0aGUgd2hvbGUgaW5mcmFz dHJ1Y3R1cmUgZmlyc3QsIGJ1dCBsZWF2ZSBpdApjb21wbGV0ZWx5IGRpc2FibGVkIHVudGlsIHRo ZSBlbmQsIGFuZCB0aGVuIHdlIHBsdWcgaW4gYWxsIHRoZQpjYXBhYmlsaXRpZXMgb2YgdXNlcnNw YWNlIHRvIGNyZWF0ZSBhIG5lc3RlZCBWTSBpbiB0aGUgZW5kLiAgU28gZm9yCmV4YW1wbGUsIEkg d291bGQgZXhwZWN0IHRoYXQgcGF0Y2ggMDMgd291bGQgYmUgdGhlIGxhc3QgcGF0Y2ggaW4gdGhl CnNlcmllcy4KCk92ZXJhbGwgdGhvdWdoLCB0aGlzIGlzIGEgbWFzc2l2ZSBhbW91bnQgb2Ygd29y aywgYW5kIGl0J3MgYXdlc29tZSB0aGF0CnlvdSB3ZXJlIGFibGUgdG8gcHVsbCBpdCB0b2dldGhl ciB0byBhIHByZXR0eSBuaWNlIGluaXRpYWwgUkZDIQoKVGhhbmtzIQotQ2hyaXN0b2ZmZXIKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18Ka3ZtYXJtIG1haWxp bmcgbGlzdAprdm1hcm1AbGlzdHMuY3MuY29sdW1iaWEuZWR1Cmh0dHBzOi8vbGlzdHMuY3MuY29s dW1iaWEuZWR1L21haWxtYW4vbGlzdGluZm8va3ZtYXJtCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: cdall@linaro.org (Christoffer Dall) Date: Wed, 22 Feb 2017 19:23:31 +0100 Subject: [RFC 00/55] Nested Virtualization on KVM/ARM In-Reply-To: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> References: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> Message-ID: <20170222182331.GA27653@cbox> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi Jintack, On Mon, Jan 09, 2017 at 01:23:56AM -0500, Jintack Lim wrote: > Nested virtualization is the ability to run a virtual machine inside another > virtual machine. In other words, it?s about running a hypervisor (the guest > hypervisor) on top of another hypervisor (the host hypervisor). > > This series supports nested virtualization on arm64. ARM recently announced an > extension (ARMv8.3) which has support for nested virtualization[1]. This series > is based on the ARMv8.3 specification. > > Supporting nested virtualization means that the hypervisor provides not only > EL0/EL1 execution environment with VMs as it usually does, but also the > virtualization extensions including EL2 execution environment with the VMs. > Once the host hypervisor provides those execution environment with the VMs, > then the guest hypervisor can run its own VMs (nested VMs) naturally. > > To support nested virtualization on ARM the hypervisor must emulate a virtual > execution environment consisting of EL2, EL1, and EL0, as the guest hypervisor > will run in a virtual EL2 mode. Normally KVM/ARM only emulated a VM supporting > EL1/0 running in their respective native CPU modes, but with nested > virtualization we deprivilege the guest hypervisor and emulate a virtual EL2 > execution mode in EL1 using the hardware features provided by ARMv8.3 to trap > EL2 operations to EL1. To do that the host hypervisor needs to manage EL2 > register state for the guest hypervisor, and shadow EL1 register state that > reflects the EL2 register state to run the guest hypervisor in EL1. See patch 6 > through 10 for this. > > For memory virtualization, the biggest issue is that we now have more than two > stages of translation when running nested VMs. We choose to merge two stage-2 > page tables (one from the guest hypervisor and the other from the host > hypervisor) and create shadow stage-2 page tables, which have mappings from the > nested VM?s physical addresses to the machine physical addresses. Stage-1 > translation is done by the hardware as is done for the normal VMs. > > To provide VGIC support to the guest hypervisor, we emulate the GIC > virtualization extensions using trap-and-emulate to a virtual GIC Hypervisor > Control Interface. Furthermore, we can still use the GIC VE hardware features > to deliver virtual interrupts to the nested VM, by directly mapping the GIC > VCPU interface to the nested VM and switching the content of the GIC Hypervisor > Control interface when alternating between a nested VM and a normal VM. See > patches 25 through 32, and 50 through 52 for more information. > > For timer virtualization, the guest hypervisor expects to have access to the > EL2 physical timer, the EL1 physical timer and the virtual timer. So, the host > hypervisor needs to provide all of them. The virtual timer is always available > to VMs. The physical timer is available to VMs via my previous patch series[3]. > The EL2 physical timer is not supported yet in this RFC. We plan to support > this as it is required to run other guest hypervisors such as Xen. > > Even though this work is not complete (see limitations below), I'd appreciate > early feedback on this RFC. Specifically, I'm interested in: > - Is it better to have a kernel config or to make it configurable at runtime? > - I wonder if the data structure for memory management makes sense. > - What architecture version do we support for the guest hypervisor, and how? > For example, do we always support all architecture versions or the same > architecture as the underlying hardware platform? Or is it better > to make it configurable from the userspace? > - Initial comments on the overall design? > > This patch series is based on kvm-arm-for-4.9-rc7 with the patch series to provide > VMs with the EL1 physical timer[2]. > > Git: https://github.com/columbia/nesting-pub/tree/rfc-v1 > > Testing: > We have tested this on ARMv8.0 (Applied Micro X-Gene)[3] since ARMv8.3 hardware > is not available yet. We have paravirtualized the guest hypervisor to trap to > EL2 as specified in ARMv8.3 specification using hvc instruction. We plan to > test this on ARMv8.3 model, and will post the result and v2 if necessary. > > Limitations: > - This patch series only supports arm64, not arm. All the patches compile on > arm, but I haven't try to boot normal VMs on it. > - The guest hypervisor with VHE (ARMv8.1) is not supported in this RFC. I have > patches for that, but they need to be cleaned up. > - Recursive nesting (i.e. emulating ARMv8.3 in the VM) is not tested yet. > - Other hypervisors (such as Xen) on KVM are not tested. > > TODO: > - Test to boot normal VMs on arm architecture > - Test this on ARMv8.3 model > - Support the guest hypervisor with VHE > - Provide the guest hypervisor with the EL2 physical timer > - Run other hypervisors such as Xen on KVM > I have a couple of overall questions and comments on this series: First, I think we should make sure that the series actually works with v8.3 on the model using both VHE and non-VHE for the host hypervisor. Second, this patch set is pretty large overall and it would be great if we could split it up into some slightly more manageable bits. I'm not exactly how to do that, but perhaps we can rework it so that we add bits of framework (CPU, memory, interrupt, timers) as individual series, and finally we plug all the logic together with the current flow. What do you think? Third, we should follow the feedback from David about not using a kernel config option. I'm afraid that some code will bitrot too fast if guided by a kernel config option, so a runtime parameter and using static keys where relevant seems like a better approach to me. But since KVM/ARM is not loaded as a module, this would have to be a kernel cmdline parameter. What do people think? Fourth, there are some places where we have hard-coded information (like the location of the GICH/GICV interfaces) which have to be fixed by adding the required userspace interfaces. Fifth, the ordering of the patches needs a bit of love. I think it's important that we build the whole infrastructure first, but leave it completely disabled until the end, and then we plug in all the capabilities of userspace to create a nested VM in the end. So for example, I would expect that patch 03 would be the last patch in the series. Overall though, this is a massive amount of work, and it's awesome that you were able to pull it together to a pretty nice initial RFC! Thanks! -Christoffer From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933268AbdBVSYB (ORCPT ); Wed, 22 Feb 2017 13:24:01 -0500 Received: from mail-wr0-f179.google.com ([209.85.128.179]:34746 "EHLO mail-wr0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933226AbdBVSXv (ORCPT ); Wed, 22 Feb 2017 13:23:51 -0500 Date: Wed, 22 Feb 2017 19:23:31 +0100 From: Christoffer Dall To: Jintack Lim Cc: christoffer.dall@linaro.org, marc.zyngier@arm.com, pbonzini@redhat.com, rkrcmar@redhat.com, linux@armlinux.org.uk, catalin.marinas@arm.com, will.deacon@arm.com, vladimir.murzin@arm.com, suzuki.poulose@arm.com, mark.rutland@arm.com, james.morse@arm.com, lorenzo.pieralisi@arm.com, kevin.brodsky@arm.com, wcohen@redhat.com, shankerd@codeaurora.org, geoff@infradead.org, andre.przywara@arm.com, eric.auger@redhat.com, anna-maria@linutronix.de, shihwei@cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [RFC 00/55] Nested Virtualization on KVM/ARM Message-ID: <20170222182331.GA27653@cbox> References: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1483943091-1364-1-git-send-email-jintack@cs.columbia.edu> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Jintack, On Mon, Jan 09, 2017 at 01:23:56AM -0500, Jintack Lim wrote: > Nested virtualization is the ability to run a virtual machine inside another > virtual machine. In other words, it’s about running a hypervisor (the guest > hypervisor) on top of another hypervisor (the host hypervisor). > > This series supports nested virtualization on arm64. ARM recently announced an > extension (ARMv8.3) which has support for nested virtualization[1]. This series > is based on the ARMv8.3 specification. > > Supporting nested virtualization means that the hypervisor provides not only > EL0/EL1 execution environment with VMs as it usually does, but also the > virtualization extensions including EL2 execution environment with the VMs. > Once the host hypervisor provides those execution environment with the VMs, > then the guest hypervisor can run its own VMs (nested VMs) naturally. > > To support nested virtualization on ARM the hypervisor must emulate a virtual > execution environment consisting of EL2, EL1, and EL0, as the guest hypervisor > will run in a virtual EL2 mode. Normally KVM/ARM only emulated a VM supporting > EL1/0 running in their respective native CPU modes, but with nested > virtualization we deprivilege the guest hypervisor and emulate a virtual EL2 > execution mode in EL1 using the hardware features provided by ARMv8.3 to trap > EL2 operations to EL1. To do that the host hypervisor needs to manage EL2 > register state for the guest hypervisor, and shadow EL1 register state that > reflects the EL2 register state to run the guest hypervisor in EL1. See patch 6 > through 10 for this. > > For memory virtualization, the biggest issue is that we now have more than two > stages of translation when running nested VMs. We choose to merge two stage-2 > page tables (one from the guest hypervisor and the other from the host > hypervisor) and create shadow stage-2 page tables, which have mappings from the > nested VM’s physical addresses to the machine physical addresses. Stage-1 > translation is done by the hardware as is done for the normal VMs. > > To provide VGIC support to the guest hypervisor, we emulate the GIC > virtualization extensions using trap-and-emulate to a virtual GIC Hypervisor > Control Interface. Furthermore, we can still use the GIC VE hardware features > to deliver virtual interrupts to the nested VM, by directly mapping the GIC > VCPU interface to the nested VM and switching the content of the GIC Hypervisor > Control interface when alternating between a nested VM and a normal VM. See > patches 25 through 32, and 50 through 52 for more information. > > For timer virtualization, the guest hypervisor expects to have access to the > EL2 physical timer, the EL1 physical timer and the virtual timer. So, the host > hypervisor needs to provide all of them. The virtual timer is always available > to VMs. The physical timer is available to VMs via my previous patch series[3]. > The EL2 physical timer is not supported yet in this RFC. We plan to support > this as it is required to run other guest hypervisors such as Xen. > > Even though this work is not complete (see limitations below), I'd appreciate > early feedback on this RFC. Specifically, I'm interested in: > - Is it better to have a kernel config or to make it configurable at runtime? > - I wonder if the data structure for memory management makes sense. > - What architecture version do we support for the guest hypervisor, and how? > For example, do we always support all architecture versions or the same > architecture as the underlying hardware platform? Or is it better > to make it configurable from the userspace? > - Initial comments on the overall design? > > This patch series is based on kvm-arm-for-4.9-rc7 with the patch series to provide > VMs with the EL1 physical timer[2]. > > Git: https://github.com/columbia/nesting-pub/tree/rfc-v1 > > Testing: > We have tested this on ARMv8.0 (Applied Micro X-Gene)[3] since ARMv8.3 hardware > is not available yet. We have paravirtualized the guest hypervisor to trap to > EL2 as specified in ARMv8.3 specification using hvc instruction. We plan to > test this on ARMv8.3 model, and will post the result and v2 if necessary. > > Limitations: > - This patch series only supports arm64, not arm. All the patches compile on > arm, but I haven't try to boot normal VMs on it. > - The guest hypervisor with VHE (ARMv8.1) is not supported in this RFC. I have > patches for that, but they need to be cleaned up. > - Recursive nesting (i.e. emulating ARMv8.3 in the VM) is not tested yet. > - Other hypervisors (such as Xen) on KVM are not tested. > > TODO: > - Test to boot normal VMs on arm architecture > - Test this on ARMv8.3 model > - Support the guest hypervisor with VHE > - Provide the guest hypervisor with the EL2 physical timer > - Run other hypervisors such as Xen on KVM > I have a couple of overall questions and comments on this series: First, I think we should make sure that the series actually works with v8.3 on the model using both VHE and non-VHE for the host hypervisor. Second, this patch set is pretty large overall and it would be great if we could split it up into some slightly more manageable bits. I'm not exactly how to do that, but perhaps we can rework it so that we add bits of framework (CPU, memory, interrupt, timers) as individual series, and finally we plug all the logic together with the current flow. What do you think? Third, we should follow the feedback from David about not using a kernel config option. I'm afraid that some code will bitrot too fast if guided by a kernel config option, so a runtime parameter and using static keys where relevant seems like a better approach to me. But since KVM/ARM is not loaded as a module, this would have to be a kernel cmdline parameter. What do people think? Fourth, there are some places where we have hard-coded information (like the location of the GICH/GICV interfaces) which have to be fixed by adding the required userspace interfaces. Fifth, the ordering of the patches needs a bit of love. I think it's important that we build the whole infrastructure first, but leave it completely disabled until the end, and then we plug in all the capabilities of userspace to create a nested VM in the end. So for example, I would expect that patch 03 would be the last patch in the series. Overall though, this is a massive amount of work, and it's awesome that you were able to pull it together to a pretty nice initial RFC! Thanks! -Christoffer