From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jike Song Subject: [ANNOUNCE][RFC] KVMGT - the implementation of Intel GVT-g(full GPU virtualization) for KVM Date: Thu, 04 Dec 2014 10:24:30 +0800 Message-ID: <547FC5DE.4010701@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by gabe.freedesktop.org (Postfix) with ESMTP id 695826E41A for ; Wed, 3 Dec 2014 18:27:19 -0800 (PST) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: "intel-gfx@lists.freedesktop.org" , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: "White, Michael L" , "Cowperthwaite, David J" , "Li, Susie" , "Dong, Eddie" , "Haron, Sandra" List-Id: intel-gfx@lists.freedesktop.org SGkgYWxsLAoKICBXZSBhcmUgcGxlYXNlZCB0byBhbm5vdW5jZSB0aGUgZmlyc3QgcmVsZWFzZSBv ZiBLVk1HVCBwcm9qZWN0LiBLVk1HVCBpcyB0aGUgaW1wbGVtZW50YXRpb24gb2YgSW50ZWwgR1ZU LWcgdGVjaG5vbG9neSwgYSBmdWxsIEdQVSB2aXJ0dWFsaXphdGlvbiBzb2x1dGlvbi4gVW5kZXIg SW50ZWwgR1ZULWcsIGEgdmlydHVhbCBHUFUgaW5zdGFuY2UgaXMgbWFpbnRhaW5lZCBmb3IgZWFj aCBWTSwgd2l0aCBwYXJ0IG9mIHBlcmZvcm1hbmNlIGNyaXRpY2FsIHJlc291cmNlcyBkaXJlY3Rs eSBhc3NpZ25lZC4gVGhlIGNhcGFiaWxpdHkgb2YgcnVubmluZyBuYXRpdmUgZ3JhcGhpY3MgZHJp dmVyIGluc2lkZSBhIFZNLCB3aXRob3V0IGh5cGVydmlzb3IgaW50ZXJ2ZW50aW9uIGluIHBlcmZv cm1hbmNlIGNyaXRpY2FsIHBhdGhzLCBhY2hpZXZlcyBhIGdvb2QgYmFsYW5jZSBvZiBwZXJmb3Jt YW5jZSwgZmVhdHVyZSwgYW5kIHNoYXJpbmcgY2FwYWJpbGl0eS4KCgogIEtWTUdUIGlzIHN0aWxs IGluIHRoZSBlYXJseSBzdGFnZToKCiAgIC0gQmFzaWMgZnVuY3Rpb25zIG9mIGZ1bGwgR1BVIHZp cnR1YWxpemF0aW9uIHdvcmtzLCBndWVzdCBjYW4gc2VlIGEgZnVsbC1mZWF0dXJlZCB2R1BVLgog ICAgIFdlIHJhbiBzZXZlcmFsIDNEIHdvcmtsb2FkcyBzdWNoIGFzIGxpZ2h0c21hcmssIG5leHVp eiwgdXJiYW50ZXJyb3IgYW5kIHdhcnNvdy4KCiAgIC0gT25seSBMaW51eCBndWVzdCBzdXBwb3J0 ZWQgc28gZmFyLCBhbmQgUFBHVFQgbXVzdCBiZSBkaXNhYmxlZCBpbiBndWVzdCB0aHJvdWdoIGEK ICAgICBrZXJuZWwgcGFyYW1ldGVyKHNlZSBSRUFETUUua3ZtZ3QgaW4gUUVNVSkuCgogICAtIFRo aXMgZHJvcCBhbHNvIGluY2x1ZGVzIHNvbWUgWGVuIHNwZWNpZmljIGNoYW5nZXMsIHdoaWNoIHdp bGwgYmUgY2xlYW5lZCB1cCBsYXRlci4KCiAgIC0gT3VyIGVuZCBnb2FsIGlzIHRvIHVwc3RyZWFt IGJvdGggWGVuR1QgYW5kIEtWTUdULCB3aGljaCBzaGFyZXMgfjkwJSBsb2dpYyBmb3IgdkdQVQog ICAgIGRldmljZSBtb2RlbCAod2lsbCBiZSBwYXJ0IG9mIGk5MTUgZHJpdmVyKSwgd2l0aCBvbmx5 IGRpZmZlcmVuY2UgaW4gaHlwZXJ2aXNvcgogICAgIHNwZWNpZmljIHNlcnZpY2VzCgogICAtIGlu c3VmZmljaWVudCB0ZXN0IGNvdmVyYWdlLCBzbyBwbGVhc2UgYmVhciB3aXRoIHN0YWJpbGl0eSBp c3N1ZXMgOikKCgoKICBUaGVyZSBhcmUgdGhpbmdzIG5lZWQgdG8gYmUgaW1wcm92ZWQsIGVzcC4g dGhlIEtWTSBpbnRlcmZhY2luZyBwYXJ0OgoKCTEJYSBkb21pZCB3YXMgYWRkZWQgdG8gZWFjaCBL Vk1HVCBndWVzdAoKCQlBbiBJRCBpcyBuZWVkZWQgZm9yIGZvcmVncm91bmQgT1Mgc3dpdGNoaW5n LCBlLmcuCgoJCQkjIGVjaG8gPGRvbWlkPgk+CS9zeXMva2VybmVsL3ZndC9jb250cm9sL2ZvcmVn cm91bmRfdm0KCgkJZG9taWQgMCBpcyByZXNlcnZlZCBmb3IgaG9zdCBPUy4KCgogIAkyCVNSQ1Ug d29ya2Fyb3VuZHMuCgoJCVNvbWUgS1ZNIGZ1bmN0aW9ucywgc3VjaCBhczoKCgkJCQlrdm1faW9f YnVzX3JlZ2lzdGVyX2RldgoJCQkJaW5zdGFsbF9uZXdfbWVtc2xvdHMKCgkJbXVzdCBiZSBjYWxs ZWQgKndpdGhvdXQqICZrdm0tPnNyY3UgcmVhZC1sb2NrZWQuIE90aGVyd2lzZSBpdCBoYW5ncy4K CgkJSW4gS1ZNR1QsIHdlIG5lZWQgdG8gcmVnaXN0ZXIgYW4gaW9kZXYgb25seSAqYWZ0ZXIqIEJB UiByZWdpc3RlcnMgYXJlCgkJd3JpdHRlbiBieSBndWVzdC4gVGhhdCBtZWFucywgd2UgYWxyZWFk eSBoYXZlICZrdm0tPnNyY3UgaG9sZCAtCgkJdHJhcHBpbmcvZW11bGF0aW5nIFBJTyhCQVIgcmVn aXN0ZXJzKSBtYWtlcyB1cyBpbiBzdWNoIGEgY29uZGl0aW9uLgoJCVRoYXQgd2lsbCBtYWtlIGt2 bV9pb19idXNfcmVnaXN0ZXJfZGV2IGhhbmdzLgoKCQlDdXJyZW50bHkgd2UgaGF2ZSB0byBkaXNh YmxlIHJjdV9hc3NpZ25fcG9pbnRlcigpIGluIHN1Y2ggZnVuY3Rpb25zLgoKCQlUaGVzZSB3ZXJl IGRpcnR5IHdvcmthcm91bmRzLCB5b3VyIHN1Z2dlc3Rpb25zIGFyZSBoaWdoIHdlbGNvbWUhCgoK CTMJc3lzY2FsbHMgd2VyZSBjYWxsZWQgdG8gYWNjZXNzICIvZGV2L21lbSIgZnJvbSBrZXJuZWwK CgkJQW4gaW4ta2VybmVsIG1lbXNsb3Qgd2FzIGFkZGVkIGZvciBhcGVydHVyZSwgYnV0IHVzaW5n IHN5c2NhbGxzIGxpa2UKCQlvcGVuIGFuZCBtbWFwIHRvIG9wZW4gYW5kIGFjY2VzcyB0aGUgY2hh cmFjdGVyIGRldmljZSAiL2Rldi9tZW0iLAoJCWZvciBwYXNzLXRocm91Z2guCgogIAoKClRoZSBz b3VyY2UgY29kZXMoa2VybmVsLCBxZW11IGFzIHdlbGwgYXMgc2VhYmlvcykgYXJlIGF2YWlsYWJs ZSBhdCBnaXRodWI6CgoJZ2l0Oi8vZ2l0aHViLmNvbS8wMW9yZy9LVk1HVC1rZXJuZWwKCWdpdDov L2dpdGh1Yi5jb20vMDFvcmcvS1ZNR1QtcWVtdQoJZ2l0Oi8vZ2l0aHViLmNvbS8wMW9yZy9LVk1H VC1zZWFiaW9zCgpJbiB0aGUgS1ZNR1QtcWVtdSByZXBvc2l0b3J5LCB0aGVyZSBpcyBhICJSRUFE TUUua3ZtZ3QiIHRvIGJlIHJlZmVycmVkLgoKCgpNb3JlIGluZm9ybWF0aW9uIGFib3V0IEludGVs IEdWVC1nIGFuZCBLVk1HVCBjYW4gYmUgZm91bmQgYXQ6CgoJaHR0cHM6Ly93d3cudXNlbml4Lm9y Zy9jb25mZXJlbmNlL2F0YzE0L3RlY2huaWNhbC1zZXNzaW9ucy9wcmVzZW50YXRpb24vdGlhbgoJ aHR0cDovL2V2ZW50cy5saW51eGZvdW5kYXRpb24ub3JnL3NpdGVzL2V2ZW50cy9maWxlcy9zbGlk ZXMvS1ZNR1QtYSUyMEZ1bGwlMjBHUFUlMjBWaXJ0dWFsaXphdGlvbiUyMFNvbHV0aW9uXzEucGRm CgoKQXBwcmVjaWF0ZSB5b3VyIGNvbW1lbnRzLCBCVUcgcmVwb3J0cywgYW5kIGNvbnRyaWJ1dGlv bnMhCgoKCgotLQpUaGFua3MsCkppa2UKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJl ZWRlc2t0b3Aub3JnCmh0dHA6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5m by9pbnRlbC1nZngK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752722AbaLDC1U (ORCPT ); Wed, 3 Dec 2014 21:27:20 -0500 Received: from mga11.intel.com ([192.55.52.93]:52828 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750840AbaLDC1T (ORCPT ); Wed, 3 Dec 2014 21:27:19 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,512,1413270000"; d="scan'208";a="642079452" Message-ID: <547FC5DE.4010701@intel.com> Date: Thu, 04 Dec 2014 10:24:30 +0800 From: Jike Song User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: "intel-gfx@lists.freedesktop.org" , kvm@vger.kernel.org, linux-kernel@vger.kernel.org CC: "Tian, Kevin" , "Cowperthwaite, David J" , "White, Michael L" , "Dong, Eddie" , "Li, Susie" , "Haron, Sandra" Subject: [ANNOUNCE][RFC] KVMGT - the implementation of Intel GVT-g(full GPU virtualization) for KVM Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, We are pleased to announce the first release of KVMGT project. KVMGT is the implementation of Intel GVT-g technology, a full GPU virtualization solution. Under Intel GVT-g, a virtual GPU instance is maintained for each VM, with part of performance critical resources directly assigned. The capability of running native graphics driver inside a VM, without hypervisor intervention in performance critical paths, achieves a good balance of performance, feature, and sharing capability. KVMGT is still in the early stage: - Basic functions of full GPU virtualization works, guest can see a full-featured vGPU. We ran several 3D workloads such as lightsmark, nexuiz, urbanterror and warsow. - Only Linux guest supported so far, and PPGTT must be disabled in guest through a kernel parameter(see README.kvmgt in QEMU). - This drop also includes some Xen specific changes, which will be cleaned up later. - Our end goal is to upstream both XenGT and KVMGT, which shares ~90% logic for vGPU device model (will be part of i915 driver), with only difference in hypervisor specific services - insufficient test coverage, so please bear with stability issues :) There are things need to be improved, esp. the KVM interfacing part: 1 a domid was added to each KVMGT guest An ID is needed for foreground OS switching, e.g. # echo > /sys/kernel/vgt/control/foreground_vm domid 0 is reserved for host OS. 2 SRCU workarounds. Some KVM functions, such as: kvm_io_bus_register_dev install_new_memslots must be called *without* &kvm->srcu read-locked. Otherwise it hangs. In KVMGT, we need to register an iodev only *after* BAR registers are written by guest. That means, we already have &kvm->srcu hold - trapping/emulating PIO(BAR registers) makes us in such a condition. That will make kvm_io_bus_register_dev hangs. Currently we have to disable rcu_assign_pointer() in such functions. These were dirty workarounds, your suggestions are high welcome! 3 syscalls were called to access "/dev/mem" from kernel An in-kernel memslot was added for aperture, but using syscalls like open and mmap to open and access the character device "/dev/mem", for pass-through. The source codes(kernel, qemu as well as seabios) are available at github: git://github.com/01org/KVMGT-kernel git://github.com/01org/KVMGT-qemu git://github.com/01org/KVMGT-seabios In the KVMGT-qemu repository, there is a "README.kvmgt" to be referred. More information about Intel GVT-g and KVMGT can be found at: https://www.usenix.org/conference/atc14/technical-sessions/presentation/tian http://events.linuxfoundation.org/sites/events/files/slides/KVMGT-a%20Full%20GPU%20Virtualization%20Solution_1.pdf Appreciate your comments, BUG reports, and contributions! -- Thanks, Jike